devopsy.kinsta.cloud

Author: Sean Porter

  • How to Manage Data in Hybrid Cloud Environments

    How to Manage Data in Hybrid Cloud Environments

    We’ve long-since established that multi-generational, multi-cloud environments are the new norm. But, how does one go about managing data in those environments?

    In this post, we’ll take a look at the ramifications of this hybrid cloud reality and best practices for managing your data for increased scalability and heightened availability.

    It’s All Cloud

    Whether it’s Amazon Web Services (AWS), Google Cloud or Microsoft Azure (or some combination thereof), cloud is all the rage, with a hybrid approach gaining momentum with 81% of cloud users working with two or more providers (according to a recent Gartner study). Not only does this strategy take companies out of the business of hosting their own applications, but it also leads to benefits such as avoiding vendor lock-in, reduced costs and optimized performance.

    Benefits to Mixing Cloud and On-Prem Environments

    Mixing cloud and on-prem environments means you can pick and choose where your workloads and data live—it’s really an optimization. For example, if your application is CPU intensive, you can run it on the cloud provider that gives you the most CPU for your dollar. Similarly, if your data workload requires significant storage, you can go to the cloud provider that offers the best bang for your buck on storage.

    Businesses should look at their application and database and then use the cloud provider that offers the best of class for that workload. A mixed cloud also gives you the ability to always have a low-cost secondary site recovery plan in place—you’re always ready to spin up on another cloud provider if you need to. 

    Managing Data: The Challenges of Hybrid Cloud

    The benefits of multi-cloud are many, but when it comes to managing your data, things can get tricky. Not only do you have multiple cloud providers to balance, you also have to consider data stored on-premises and how that data is accessed from the cloud. 

    You also have to consider data replication between clouds. If you’re not doing a full mirror of your data between each cloud, it’s critical you remain acutely aware of what data lives where. You’ll want to consider creating extensive documentation and diagrams of what data lives in which cloud. If you are replicating data between clouds, you also have to justify the additional operating and engineering cost to accomplish your goal. 

    How to Manage Data in Hybrid Cloud Environments

    Fortunately, there are several libraries providing abstractions for cloud services, such as object stores. Leveraging abstractions within your applications allows you to move more freely between cloud providers. 

    Companies should strongly consider storing their data in a consistent manner within each cloud, leveraging the same technologies to reduce the sprawl, maintenance and expertise required to operate those technologies. For example, use a relational database that all the cloud vendors provide—for data updated and changed frequently, it should live in the scope of the local cloud provider to reduce latency and network costs, because traffic within the cloud provider is generally free. If you were to write to a database outside the cloud provider, it’d cost more and be considerably slower.

    The leaders in the cloud provider space offer a fairly secure environment for storing sensitive data, as long as you’re following good practices and using your own source of encryption. However, it all comes down to your individual environment. For some organizations, there will be workloads that should remain on-premises—for instance, highly sensitive data for which regulations require on-premises storage.

    If data has to be stored and encrypted on-premises, many cloud providers offer a secure means of connecting your cloud environments to your on-premises environment for access to sensitive information, such as providing secure VPN connectivity to virtual private clouds. 

    Multi-cloud might not be for everyone; you have to weigh the benefits with the costs, especially as those factors relate to your individual organization. While there are pitfalls, the practices and tooling to effectively manage your data in hybrid cloud environments are all out there. Because so many of us are choosing to run our workloads on multiple clouds, collectively we’ve overcome many of the aforementioned challenges and can take advantage of all the benefits hybrid cloud provides.

  • DevOps Tools for the Monitoring Ecosystem

    DevOps Tools for the Monitoring Ecosystem

    DevOps is a cultural and process transformation—the goal of it is to achieve a higher rate of change and quality of software. There are categories of tools to help facilitate this transformation through automation and increased measurement and visibility. You can’t have or observe progress without measurement, and throughout your transformation you need to maintain a high degree of visibility in order to move quickly, safely. Not only is the idea of cultural and process transformation daunting for many organizations, adopting the tools to help you on that journey is even more so, as it requires a great deal of learning. 

    An integrated set of DevOps tools for monitoring has the power to improve visibility and productivity, achieve higher-performing systems and establish cross-functional collaboration. The right toolset is more than the tools themselves—it’s about developing the culture, discipline and practices that come to define your product/service and your workplace.

    This article will outline some of the best DevOps tools and practices when it comes to the monitoring ecosystem that help developer and operations teams work together effectively.

    Monitoring Tools

    A good monitoring platform lets you monitor infrastructure and application performance, whether on-prem, in the cloud or across containerized environments—so you have complete visibility into every system, all the time. Whether you want to monitor Kubernetes, IoT devices or bare metal, the right monitoring tool helps make it possible.

    An effective monitoring tool improves system performance and productivity, and helps you reduce downtime. You can adequately plan for upgrades and new projects, and better allocate your time and resources. You can detect problems—and solve them—before they impact users.

    There are a ton of great monitoring tools out there; I won’t go into them all here, but I’ll highlight the top three in terms of popularity.

    Love it or hate it, Nagios is still very much a widely used tool and formed the foundation for monitoring as we know it today. Plus, the Nagios service check is actually awesome (and underappreciated). Where Nagios falls short, however, is when it comes to monitoring ephemeral environments, such as Kubernetes and Docker, at scale. In reality, Nagios falls short when it comes to monitoring at scale in any environment, whether that’s virtualized or not. Enter Prometheus, an open source monitoring tool (and, like Kubernetes, a part of the CNCF) that uses a pull-based architecture. Instead of pushing metrics to the monitoring tool, it pulls metrics from services. Due to Kubernetes’ built-in exporters, Prometheus is often the go-to for monitoring Kubernetes, but it falls short when it comes to monitoring older infrastructure (see more pros and cons of Prometheus here).

    That brings me to Sensu, which offers the best of both worlds: you can reuse Nagios service checks in Sensu and monitor ephemeral environments at scale. If you have a strictly container-based environment, then Prometheus is a good choice; if you have a mix of multi-generational infrastructure (which we’re finding more often than not with our customers), then Sensu is the way to go.

    Configuration Management Tools

    Configuration management tools allow you to automate the provisioning and deployment of systems, enforce desired configurations and remediate configuration drift. By modeling your infrastructure as code, you can apply software delivery practices such as version control, automated testing and continuous delivery to infrastructure and applications.

    Automating work that used to be manual, repetitive and error-prone results in greater speed, predictability and scalability—and the assurance of standardized configurations across test, developer and production environments. Eliminating snowflake servers reduces time (and headaches) and lets you deploy software faster and more reliably.

    Popular configuration management tools include: Ansible, which is written in Python, agent-less and utilizes an imperative (as opposed to a declarative) approach. Puppet is another popular choice—it relies on a declarative config management approach and uses domain-specific language and an agent/master architecture. Finally, there’s Chef, which is written in Ruby and Erlang, and is modeled similar to Puppet.

    Alerting Tools

    Alerting tools provide both actionable and informational system alerts, and can be customized to fit the complexities of your systems. For example, your alerting system needs to be sensitive enough to cover an outage—but not so sensitive that you’re catching frequent, intermittent problems that users aren’t going to see and that would inundate you with needless alerts.

    Alerting tools help lay the foundation for your alerting policies, so you can determine who to notify, how to track issues and outcomes and how to prioritize remediation. Popular tools include PagerDuty, which offers an on-call management platform with add-ons for analytics, event intelligence and automated incident response. There’s also ServiceNow, which utilizes automated workflows for ITSM as well as customer service and business processes. You can send your alerts into Slack to consolidate alerts into the same platform you use for collaborating with your team.

    Metrics Storage

    Once you’ve automated configuration management, alerting and monitoring, you’ll have a whole lot of data at your disposal to learn from. The challenge: How do you securely store and analyze it? You need a storage system that lets you aggregate and learn from system capacity, user behavior, service levels, security risks and more.

    The insights you gain from your metrics inform decisions across all layers of your business, improving your ability to meet SLAs, satisfy customer expectations and make the case for new strategic investments. Data-driven decisions promote a culture of continuous learning and improvement.

    Popular tools for storing metrics includes InfluxDB and TimescaleDB, time-series databases (TSDBs) that are well suited for long-term storage, and Splunk, which uses a search engine database model to store and query your data. Amazon Web Services (AWS) supports a wide range of storage purposes, including relational and non-relational databases, a data warehouse for analytics, a TSDB, a leger database to store transactions and more.

    Visualization Tools

    A visualization tool might be considered the pièce de résistance of your DevOps toolchain for monitoring: You get to combine all of your data, sort and visualize it, and display it on customizable dashboards.

    Visualization tools provide context and meaning, allow you to track changes and improvements over time and give management a real-time view that helps guide strategic decisions. Customization options make it easy for team members to design and share their own dashboards.

    Grafana is a popular open source visualization tool and can be used on top of a variety of different data stores, including Graphite, InfluxDB and Elasticsearch. For more on how Grafana can integrate with your monitoring and TSDB, check out this use case.

    Evaluating Your DevOps Tools

    No matter where you are in your DevOps journey, it’s wise to re-evaluate the tools you’re using and identify where you can fine tune. Think about the DevOps tools in the monitoring ecosystem as more than their capabilities. How you use them begins to define your habits, values and work culture—and accordingly, the quality of your product or service and the value you bring to your users.

    — Sean Porter

  • How Monitoring and Observability Feed Each Other to Create a Holistic Approach to Visibility

    How Monitoring and Observability Feed Each Other to Create a Holistic Approach to Visibility

    Monitoring versus observability is a hotly debated topic. It’s been argued they’re two distinct things—the former just a high-level overview of a problem after the fact, while the latter enables you to be proactive. Observability has also been dismissed as jargon, much how DevOps sounds to the seasoned operator.

    The thing is, the whole debate misses the point: it’s not monitoring versus observability–the two are not mutually exclusive, but symbiotic. Like the Ouroboros—the ancient Egyptian symbol of a snake eating itself—monitoring and observability both feed each other.

    Monitoring tells you what’s broken and where you need to have a higher degree of visibility. In other words, if something is broken and you’re having difficulty telling why, you need to increase visibility (i.e., by adding more checks and instrumentation). The same can be said for observability—a term used to describe telemetry, event logging and trace data (and, it can be argued, in many ways a subcategory of the traditional sense of performance monitoring). Observability gives you the richest context, but the difficulty is in understanding the holistic view—AKA, where to look next.

    Say your observability tool shows 99th percentile latency and you see a huge uptick: you have to guess as to why there’s such an increase. Based on your own operator knowledge, it’s either the network introducing the latency or the database is unable to respond in a reasonable or consistent time—for example, the disk could be slow or you’re on a multi-tenant system and someone is consuming CPU time.

    The difference with monitoring in this scenario is it would let you know you exceeded your trend on latency, therefore something has gone awry. It’s the same path, but you still have to make logical jumps based on your own experience—unless you’ve gone ahead and contextualized these from the get-go by creating runbooks and checklists for certain failure scenarios. If a monitoring check is properly created it will provide guidance as to what to look at first, then the observability tool gives you the much lower-level insight to go through each of those prescribed steps. Another way to look at it is monitoring as step one and observability as step two. This way, you can see how monitoring and observability feed each other, rather than existing as wholly separate concepts or toolsets.

    It all comes down to context. Traditional tools for monitoring and observability encompass everything from traditional logging to telemetry data, but none of that matters unless you have context as to why a certain behavior is happening. With this traditional approach, your users become your monitoring–tweeting at you when your site is down—and that’s bad for business. You’re still going to have to guess the context, spelunking through a multitude of data points to figure out what went wrong. It’s a polarizing topic—perhaps not quite as polarizing as monitoring versus observability—but there is such a thing as too much data. My experience has taught me a curated approach to monitoring and observability makes better use of resources and helps combat alert fatigue.

    Context is key. Monitoring is the “what is down?” and the context that helps lead you to determine your “why.” Observability tooling lets you dig further into those whys. A modern solution needs to go beyond just alerts to offer rich context and actionable insight.

    Pairing monitoring best practices with an effective observability solution can both tell you something is wrong and offer rich context. You get to the why faster, making it easier to arrive at the right conclusions in a timely fashion. One approach that marries the concepts of monitoring and observability is to apply workflow automation principles to monitoring. Operators need to have the tools to automate their monitoring workflows (such as auto-remediation), and reserve alerts for complex tasks that can’t be automated.

    By laying aside the debate of monitoring or observability and acknowledging the ways in which they feed each other, operators can have a holistic view and deep context into their systems—ultimately improving their workflows and offering better end-user experiences.

    — Sean Porter

  • Challenges of Big Infrastructure Monitoring

    Challenges of Big Infrastructure Monitoring

    The growing complexity of today’s IT reality is indisputable: Software is becoming more complex, and the infrastructure it runs on is equally so. This complexity comes, in part, from adding layers upon layers to the infrastructure we manage—from bare metal servers to VMs to containers to function-based computing—and from how quickly we cycle in new technologies and how slowly old technologies are cycled out.

    Ephemeral and multi-generational infrastructure (often side by side) are the new normal. Old and new technologies co-mingle, and while new technologies may be shiny and exciting, they quickly become ordinary as the next new thing comes along.

    Here’s a sobering stat: In Q4 2017, IBM’s mainframe business grew by 32%. So even while cloud, AI and security initiatives are on the rise at IBM, mainframes sustain a bulk of their business. Old tech perseveres, and we have to find new solutions to bridge the gap between old and new.

    The hodgepodge of technologies—and the challenges they bring—are compounded further when we’re talking about big infrastructure.

    What Qualifies as Big Infrastructure?

    For the purpose of this discussion, big infra is an organization that has been around for at least 10 years. That means they’ve had success, they’ve grown and their infrastructure has grown along with them, and the technologies at play have expanded as well.

    Your organization’s exact age, size and revenue aren’t particularly material; what matters is you’ve had enough time to accumulate resources, people, practices and technologies to create what I think of as a trail mix problem—basically enough legacy infrastructure to start creating problems.

    The Challenges of Big Infrastructure

    The challenges of big infra stem from the technology itself and from the way people use that technology. As an organization grows, its teams often become larger and more siloed, and departmental goals become disconnected. Even when working toward the same goal, departments diverge in their methods for reaching that goal.

    Now, add tools and technology to the mix: Over time, teams choose tools they prefer, or build their own, which works out okay for a while until they have to collaborate with another department that uses different tools and a completely different set of practices.

    Now, add in different work styles and personality types and company politics, and everything gets tangled up fairly quickly. Trying to keep track of who’s using what, how they’re using it and what’s working (or not) is no easy task.

    One customer, for example, had one team using Chef and was all in, and were trying to get buy in from other teams as well, but there was another team that was early days Docker, hand rolling Docker hosts, and didn’t want to adopt Chef. Then there was a different team who was early days Kubernetes and didn’t want to use Docker, and yet another team using Puppet. Once they had to work with each other, things broke down—someone has to win out. You almost have to maintain a logical map of what team uses what tools as a precursor to prepare for a meeting with these business units. Not only is there no consistency in how applications are deployed, there’s also no consistency in how these different technologies are monitored.

    For example, in the past, Netflix stated monitoring accounts for 30% of their infrastructure cost, and recently mentioned they’re planning to spend $1.9 billion on infrastructure in 2019. Think about the scale and complexity involved—they’re storing and analyzing the logs, behavior and metrics for systems supporting some 125 million subscribers.

    Connecting Disparate Data Across Big Infra

    So, how can you streamline and connect all of the data spread across fragmented tools and systems?

    One solution is a data lake, a centralized repository that lets you funnel data into one place, so you have a holistic view of your infrastructure. In reality, big infra ends up spawning multiple data lakes that become siloed from one another.

    You also likely have a square peg/round hole situation—data in one format needs to be transformed to another format before it can get into one of the data lakes.

    This is where a monitoring tool becomes essential. Look for a monitoring tool designed to collect data in disparate data types and transform it into any number of formats, so you can set up an automated workflow to get data flowing from legacy systems (e.g., mainframes) alongside modern infrastructure like containers.

    An example of this idea is with Sensu, and our support of the Nagios plugin spec and Prometheus exporter metrics (amongst many other metric formats). By using a monitoring tool that works with various metric formats, companies can plug the service checks they’re already using plus data from multi-cloud infrastructure into their monitoring setup and get valuable information about their systems.

    A visualization tool such as Grafana is the icing on the cake. With Grafana, you can hook up data wherever it’s stored (e.g., Graphite, InfluxDB, Prometheus, etc.), visualize the data and display it on shareable graphs and dashboards. Using a monitoring tool as a data source for Grafana means companies can display monitoring incidents and inventory—and even mix that data with different data sources on the same Grafana dashboard. Combining a holistic monitoring solution with Grafana’s visualization capabilities gives a data-driven view of your infrastructure that you need—even if it remains a multi-generational, multi-siloed, multi-data-laked, 2,000-different-format-typed trail mix of big infra.

    — Sean Porter