devopsy.kinsta.cloud

Tag: observability

Report: More Appreciate the Value of Observability

Observability may be a core tenet of any best DevOps practice, but the number of organizations that actually engage in it is still not nearly as high as it should.

A survey of 400 software engineers published this week by Honeycomb, a provider of an observability platform, finds nearly half said they are not currently practicing observability. However, the number of organizations that expect to embrace it is growing. The Honeycomb survey finds 75% of those respondents who haven’t implemented an observability plan to do so at an advanced level within the next two years.

Conducted by ClearPath Strategies, the survey also finds that about a third of respondents are still evolving their observability capabilities. Only 10% of survey respondents exhibited what the survey describes as highly advanced capabilities.

Liz Fong-Jones, principal developer advocate for Honeycomb, there is some correlation between the number of developers an organization employs and their reliance on observability. Among organizations that have advanced observability capabilities, 42% have more than 100 developers.

In general, Fong-Jones said interest in observability is on the rise because it is becoming much easier to instrument applications. The emergence of open source agent software has made it much easier for observability platforms to collect relevant metrics that go way beyond simply monitoring an IT environment by sampling application traffic and analyzing logs, she noted. Observability platforms, in contrast, make use of technologies such as dynamic tracing to provide deeper insights into the root cause of an application issue.

That approach also ties more closely to a specific user benefit versus simply collecting metrics that don’t surface any actionable intelligence, she added.

Fong-Jones noted that observability platforms are now also playing a key role in eliminating technical debt. As organizations embrace microservices, for example, it’s becoming more challenging to identify bottlenecks and potentially redundant microservices. Observability platforms make it easier for DevOps teams to uncover these issues at a time when applications are becoming more complex, she said.

There’s no doubt reliance of observability platforms will continue to substantially increase as organizations dive deeper into best DevOps practices. Most applications deployed by most enterprise IT organizations are not that deeply instrumented. As instrumentation becomes more pervasive, IT organizations will be able to drive more value from observability platforms. That could be crucial because it’s not uncommon for IT teams to spend weeks trying to find the source of an issue that might only take a few minutes to fix.

Observability may not always be top of mind for IT teams but within the context of a best DevOps practice, it’s crucial. Many organizations are clearly racing to deploy applications without being able to observe how application code is behaving. In the absence of observability, it’s a foregone conclusion that tensions between developers and IT operations teams will rise. The challenge and the opportunity are to get everyone involved in IT on the same page in a way that ensures everyone involved is talking about the same thing at the same time.

May 7, 2020
New Relic Moves to Democratize Distributed Tracing

New Relic has launched a service that leverages cloud resources to make analytics based on distributed tracing more accessible to DevOps teams.

Andrew Tunall, general manager for New Relic serverless and emerging cloud services, said the New Relic Edge with Infinite Tracing service democratizes access to distributed tracing analytics by providing access via a software-as-a-service (SaaS) application that is managed entirely by New Relic. Rather than having to set up a platform to analyze distributed traces, Tunall said the New Relic Edge with Infinite Tracing service provides access to observability analytics for up to eight days.

The service observes all application traces across distributed systems in a way that makes it easier to identify actionable issues, said Tunall. The amount of telemetry that’s emitted across a distributed system can easily involve tens of millions or even hundreds of millions of spans per minute. By keeping distributed tracing data on the public cloud, New Relic is also minimizing the amount of egress data from the cloud, which he noted serves to significantly reduce potential cloud computing costs given the fees cloud service providers charge for moving data off their platforms.

Tunall said it’s become more feasible to employ distributed tracing rather than relying on sampling because modern applications are more highly instrumented. DevOps teams have realized access to telemetry data is at the core of any effort to enable observability within the context of best DevOps practices. The challenge has been that setting up the tools to achieve that observability goal has required the expertise of the equivalent of a site reliability engineer.

The New Relic Edge with Infinite Tracing service, in contrast, provides easier access to observability tools based on distributed tracing to any member of the DevOps team, he said. That approach means for the first time in many cases, developers and members of the IT operations team will have access to a common set of tools for at least discovering the root cause of an application issue. In fact, the existence of such tools might go a long way toward fostering increased adoption of best DevOps practices.

It’s not quite clear to what degree the rise of distributed tracing will impact the use of sampling to monitor IT platforms. In most cases, sampling is relied on to identify a potential issue, while distributed tracing provides the means to drill down to the point where the root cause of any issue is readily identifiable. However, as open source agents for capturing application metrics become more widely available and the cost of distributed tracing continues to decline, reliance on traditional sampling techniques might decline. At this stage, however, many organizations have not fully appreciated the difference between being able to completely observe an application and simply monitoring it, so it may be a while before distributed tracing is applied pervasively.

In the meantime, at a time when applications are becoming more complex, the tools needed to manage them are, fortunately, becoming more accessible.

May 5, 2020
Uncover the Secrets in Your Apps Through Observability
When it comes to your applications, the question to ask yourself is: “How helpful is monitoring if you aren’t proactively capturing the data you need to understand what is causing the errors you find?” The answer: Not very. That is why it is important for us to go beyond monitoring and work on expanding our capabilities in terms of observability.

When applied to the world of software, observability is how well you can understand what’s going on inside your application based on accessible outputs. Looking at this from outside the realm of errors and application failures, etc., observability might describe how well a developer understands how a certain feature works the way it does.

This example clearly shows that the metrics and context needed to attain what we can call a reasonable level of observability depends on the requirements of the individual system. We wouldn’t necessarily expect a developer at one company to understand how the features of another company’s application work behind the scenes. Likewise, an e-commerce application and a healthcare system will require different metrics to be observable.

Bringing Observability to the Team

Every company that operates software has at least the most basic level of observability thanks to log files. Unfortunately, in most cases we don’t know ahead of time what’s going to break, meaning no data gets logged. Occasionally, we may have a hunch that a certain method will be more likely to fail but even then, the data that’s written to the logs is usually shallow and requires additional investigation and context.

A key measure of observability is how well you can answer “why this happened” without needing to ask additional questions. Teams that rely heavily on log files to understand and troubleshoot issues generally have low observability, as we all know that the logs often lead to hours (if not days) of follow-up questions. To improve observability, it’s important to focus on proactive solutions.

Organizations, especially at the enterprise-level, use much more than log files to collect data around application errors and slowdowns. Application Performance Monitoring (APM) tools also play a significant part in most teams’ tooling stacks. Traditional APM tools provide additional context by helping teams identify when and where their application is experiencing performance or availability issues.

These tools, and more, are commonly used by companies to monitor application behavior and to investigate the root cause of issues. Each tool that provides additional context into the internal function of the application increases our observability. Once your tool stacks provide the full context needed to understand the root cause of any issue, you’ve reached a reasonable level of observability.

Creating a Culture that Supports Observability

Just like with every other buzzword in the industry–CI/CD, DevOps, etc.–achieving a reasonable level of observability means cultivating a culture that values and supports such a goal. There are several layers that need to be addressed and integrated together, but let’s break it down.

The first step is to create a quality-focused, team-wide engineering mindset toward system hygiene. From beginning to end all members of the team should be trained to build and employ code with quality, first and foremost. The mindset of engineers must go beyond the desire to write high-level code, but also involve the planning process and understanding the purpose behind code changes. It is using foresight to plan for the unpredictable as much as we can.

We all know that releasing code to production is basically “sending it out into the wild,” and we can all do a better job of making sure we have all the equipment we need to understand its behavior. That includes our mindset, as it forces us to consider our goal much earlier in development, namely by answering the following questions:
- What are we trying to improve?
- How can we measure success or failure?
- What metrics do we need to measure this?
Once this has been established, radical transparency and open communication follow naturally and further enable improvements in this area. When standards are established and issues are clearly communicated, problem areas in development can more easily be identified and resolved.

Without the human aspect in place, no tool in the world can give an organization a reasonable level of observability.

Conclusion

Observability is about more than just monitoring your system–it’s about understanding it. In order to achieve a higher level of observability, tooling and culture are two major factors that should be addressed. There are many different ways to attain the best level of observability for your organization, so understanding what you need in order to be proactive is crucial.

— Chen Harel
April 10, 2020
Threading the Needle on Kubernetes Complexity with AI-Powered Observability

If you run down the laundry list of major industries–transportation, retail, healthcare, automotive, financial services–you’ll see that the major players in each have essentially become software companies. What they do and how they serve their customers rests entirely on their software working perfectly. Underpinning this is a massive digital transformation that has shifted organizations away from the old, static ways of enterprise IT monoliths. Instead, today’s businesses are working in more dynamic IT environments–agile, multi-cloud and, especially for new digital services, based on containers and microservices. All of which means embracing more IT complexity than ever before.

Kubernetes Brings More Data to the Table, but that Impacts Observability

A key part of this more complex environment is the soaring adoption rate of Kubernetes over the last few years. In a survey of 5,000 enterprises, 40% said they were currently running Kubernetes in production and over 80% reported that they’d adopted Kubernetes over any other container management system. Another recent survey revealed that 68% of CIOs are already using containers and a total 86% expect to deploy them over the next year. Enterprise IT is increasingly gravitating to containers and especially to Kubernetes for orchestrating their containerized applications.

What drives organizations toward containers and Kubernetes is the flexibility given to development teams to release services, features or apps faster. But faster doesn’t necessarily mean better or higher quality, unless you also apply cloud-native architectural patterns, include quality gates into your delivery pipeline, embrace new deployment models such as canary or feature toggling and provide full visibility and short feedback loops from production back to engineering.

Kubernetes itself provides great visibility into node or pod health. Kubernetes even ensures the health of your pods through its built-in health check mechanisms. Thanks to components such as service meshes, you also get live information about dependencies between services running in your pods. This data can, in theory, provide organizations with the necessary insights into their environments to understand whether their organization is releasing better value faster to their end users and unlocking new business value.

The reason I said these are theoretical benefits is because, in practice, all that extra data doesn’t necessarily mean more visibility. Often it can mean less visibility–and with that, less insight into how the different elements of your environment may be impacting speed of production deployments, overall system performance and the end-user experiences. I’ve seen Kubernetes environments that are packed with literally billions of interdependencies between applications, containers, microservices and clouds. Asking human beings to keep track of that level of data manually is an impossible task now. And relying on traditional, manual approaches to manage and monitor IT environments shortchanges the value that organizations should be getting out of Kubernetes.

Kubernetes Environments Need Full Stack Observability—that Means Needing Deterministic AI

Getting the full potential out of Kubernetes means having observability across the full technology stack and those billions of interdependencies. But as more organizations are discovering, that can’t happen without AI. Specifically, a deterministic AI engine. In fact, in a 2019 survey, 88% of CIOs said they were looking to AI as a critical solution for overcoming IT complexity. As long as Kubernetes is fueling complexity, overcoming that complexity means making deterministic AI an integral part of any Kubernetes environment.

A deterministic AI applies specific algorithms to detect system anomalies and apply its collected dependency information toward detecting the root causes behind those anomalies. When that deterministic AI engine ingests critical Kubernetes events, such as state changes and workload changes, it provides visibility into the Kubernetes stack’s interdependencies. That AI-powered, full stack observability matters because it gives IT new capabilities for their environments, including instrumenting and mapping containers, managing and allocating container resources and drilling down into container runtimes at work. All of that is possible due to the level of observability that you can only get with deterministic AI; traditional metrics and dashboards just can’t compete.

By providing full stack visibility and interdependency mapping across a Kubernetes environment, deterministic AI is empowering IT teams to identify, in real time, the root cause of degradations in business service availability–from reliability and resource usage to system performance and user experience–faster than they ever could manually. That level of automated speed and visibility also makes it possible to immediately resolve those issues at the source before they can impact the end user. The automation provided by deterministic AI also works its way into DevOps’ production pipelines, accelerating the release and feedback cycles of new deployments and allowing teams to more quickly incorporate user feedback into the next iteration for even better experiences.

In short, a deterministic AI engine makes life better for both IT and the customers they’re aiming to serve. Deterministic AI takes work that used to be time-consuming, tedious and counterintuitive when done manually, and performs it fast and automatically. That then frees up the teams that used to do that to focus on work that is more satisfying and drives new business value.

AI-Powered Observability Relies on Quality Data and Quality Monitoring

All that said, a deterministic AI’s ability to provide reliable, precise full stack observability in a Kubernetes environment relies on it being given consistently quality data it can process. The algorithms are dependent on the data that’s ingested by the AI engine. If there’s no standard monitoring capability being used across the board, then you invite a scenario where different teams bring their own preferred monitoring tools into the fold. The bigger a Kubernetes cluster gets, the more diverse the set of monitoring tools can be. And the greater the diversity of monitoring tools, the more inconsistent the data being fed into the deterministic AI engine.

The way out of this predicament is to leverage a single monitoring tool as a self-service capability into Kubernetes clusters, one that can support the diversity of technologies that are introduced into the stack and provide a baseline standard of system and event data that developers, IT administrators and the deterministic AI all lean on. In other words, when you feed consistent, quality data into the AI, you make it smarter. And a smarter deterministic AI engine provides the full stack Kubernetes observability that organizations need.

Building a Positive Feedback Loop Between Kubernetes, Deterministic AI and Observability

Kubernetes and deterministic AI are a natural fit. Kubernetes provides a platform for managing and instrumenting containerized applications. The deterministic AI provides a tool for mapping out the Kubernetes stack, offering insights into interdependencies and degradations where they arise and resolving them before they can affect end users. The data from Kubernetes makes AI better, and a smarter deterministic AI engine means more precise observability across your technology stack.

It’s a self-reinforcing feedback loop that eliminates the usual pitfalls of container-induced IT complexity. Instead, it creates new standards of visible, consistent and quality data that organizations can leverage for faster and better deployments–and in turn, better user experiences and greater business value.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or August 2020.

— Andreas Grabner

March 23, 2020
Observability-Driven Development: From Software Development to DevOps and Beyond
In today’s world of cloud platforms and applications, typically delivered as SaaS (Software as a Service), it is critical for organizations to ensure efficient collaboration between software development and operations teams. The software development team codes the software application or platform, while the DevOps team streamlines and automates the entire production lifecycle, from development and testing to deployment.

Today, applications are updated anywhere from weekly to daily, requiring fast turnaround. In the old days, every change request would go to the different stages–software design, development, testing and release. This can’t be sustained in an agile environment where updates must be as close to instantaneous as possible.

How can organizations achieve this, and where does observability fit into it all?

What Is Observability and ODD?

Observability-driven development (ODD) adds another layer to software development by encouraging the development team to think about the application availability and uptime throughout their development process and similar to unit-testing development, wrap their code with more verbose logging, metrics and KPIs. This gives the IT operations team more data on the application and improves overall observability, allowing an organization to detect predictable and unpredictable permutations of failures that may occur in the future. ODD observes the behavior of a system throughout the entire development cycle in order to learn what behavior is normal and abnormal, and to detect potential weaknesses.

Observability provides everything with a real-time status of a product–this includes software developers, DevOps and any other department that requires it. Each role has different needs. Software developers need to know why a certain feature has to be added or developed in order to understand its business value and prioritize it over other features or updates. If a certain feature is likely to require many future updates, this also needs to be taken into consideration in the development phase so that the code can be adjusted easily without negatively affecting DevOps teams when they test and deploy the feature.

These are just examples of things that must be taken into consideration during the development process. Organizations must be able to create transparency and effective communication between the different departments and roles.

Inefficient Collaboration in the SaaS Market

DevOps need to ensure zero downtime, fast updates, constant monitoring and more. This can be done by maintaining an operational mindset during the development process. Traditionally, developers write the code, QA do the testing and operations take care of deployment, but there needs to be more collaboration between these three stages with each doing their part with an understanding of everyone else’s roles and needs pertaining to the product. When this isn’t done, numerous problems arise. Operational inefficiencies can ultimately lead to a delay in delivery time and even affect system availability. When there are multiple teams involved, especially in different locations and even time zones, there can be ineffective communication and hand-offs, particularly between software development and operations teams.

Meanwhile, ad hoc changes must be implemented, negatively affecting product stability and release cycles. Miscommunication and inefficient operations lead to confusion over who is responsible for what task, feature and target. These are all issues that can result in significant downtime, losses, wasted resources and frustration within the organization. Effectively streamlining the software development cycle using full transparency and seamless collaboration between development and operations solves these problems.

ODD Can Improve Your SaaS Product

A strong proactive observability platform can predict and address issues before they occur, thus increasing effectiveness and speed when it comes to updating and tracking changes, as well as releasing new features. It simplifies complexities and management, and allows to automate operations so that organizations can maximize efficiency.

Benefits of ODD include:
- Common goals.
- Shared metrics.
- Execute on goals and recognize results.
- Performance guidelines.
- Application governance strategy.
- Alignment between IT and business perspectives.
Employing ODD when developing cloud applications is critical. This is due to the highly distributed nature of cloud architectures, increasing the chances of a failure exponentially compared to traditional systems. As a result, developers should be educated to think and implement observability to create highly available and resilient applications and systems.

How to Integrate ODD in Your R&D Practices

Today, development teams and practices are under constant pressure to meet scrum deadlines and release features on time. They are measured on their code effectiveness and meeting deadlines, while ODD adds another layer of measurement of serviceability and observability metrics to their considerations.

Implementing such an approach requires organizations to provide their development teams with the right toolset and knowledge on how their code affects the SRE and IT operations teams. From a technical standpoint, development teams should be familiar with all levels of monitoring for application infrastructure (for example Kafka, RabbitMQ, Kubernetes), application performance metrics (user experience, transactions per second, etc.) visualizations and their respective remediation processes.

ODD culture changes the way organizations think about their development process, and essentially injects an operational mindset to their daily practices and processes which should increase the availability and performance of their cloud application while streamlining collaboration and work processes, and improving teams’ satisfaction and productivity.

— Avi Shalisman
February 20, 2020
Cloud Native Tracing and Observability: Why You Care

Is observability the new monitoring? Or is observability, and tracing, fundamentally different? Like any IT industry trend, it can be difficult to discern as many jump on the trend bandwagon, appropriately or not. The Splunk .conf 2019 event presented Splunk with the opportunity to bring clarity to the terms and the reasoning behind the acquisitions of SignalFx and Omnition.

Monitoring is mostly about taking events, telemetry data and established data points and thresholds into monitoring, alerting and problem management processes and tools. Its tried and true, and has progressed with the evolution of typical monolith, distributed apps and systems. Enter cloud-native applications, composed containers, microservices and service meshes. With great flexibility comes complexity, and cloud native isn’t immune to such an adage.

What would have been a monolithic application is now shattered into hundreds or even thousands of smaller pieces of application and app technology software. The complexity of externally monitoring cloud native quickly surpasses the capabilities of most monitoring approaches.

Observability, as recently acquired SignalFx CTO Arijit Mukherji shared this week, is built upon three pillars: metrics, tracing and logs. Metrics show when you have a problem, tracing points you to the problem and logs help find the root cause—a reasonable way to define and segment observability. It also, not surprisingly, fits well with the logic of Splunk plus SignalFx plus Omnition.

To move beyond monitoring requires instrumentation, built into the software and APIs as part of the software creation process, a DevOps process. SignalFx in part brought Splunk auto-instrumentation, which during run-time, identifies frameworks and libraries in use within applications and can capture tracing instrumentation. Omnition brings even deeper tracing chops to perform tracing across large service meshes of microservices. Add Splunk’s capability to correlate data across the business, applications and operations data and you complete the picture with the ultimate goal of making observability easy for developers.

The above might explain why developers, ops and DevOps care about observability and tracing, but should the business care? SignalFx CEO Karthik Rau connected the dots nicely during a conversation this week. Digital transformation strategies require speed and agility, but also demand more risk-taking. Confidence in risk-taking comes when accompanied by the ability to respond rapidly to changes and failures. A software deployed 10 minutes ago may need to be backed out or corrected rapidly when the users’ experience goes negative. That requires a rapid determination of what the problems is, and the ability to take immediate corrective action, including automated action.

Cloud native and DevOps not only enable disruptive, digital transformation strategies but must be accompanied by rapid and automated responses when negative business and customer impacting conditions occur. Customers don’t care when CPUs are taxed, but they do care when a mobile app’s responses fail or are slow. The move to cloud native is served well when backed up by such capabilities to respond in near real-time when problems occur.

— Mitchell Ashley

October 29, 2019
Four Steps to Implement an Observability Strategy for Microservices

You’ve finally decided. You’re making the move to microservices and re-architecting your entire infrastructure. Some services will still be on-premises (those databases and their legacy apps are never moving), but a cloud-based microservices architecture will allow your teams to build and bring new products to market faster.

But how do you handle the complexity that comes with cloud and microservices? You know you need a strategy for monitoring and managing the complex dependencies and ephemeral nature of your new architecture, but you don’t know where to start. What’s more, you don’t want to spend time and money now only to realize a few months down the line that you’re on the wrong road. Your investment needs to be future-proof.

You need to implement a strategy for observability. Observability goes beyond monitoring to give you greater operational insight, leading to shorter incidents, fewer quality problems, a better product and happier customers. Your company has unique observability needs, which means how you approach your observability strategy will also be unique. As is usually the case, it’s important to start with a roadmap and goals. Based on my experience, the following four things are key to making a considered, comprehensive observability strategy that’s built for the long haul.

Choose an Observability Platform that Fits Your Future

Perhaps counter-intuitively, the right platform isn’t necessarily the one that’s the right fit for you today. If you’re investing in new technologies such as microservices, containers and Kubernetes, you’re going to see a significant increase in the scale and ephemerality of your environment. A monitoring platform that may suit your current infrastructure likely won’t be able to keep up and meet your future needs. Instead, consider how your infrastructure will look three, four or even five years down the road, and then select the platform that best fits that long-term strategy. It doesn’t matter if you choose a commercial tool or an open source one: What matters is that you give considered thought to your future infrastructure and choose a platform that can address the future needs of your environment—and protects your investment at the same time.

Monitor the Metrics that Really Matter

If you’re still in the early days of your shift from monitoring to observability, you’re likely taking a service-based approach, monitoring specific pieces of your ecosystem with a focus on individual incidents. The real value of observability is the ability to provide visibility into big-picture issues, such as measuring user experience, performance against SLOs and other key metrics important to the business. In modern environments, small upticks in a microservice can cascade into increased latency for an entire product, a specific feature, or even a specific customer. Having an observability platform that measures and collects data across multiple dimensions–e.g. customer/business-specific, application/service specific and infra/host/container specific–will allow you to understand the telemetry in many different ways. The platform should also aggregate the interactions between your services and unlock a wealth of data for engineers to investigate when issues do arise, and provides visibility into the metrics the business cares about most

Embed Observability into Your Incident Management Process

Incident management is a hot topic right now. I speak with CTOs across organizations and industries who have fantastic tools that provide detailed telemetry, but everything breaks down when an incident occurs. They need to better embed observability as a part of their incident management process. For example, ask your incident management team how observability helped in a given incident and record those results. This keeps track of the ongoing benefit. Next, ask how observability could be improved to get the team closer to the crucial clues they need during an incident. And finally, make sure to fold these items into your remediation plans so the team has a clear playbook when incidents do occur.

Establish a Culture of Observability Across Your Organization

Simply deploying a new technology of any type is not enough. You need to ensure you’ve built the appropriate processes and norms around the new platform to ensure its success—this is especially true with observability. Modern observability platforms can be incredibly powerful in providing insight into the performance of your ecosystem, but without an investment in your organization’s ability to use the platform, you’ll barely scratch the surface of its true value.

Observability requires ongoing investment and improvement. Start by building a core group of champions, then fold training into onboarding, continuing education or brown-bag sessions. Using subject matter experts to create the content is a good way to get everyone off on the right foot. Use the output from observability tools in company meetings, team syncs or support settings to keep everyone on the same page. This sort of standardization raises the collective skill of the entire organization and demonstrates the value of observability to those who are not already on board.

Following these four steps will set you off in the right direction on your observability journey. If your organization is like most, you keep an eye on what you think could go wrong. While that’s unquestionably important, a mature observability strategy will give you an insight into previous unknowns and help you more quickly understand why incidents occur. And as you continue on your observability journey and your understanding of what and why things break improves, you’ll be able to implement increasingly automated and effective performance improvements that impact your company’s bottom line.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.

— Arijit Mukherji

October 28, 2019
How Monitoring and Observability Feed Each Other to Create a Holistic Approach to Visibility

Monitoring versus observability is a hotly debated topic. It’s been argued they’re two distinct things—the former just a high-level overview of a problem after the fact, while the latter enables you to be proactive. Observability has also been dismissed as jargon, much how DevOps sounds to the seasoned operator.

The thing is, the whole debate misses the point: it’s not monitoring versus observability–the two are not mutually exclusive, but symbiotic. Like the Ouroboros—the ancient Egyptian symbol of a snake eating itself—monitoring and observability both feed each other.

Monitoring tells you what’s broken and where you need to have a higher degree of visibility. In other words, if something is broken and you’re having difficulty telling why, you need to increase visibility (i.e., by adding more checks and instrumentation). The same can be said for observability—a term used to describe telemetry, event logging and trace data (and, it can be argued, in many ways a subcategory of the traditional sense of performance monitoring). Observability gives you the richest context, but the difficulty is in understanding the holistic view—AKA, where to look next.

Say your observability tool shows 99th percentile latency and you see a huge uptick: you have to guess as to why there’s such an increase. Based on your own operator knowledge, it’s either the network introducing the latency or the database is unable to respond in a reasonable or consistent time—for example, the disk could be slow or you’re on a multi-tenant system and someone is consuming CPU time.

The difference with monitoring in this scenario is it would let you know you exceeded your trend on latency, therefore something has gone awry. It’s the same path, but you still have to make logical jumps based on your own experience—unless you’ve gone ahead and contextualized these from the get-go by creating runbooks and checklists for certain failure scenarios. If a monitoring check is properly created it will provide guidance as to what to look at first, then the observability tool gives you the much lower-level insight to go through each of those prescribed steps. Another way to look at it is monitoring as step one and observability as step two. This way, you can see how monitoring and observability feed each other, rather than existing as wholly separate concepts or toolsets.

It all comes down to context. Traditional tools for monitoring and observability encompass everything from traditional logging to telemetry data, but none of that matters unless you have context as to why a certain behavior is happening. With this traditional approach, your users become your monitoring–tweeting at you when your site is down—and that’s bad for business. You’re still going to have to guess the context, spelunking through a multitude of data points to figure out what went wrong. It’s a polarizing topic—perhaps not quite as polarizing as monitoring versus observability—but there is such a thing as too much data. My experience has taught me a curated approach to monitoring and observability makes better use of resources and helps combat alert fatigue.

Context is key. Monitoring is the “what is down?” and the context that helps lead you to determine your “why.” Observability tooling lets you dig further into those whys. A modern solution needs to go beyond just alerts to offer rich context and actionable insight.

Pairing monitoring best practices with an effective observability solution can both tell you something is wrong and offer rich context. You get to the why faster, making it easier to arrive at the right conclusions in a timely fashion. One approach that marries the concepts of monitoring and observability is to apply workflow automation principles to monitoring. Operators need to have the tools to automate their monitoring workflows (such as auto-remediation), and reserve alerts for complex tasks that can’t be automated.

By laying aside the debate of monitoring or observability and acknowledging the ways in which they feed each other, operators can have a holistic view and deep context into their systems—ultimately improving their workflows and offering better end-user experiences.

— Sean Porter

October 24, 2019
Keeping NoOps From Going Rogue

“We’re NoOps—everything is in the cloud and we have no infrastructure to manage.” Before running that victory lap and shifting everyone to the dev side, consider several NoOps risks carefully. I’m really thinking about teams using all-SaaS stacks, especially in martech, edtech and other areas. However, the rise in microservices and serverless means more teams are leaning toward NoOps. At least a few people on your team should always watch for rogue issues on the ops side of the equation.

Locked-in Vendors

Risk: Cloud-based tools in the same category rarely have the same functionality. Once one is selected, you’re committed to its specific features. Moving away from a tool may mean dropping a popular application feature or changing workflow with coding to recreate functionality.

Mitigation: Evaluate unique features and customized data fields carefully before use. Solving data format differences is relatively easy for common fields. Look for tools with well-documented APIs and integrations or connector support for analytics. If there are no integrations out there, that’s a red flag.

Ghosting

Risk: Lock-in becomes a big problem if the vendor ghosts. SaaS startups often meet one of two fates: they are acquired or they crash and burn. A tool critical to your workflow may disappear overnight. In the acquisition scenario, you may have some time and support in migrating from your tool to its successor.

Mitigation: Always have a Plan B for every cloud-based tool in your stack. Competitors are introducing new tools frequently. A trade study done two years ago may be completely obsolete on today’s fast-moving landscape. Proactive migration to newer solutions may reduce vendor risk and improve features.

Limited Observability

Risk: Overheard in a conference recently—Those who develop the code are more motivated to build in useful observability for those in operations if those folks are the same team. In NoOps, that’s not in play. If a cloud-based tool in your stack is down frequently or providing strange results, it may be very difficult to tell what is happening. Observability is often replaced by faith that the SaaS vendor has done its job and you aren’t experiencing a corner case it hasn’t tested.

Mitigation: Observability and monitoring aren’t the same, but when you don’t have observability inside SaaS tools, monitoring helps. Application performance management (APM) tools are emerging with new ways to build instrumentation around microservices. Create test cases with synthetic data to check your functionality—you’ll need these to provide evidence for the vendor in any bug report.

Unoptimized Workloads

Risk: You’re also often at the mercy of the SaaS vendor when it comes to speed of results. It’s a common tripping point to pilot a SaaS workflow with a few users and test cases, and then find out it doesn’t scale when everyone is brought in.

Mitigation: This is one the biggest NoOps risks where ops expertise pays off. Where is the problem, exactly? Application problems may be candidates for workload acceleration—such as running on a GPU (or FPGA) enabled cloud instance. Reporting problems may be better-suited for an analytics tool, which can be run offline with extracted data.

Tangled Analytics

Risk: Every SaaS tool reports different metrics in different ways. Worse, users in a multi-cloud environment face learning and visiting multiple applications to look at data. Different applications report similar facts using different terms. As the tool stack and data sets grow, it takes longer and longer to analyze data.

Mitigation: Dashboards pay off early with one destination for data across the stack. Earlier, I mentioned the need for integrations or connectors—this should be a requirement for tool selection, both in the stack and the analytics tool. Deciding on which KPIs to track in the dashboard also focuses conversation and minimizes digging.

Closing NoOps Risks

There’s also the big topic of multi-cloud security and compliance, probably worth a series of posts by themselves. An Ops team working proactively in tool selection can comprehend many security risks before committing to tools and exposure. More value unfolds over time, as the security posture evolves with discovery of new threats.

The tech pendulum is always swinging. For many organizations, NoOps swings the pendulum a bit too far from DevOps and leaves risks open. A better philosophy is MinOps, where a small team is deployed, watching for and even anticipating problems.

I’d be interested to hear if you’ve tried NoOps and been successful over time, or if you reverted to a MinOps approach.

— Don Dingee

April 29, 2019
Log Management: What DevOps Teams Need to Know
One of the most important shifts of this decade is the rise of interconnected data across distributed systems. This explosive growth not only sparked the need to rethink cloud and global IT strategies, but it also disrupted traditional development, DevOps and ITOps practices. These changes are pushing requirements for higher-velocity development of features and products—together with more responsive support requests—onto technology professionals. These new requirements have prompted teams to evaluate new tools and agile processes that help support the needs of a competitive modern business.

Regardless of the pace of change within your group, basic functions such as infrastructure monitoring are crucial to operating production applications. At this new speed of business, concepts such as observability are important for enabling successful deployments with fewer user interruptions—not only in production, but also in the early stages of your continuous integration/continuous delivery (CI/CD) pipeline. As part of “shifting left,” log and event management in your CI/CD pipeline empowers developers to monitor and observe application behavior before releasing it to production. While some might say this adds additional layers of process, time, effort and due diligence in dev environments, it can help you evolve your organization’s fundamental software development behaviors while reducing preventable issues from post-deployment into production. It can create a smoother, more seamless user experience and reduce the need to fight fires or rearchitect your solution after the application is live in production and at scale.

Setting the Testing and Dev Stage

The rapidly evolving technology landscape has increased the need for log management and observability across distributed systems and containers. Changes to how applications and services are created, the ability to deploy applications across combinations of global logging-as-a-service (LaaS) providers and the pervasiveness and volatile lifespan of containers, along with the capacity to build services using various development languages, has increased the need to collect, monitor and trace data points across connected systems, which provide critical end user-facing value.

Traditionally, many considered log management a tedious task because each instance of a new server required you to run a search command across the logs local to that server. That approach isn’t scalable when every cluster of more than 20 servers is built up in minutes, and each application has independent logging formats. In this scenario, you’d need to run 20 individual searches across multiple systems, compare timestamps across servers residing in different time zones and so forth. With the decreasing cost and increasing prevalence of containers and virtual systems, the number of systems running within a typical organization has grown exponentially as the business prospers.

In addition to increased usage of virtual systems and containers, independent scrum teams of developers can build in parallel using technologies which are the best fit for the services they are producing, while also sharing environments for builds, staging and production. With these distributed services, aggregated log systems can quickly determine the root cause for debugging their application’s errors, exceptions or performance bottlenecks. A log aggregation system can reduce the element of “personalization” that can often happen when developers are using different technologies with various logging formats, and also reduces the noise by filtering out information that delays the team’s ability to find the source issue. This can cause confusion or slowed productivity when one developer is left to understand another’s non-standardized, yet shared, workload.

The reduced cost, increased collaboration and prevalence of container services are collectively driving an uptick in virtual environments and containers, often requiring more logs to be created in testing and dev environments, which can force organizational shifts to enable full observability across their infrastructure.

The 3 Pillars of Observability

Businesses are in the midst of a shift toward adopting these principles—the three pillars of observability—and using a three-pronged method of monitoring that takes data from several vantage points to depict a more accurate and consumable view of overall health and stability:
1. External Monitoring: An example of this is health checks run against your internal and external applications and websites to see the “digital experience” of your users.
2. Metrics and Distributed Tracing: This enables you to trace communications between applications distributed across systems/containers to identify errors and exceptions from your applications and resolve latency quickly.
3. Events and Logs: Data, which helps provide contextual information about events, enables you to identify issues in the code when combined with information from the first two pillars.
The above types of information are critical to DevOps to provide elastic and resilient services. Bringing application, system and infrastructure events into logs together with external monitoring and metric collection is quickly becoming the standard. With powerful log-parsing features such as live-tail search accessible from a web browser and the ability to visually pair log insights with added health metrics, the real value of log data is made apparent with little effort.

Site reliability engineering (SRE) is another area where significant evolution has occurred. ITOps has always been asked to react to fire and run from fire to fire. The evolution from ITOps to DevOps to SRE is forcing the alignment of different types of data to be used very early in a CI/CD pipeline, and enabling SRE teams to get involved in architectural discussion at project inception instead of in the latter stages or at release. The SRE is more involved in monitoring, understanding the development and architectural principles used to develop application services and build in automation, and high availability and circuit breaker patterns from inception.

This approach has allowed teams to be proactive and responsive rather than reactive. It has equipped our teams (which keep our business running successfully and our customers loving what we do) with the ability to focus on being scalable, elastic and resilient. Instead of troubleshooting systems in production, we auto-scale and replace to minimize disruptions and investigate that incident offline when possible.

Best Practices

Given the increasing pace of application development required to support the needs of the business, the spread of containerized applications and virtual environments and the emergence of new disciplines for observability (due to the lowered costs and increased reliability), log and event management has become a critical aspect for all involved in building, supporting and even using mission-critical applications. It’s moving to a defined area where logs should be compartmentalized, and access control should be defined:
1. Compartmentalizing Logs: Critical as a best practice so we can differentiate development, staging and production environments and segregate logs based on practical grouping. Using a log aggregation service provides you with a consistent experience and a common set of capabilities across all your environments.
2. Access Control: Another best practice that should be implemented more broadly; for example, developers don’t need access to all of your production logs if approximately 95 percent of their work happens prior to reaching production environment. In other words, even though tech pros are working to “de-silo” log management, we still want to silo access control and manage visibility. This can also help to ensure personally identifiable data (PII) isn’t exposed to anyone it shouldn’t be.
Looking Toward the Future of Log Management

As the landscape evolves and best practices become more defined, we should expect revitalized strategies with elevated expectations for the traditional scenarios where basic/free monitoring is leveraged. While the pillars of observability are leveraged more broadly in the ITOps and DevOps communities, we’ll be able to move more efficiently from an end user-facing incident—for example, an application error traced across distributed systems—straight into the logs to find the specific error causing the issue. We’ll be able to see greater correlation between logs and other custom metrics, as well as much broader use of these techniques outside of production environments.

DevOps teams need an intuitive event and log management system that is as de-siloed as we are. No longer can log management systems be relegated just to one on-premises storage system or one virtual environment; they need to fit the reality of how work is increasingly done today and will be in the future.

— Keith Kuchler
October 4, 2018