devopsy.kinsta.cloud

Tag: traces

  • Splunk Adds Data Management Tools to Observability Platform

    Splunk Adds Data Management Tools to Observability Platform

    At its .conf22 event, Splunk today announced it is making it easier to both onboard data and then manage it across hybrid IT environments via the Splunk Cloud Platform.

    In addition, Splunk Enterprise is being extended to add support for Microsoft Azure with SmartStore for Azure to store cold data alongside existing support for Amazon Web Services (AWS) and Google Cloud Platform (GCP). There is also a developer edition of Splunk Cloud that enables developers to create and test their own applications, which is currently in preview.

    Garth Fort, senior vice president and chief product officer for Splunk, said the overall goal is to make it easier to automatically tune the parameters used to store data in the Splunk observability platform. As more organizations embrace observability as a DevOps best practice, many of them are encountering challenges managing the large volumes of data they are now collecting in the form of logs, metrics and traces.

    The Data Manager for Splunk Cloud Platform will initially be made available on AWS and Azure, with support for GCP to come later this summer. In effect, it provides IT teams with a control plane for data management that spans multiple clouds.

    Ingest Actions, meanwhile, makes it possible to apply granular controls that enable IT teams to filter, mask and route data in motion within the Splunk Platform or to external AWS S3 cloud storage services.

    Other capabilities Splunk unveiled at the conference include an expansion of Federated Search tools to enable a single pane of view across their entire Splunk ecosystem and a fully managed Splunk Assist service for Splunk Enterprise 9.0 that provides customers with deep insights into their security environment.

    Finally, the company has enhanced Splunkbase, a framework for accessing curated collections of more than 2,500 applications and integrations provided by Splunk and its third-party partners.

    As the pioneer of IT operations analytics, Splunk has a large footprint within enterprise IT environments. It is now looking to expand that base to become an observability platform provider. It’s still early days as far as the transition to observability is concerned, but the goal is to make it simpler for IT teams to query platforms and surface issues that could lead to an IT disruption before it happens.

    It’s not clear how many organizations have the skills and expertise required to formulate those queries, but over time it should become more feasible to augment an IT staff with machine learning algorithms capable of discovering those issues.

    In the meantime, IT organizations are collecting more data than ever about their IT environments. In theory, that data will provide the context needed to optimize application environments that get more complex with each passing day. The challenge, of course, then becomes finding a way to efficiently store and manage all that data both before and after it has been analyzed.

    Regardless of the approach to observability, the need to go beyond simple monitoring of IT environments has never been more apparent.

  • Opportunities and Challenges of Observability

    Opportunities and Challenges of Observability

    Observability platforms that unify metrics, logs and traces across applications and IT infrastructure are transforming how IT is managed. Most DevOps teams today are able to continuously monitor IT environments using tools that track a set of pre-defined metrics; observability platforms make it possible to aggregate data so that DevOps teams can launch queries that help them uncover the root cause of IT issues. DevOps teams leverage observability to allow them to understand what is happening within their applications more effectively and efficiently.

    Historically, applications were black boxes—meaning it was (and still is, in many cases) extremely difficult to know what is happening under the hood of the application.

    With the advent and subsequent growth of observability tools and techniques in the DevOps space, teams went from struggling to collect and understand the available data (which could vary from robust to almost non-existent) to easily aggregating data from numerous sources and being able to understand what that data means to being completely deluged with data—so much data it’s hard to keep up.

    Gather Data is No Longer Enough

    Josh Chessman, vice president of strategy and innovation at IT infrastructure monitoring specialist Netreo, explained that as applications become more and more complex, (e.g., running across multiple disparate environments with different technologies and in different locations) being able to collect data from these different environments is no longer enough.

    “Instead, DevOps engineers need to be able to collect the data and efficiently analyze it, too, to understand what is happening within the application,” he said. “Observability is the concept that drives this.”

    Chessman pointed out there are numerous observability issues DevOps teams face, but that they vary from organization to organization (and sometimes from team to team within an organization). 

    “Part of the problem is that many organizations simply lack tools to provide the data collection and analysis needed to move to an observability paradigm,” he said.

    This could be for numerous reasons, including budget (if you already have an existing tool, can you afford to acquire a new tool?), functionality (not all tools that claim to provide observability really do), implementation (this stuff is not easy and a poorly implemented tool could actually make things worse) and more.

    “One of the larger challenges I’ve seen is that organizations will go out and acquire an ‘observability’ tool and assume that is all they have to do,” Chessman said. “Unfortunately, that is not the case. Acquiring the tool gets you the technology, but the team must also be structured and ready to take advantage of the concepts and functionality provided by the tool.”

    GitLab’s senior director of product management Kenny Johnston explained that, on average, DevOps teams use five different observability tools which tend to inadequately connect developers to observability.

    In turn, developers struggle to respond to incidents swiftly while jumping across multiple tools and sources of truth and struggle to assign service level objectives (SLOs) and error budget definitions to the responsible product development teams.

    “To remedy this, we recommend adopting a unified software development solution that replaces the DIY ‘duct tape’ of products constraining developers and helps them deliver software more securely, at speed and scale,” he said. 

    Johnston said the most flexible and future-proof solutions today are open source tools like Prometheus and its ecosystem, as well as the OpenTelemetry toolset for more advanced use cases such as tracing.

    Prometheus has become very common in the developer space and, unlike other open source solutions or libraries, its format automatically emits metrics.

    “This allows for simple and future-proof interoperability, as one of the hardest things to change later down the line is code running,” he said. “Using open standards reduces the need for developers to rewrite and reintegrate the same type of code over and over again.”

    Observability Requires Understanding Your Goals

    From Chessman’s perspective, understanding what your goals are is critical for observability. 

    “Observability tools can do many things, but their primary function is to collect and analyze data,” he said. “While that is all well and good, if you do not define your KPIs from the beginning, all the collected and analyzed data is meaningless since you have no concept of what good and bad are.”

    Trying to define good and bad from the data collected is likely to result in failure because you are looking at where you are to figure out where you want to go.

    KPIs vary from application to application and, if you do not have them identified and well-defined before beginning down the observability path, it is that much harder, he said.

    Most tools (observability or otherwise) are capable of collecting hundreds if not thousands of metrics, but not all—not even most, in many circumstances—will be relevant to an organization. 

    “Instead of having to sift through the cruft to find the valuable data, it is better to start with identifying the valuable data and then figure out how to get there,” Chessman said. “If you don’t identify your KPIs in advance you could end up using the wrong KPIs for the wrong scenarios.”

    Johnston added that KPIs, like DevOps, are an iterative process and starting with KPIs helps developers create systems that can meet performance and availability expectations.

    “It’s important that DevOps teams continuously review their KPIs to allow for regular feedback on successes and failures,” he said.

    From his perspective, the best place to start for maintaining and improving observability is by storing observability definitions, instrumentation, SLOs, thresholds, incident response templates and dashboards as code.

    “This allows for rapid improvement via code changes by any team member, whether that’s a site reliability engineer (SRE), developer or platform team,” he said. 

     

  • Log Management Challenges Grow With Observability Adoption

    Log Management Challenges Grow With Observability Adoption

    It appears that, in many IT environments, observability is becoming too much of a good thing. A global survey of more than 315 IT executives, cloud application architects, DevOps professionals and site reliability engineers (SRE) conducted by Dimensional Research on behalf of log data management platform provider Era Software, found 96% are witnessing an explosion of log data.

    The survey also found 79% of respondents reported the overall cost of observability and log management will skyrocket in 2022 if current tools don’t evolve.

    More than three-quarters of respondents (78%) also noted attempts to manage volumes of log data have had mixed or unwanted results such as an inability to access data. Nearly two-thirds (65%) are also evaluating their observability options while the other 41% are considering it.

    Stela Udovicic, senior vice president of marketing for Era Software, said the survey makes it clear that as organizations rely more on logs to analyze IT events, the cost of storing all that log data is becoming a significant challenge. Overall, the survey finds usage of observability tools and platforms has jumped by 180% as IT teams struggle to manage increasingly complex IT environments.

    On the plus side, more business users are benefiting from log data. A full 83% of respondents report that business stakeholders outside of IT use insights from log data, with 96% reporting log data is being used to solve business problems.

    Observability has always been a core tenet of DevOps best practices, but achieving it has always been a challenge. Monitoring tools are designed to consume predefined metrics to identify when a specific platform or application is performing within expectations. The metrics tracked generally focus on, for example, resource utilization. However, whenever there is an issue, it can still take days, sometimes weeks, to discover the root cause of an issue through what amounts to a process of elimination.

    In contrast, observability combines metrics, logs and traces—a specialized form of logging—to instrument applications in a way that makes it simpler to troubleshot issues without relying solely on a limited set of metrics that have been predefined to monitor a specific process or function. DevOps teams can employ queries to interrogate data in a way that makes it easier to discover the root cause of an issue. Observability platforms correlate events in a way that makes it easier for analytics tools to identify anomalous behavior indicative of an IT issue.

    The issue that arises is that IT teams need to find a way to store all the data generated by the logs they need to analyze. Each IT team can, of course, could reduce the amount of log data they retain. However, there is always a concern that in the event of an incident critical log data won’t be available.

    Udovicic said most IT organizations have historically viewed storing log data as a necessary evil. The primary issue most of them encounter is the simple fact that log data is not easy to work with, she noted.

    Regardless of how organizations view log data, the amount of it that needs to be managed will only increase as DevOps workflows become more mature. The issue IT teams need to come to terms with now is finding a way to minimize the cost of storing it all.

  • LogDNA Survey Surfaces Observability ROI Concerns

    LogDNA Survey Surfaces Observability ROI Concerns

    A Harris Poll of 200 senior engineering professionals suggests it may be a while before most DevOps teams are able to achieve true observability in their IT environments.

    The survey was conducted on behalf of LogDNA, a provider of an observability platform, and found nearly three-quarters of respondents (74%) are struggling to achieve their observability goals despite investing hundreds of thousands of dollars. In fact, well over a third (38%) said they are already investing $300,000 annually on observability tools.

    Despite those investments, less than half of respondents are very satisfied with their ability to use log data. Key areas of frustration include the challenges of collaboration with colleagues spanning multiple teams (67%), the fact that the tools are not easy to use (66%) and the routing of security events (58%).

    Nevertheless, the bulk of survey respondents said they are optimistic about observability. A full 85% of participants said they believe true observability is possible.

    LogDNA CEO Tucker Callaway said the core issue that many organizations fail to recognize is that observability is really a data literacy and management challenge. Organizations are collecting logs, metrics and traces from a wide range of tools and platforms without thinking through optimal data pipeline construction. The result is an aggregation of data that has not been normalized to simplify surfacing actionable intelligence, he said. In fact, many organizations quickly discover that all they’ve really accomplished is increase their data storage costs, said Callaway.

    In the absence of that data normalization, Callaway noted it becomes exceedingly difficult for cross-functional teams to query all the data being collected within a repository.

    At its core, observability promised to make it easier to identify IT issues before they cause a disruption. Observability in one form or another has always been a core tenet of any DevOps best practice. Initially, DevOps teams focused on continuous monitoring as the most effective way to proactively manage application environments. However, it can still take days, sometimes weeks, to discover the root cause of an issue. Observability platforms make it possible to correlate events in a way that makes it easier for analytics tools to identify anomalous behavior that might be indicative of the root cause of an IT issue.

    In contrast, traditional IT monitoring focuses on predefined metrics to identify when a specific platform or application is performing within expectations. Observability platforms combine metrics, logs and traces—a specialized form of logging—to make it easier for IT professionals to interrogate data being generated by a wide range of DevOps tools and platforms.

    Of course, the assumption is that DevOps teams actually know what queries to launch to determine what the root cause of an IT issue might be. In the longer term, machine learning algorithms should make it easier to identify anomalies within IT environments. In the meantime, however, achieving observability still requires a fair amount of DevOps expertise that is not always readily available.

    Eventually, observability will become much more automated than it generally is today. As that transition occurs, IT teams should find the ROI of observability is steadily improving.

  • Grafana Labs Expands into Incident Management Realm

    Grafana Labs Expands into Incident Management Realm

    During its online ObservabilityCON 2021 conference, Grafana Labs this week previewed an incident management platform in addition to making available an on-premises edition of its distributed tracing tool.

    Tom Wilkie, vice president of product for Grafana Labs, said the Grafana OnCall incident management platform is a natural extension of the observability tools the company already provides using the widely employed open source Grafana dashboard. The platform, which is expected to be generally available in 2022, is based on software Grafana Labs gained by acquiring Amixr earlier this year.

    Grafana OnCall unifies all the alerts created by Grafana, Prometheus and the Alertmanager tool the Prometheus community created to streamline the number of alerts generated by the platform. The goal is to reduce the alert fatigue a DevOps team might experience when dealing with an IT issue that can be traced back to a single root cause, noted Wilkie.

    Grafana

    At the same time, Grafana Labs made Grafana Enterprise Traces available for on-premises IT environments. It is based on Grafana Tempo, a cloud service that makes it possible to store and then analyze distributed traces using inexpensive object storage resources. The latest version of the core open source Tempo project has also been updated to add service graphs and the ability to search recent traces.

    Grafana Labs this week also updated Loki, an open source log management platform, to add support for out-of-order log lines, custom log retention policies per log stream and per tenant and a deletion API that makes it easy to scrub log data. Version 2.4 of Loki also adds additional high-availability capabilities.

    Other updates to the Grafana Labs platform include support for recorded queries, a set of pre-built dashboards that enable users to quickly drill into their Prometheus metrics data, the ability to consume metrics from the AWS CloudWatch service provided by Amazon Web Services (AWS), improvements to its ability to monitor Kubernetes clusters and updates to the open source load testing tool the company gained by acquiring k6 last year.

    Grafana Labs is also committing to expanding its integration with Microsoft Azure cloud service alongside its integrations with AWS, noted Wilkie.

    While competition among providers of observability platforms is already fierce, Wilkie said an observability platform based on an open source dashboard that can already collect data from more than 100 sources provides a unique advantage. At the core of the Grafana Labs platform is a centralized framework for collecting data generated primarily by the open source Prometheus monitoring platform being developed under the auspices of the Cloud Native Computing Foundation (CNCF).

    While Prometheus is used most often in Kubernetes environments, it is starting to gain traction in monolithic application environments as well; it provides an open source approach to defining a de facto standard for collecting metrics and other data that can be consumed by an observability platform.

    It’s not clear yet to what degree IT teams are distinguishing between monitoring and observability. Monitoring focuses on predefined metrics to identify when a specific platform or application is performing within expectations. Observability combines metrics, logs and traces to instrument applications in a way that makes it simpler to troubleshoot issues without relying solely on a limited set of metrics that are pre-defined and monitor a specific process or function.

    Regardless of whether IT teams view observability as being distinct from monitoring, it’s clear that the ability to manage IT environments is improving at a rapid rate. The challenge, of course, is the IT environment itself is becoming more complex with each passing day.

  • Observability Vs. Monitoring: What’s the Difference?

    Observability Vs. Monitoring: What’s the Difference?

    As more and more IT organizations pivot from strategies rooted in monitoring to ones that aim to achieve observability, one of the first questions that teams need to answer is, “What is the difference between the two?”

    That’s a somewhat loaded question because there are some differences of opinion out there regarding what, exactly, distinguishes monitoring from observability. But in general, the differences can be distilled into two main points, as we will explain.

    What is Observability and What is Monitoring?

    The first step in understanding the difference is to understand what each term means on its own.

    The classic definition of observability is that it is the use of external data outputs to understand the internal state of a system. In the IT industry specifically, observability could be defined as the use of logs, metrics and distributed traces to understand the state of a complex software system.

    Meanwhile, monitoring is the process by which data is collected from a system. In IT, monitoring is how teams collect logs, metrics and traces and move them to a place where they can be analyzed.

    Similarities

    There are some obvious similarities between the two. Both involve (at least in part) the collection of diverse sets of data. Both also serve the goal of helping teams identify problems within their software stacks and deliver a high-quality user experience.

    Two Key Differences

    But the similarities end there. Ultimately, monitoring and observability are distinct concepts and processes, with two main differences that distinguish them from each other.

    Monitoring collects data. Observability interprets it.

    The first is that observability focuses on interpreting and understanding data, whereas monitoring is merely the collection of data.

    In this sense, you can think of monitoring as one of the processes that makes observability possible. But observability requires more than monitoring. In addition to collecting data, observability also involves the correlation of data from multiple sources and the identification of relevant patterns or anomalies within that data.

    Monitoring Tells you What is Wrong. Observability Tells you Why it’s Wrong

    The second main difference is that monitoring simply alerts you when something goes wrong. Observability goes further by helping you understand why it’s wrong and how to fix it.

    For example, a monitoring tool might tell you that your application’s response rate is no longer adequate. But you need observability to figure out which specific microservices within the application are causing the problem. Using that information, you can plan and execute an effective response to reliability issues using an incident management platform.

    With monitoring alone, you only know that something is broken, not how to fix it.

    Observability and Monitoring: Better Together

    Although it’s common to speak in terms of “observability versus monitoring,” the reality is that you can’t have observability without monitoring. (You could monitor without observing, but doing so would significantly reduce your ability to understand and resolve problems quickly.)

    So, don’t think of it as an either/or proposition. Think of them as concepts and practices that go hand-in-hand, even if they perform different functions.

  • Sumo Logic Extends Observability Scope and Reach

    Sumo Logic Extends Observability Scope and Reach

    During the online Illuminate user conference, Sumo Logic revealed how it is integrating capabilities it gained with the June 2021 acquisition of Sensu, a provider of open source monitoring tools, with its continuous intelligence platform in addition to adding support for data collected via open source OpenTelemetry software.

    Sensu Go, a set of tools for implementing monitoring-as-code, is now a core part of the Continuous Intelligence Platform. The company is also now making available in beta Sumo Logic OpenTelemetry Distro and Ecosystem Support, a single agent through which all logs, metrics and traces can be collected from instrumented applications. Sumo Logic is also supports the AWS OpenTelemetry Distro, an offering from AWS that provides similar capabilities. Sumo Logic has also increased the capabilities of Sumo Logic Free to include monitoring-as-code tools created Sensu.

    Bruno Kurtic, vice president of product and strategy, said as responsibility for managing applications shifts left toward developers it’s critical to provide observability capabilities that can be programmatically infused into applications. As part of that effort, Sumo Logic is now also offering Sensu Plus, which provides access to an analytics engine for the data the platform collects.

    Sumo Logic is also adding machine learning algorithms to provide more context to the alerts the platform generated via an Alert Response feature that is part of the core platform.

    The company is also adding support for the operator software that Red Hat created to make it simple to deploy Sumo Logic Continuous Intelligence Platform alongside the Red Hat OpenShift platform for building and deploying applications on top of Kubernetes clusters. In addition, Sumo Logic is also providing integration with a total of 38 either new or updated observability applications that integrate with a wide range of third-party platforms.

    Finally, Sumo Logic is extending the security observability platform it provides based on its platform to add support for containers and serverless computing functions alongside virtual machines and integrations with third-party endpoint detection and response (EDR), network detection and response (NDR), threat intelligence and managed detection and response (MDR) platforms. The company has also added or updated 26 applications for specific third-party platforms.

    Kurtic said Sumo Logic makes it possible for developers and IT operations teams alike to observe an IT environment on an end-to-end basis via a single pane of glass. In some cases, the members of an IT operations team will require a user interface to manage that environment. Developers, meanwhile, might prefer an application programming interface (API) through which observability data is made accessible from within a continuous integration/continuous delivery (CI/CD) platform.

    Regardless of the type of user, however, IT teams should not have to hire their own data science team to analyze observability data. Instead, the observability platform should embed machine learning algorithms that surface all the benefits of data science without the need to hire additional headcount, he said.

    One way or another, DevOps teams will soon routinely benefit from the additional levels of context that observability platforms provide. The challenge will then be determining how best to fix a raft of issues that many DevOps teams today simply didn’t know existed.

  • Metrics, Logs and Traces: The Golden Triangle of Observability in Monitoring

    Metrics, Logs and Traces: The Golden Triangle of Observability in Monitoring

    It isn’t surprising that the job of monitoring infrastructure and application environments has grown more complex as the underlying technologies evolve. Nor is it surprising that the need to get monitoring right has expanded as our business reliance on IT grows ever deeper, so it is important to have a good grasp of the core tools at your disposal.

    What you might call the golden triangle of observability includes metrics, logs and traces. Each plays a specific role in infrastructure and application monitoring so you need to understand what they bring to the table.

    Metrics

    Metrics probably represents the most valuable of the three monitoring tools because:

    • So many resources are designed to spit out various bits of health and performance information (and there are loads of tools to collect this information).
    • They are efficient.
    • They are frequently generated.
    • They are easily correlated across elements of your infrastructure.

    Everything from operating systems to applications generate metrics which, at the least, are going to include a name, a time stamp and a field to represent some value. Since so many resources come ready to tell us about themselves, metrics is an obvious place to start when it comes to monitoring.

    Most all metrics will enable you to tell if a resource is alive or dead, but if the target is valuable enough you’ll want to be able to ascertain what is actually wrong with the system or going wrong.  As you can imagine, the latter will require detailed information about what is happening inside the system, so called white-box monitoring that relies on internal instrumentation.  The more rudimentary black-box approach draws conclusions about the health of a system based on externally visible indicators (is it responding to any system calls?).

    But perhaps the most important thing to understand about metrics is that last bullet about  being able to correlate metrics across infrastructure components.  Given the complex interdependencies common to IT environments today, the ability to stitch together metrics to get a bigger picture view is a real time saver. And it becomes even more critical as we move to cloud-native environments because of the dynamic nature of cloud  infrastructure and the ever-changing relationship between that infrastructure and the applications.

    An initial challenge of harnessing metrics is the variety of the information available and the number of tools needed to collect and make sense of that information. Then there is the question of how you store data in so many formats from so many resources. But the resultant upside more than makes up for the effort required to figure out how to harness the information.

    It is also worth noting that, given there is no standard API for metric collection, many organizations use agents to collect data that is either pushed to a central location for analysis or pulled by that central resource. Gartner says agents frequently referenced by customers include push agents Collectd and Telegraf, while Prometheus is cited as a tool to pull information from targets.

    Logs

    Logs are perhaps the second most important tool in the monitoring toolbox because virtually everything logs information about what they are doing at any given time. What’s more, logs tend to give more in-depth information about resources than metrics.  So, if metrics showed the resource is dead, logs will help tell you why it died.

    The problem with logs is there can be too much of a good thing. With everything in your environment tracking what they are doing and anxious to share that information, it is easy to see how that could result in a mountain of data. Instead of streamlining the monitoring process, you are simply creating a big new centralized haystack.

    And like metrics, differences in log formats and the abundance of tools available to collect and make sense of logs, complicates the job of getting the most out of this rich trove of material. There are, however, a number of common tools used for collecting logs, such as syslog and open source tools such as Fluentd.

    The trick to getting the most out of these tools is limiting what you collect to keep it manageable, and, where possible, to home in on common fields so you can more easily find the needles in the haystack.

    Traces

    Last but not least in the monitoring triangle is application trace data, which “traces” information about specific application operations. With so many application interdependencies these days, these operations will typically involve hops through multiple services (so called spans).

    Traces, then, add critical visibility into the health of an application end-to-end. They are, however, solely focused on the application layer and provide limited visibility into the health of the underlying infrastructure. So, even if you collect traces, you still need metrics to get the full story of your environment. APM tools feed trace information to a centralized metrics store, so traces provide a nice source of data for an app-centric view.

    The need for the viewpoint that traces can provide is exacerbated in container-based microservice architectures that are nothing more than a collection of stitched together services. These environments can be addressed with something called distributed tracing.

    Do You Need All Three?

    There is obviously overlap among metrics, logs and traces. But the quick answer to the question of whether you need all three types of monitoring tools is, It depends. The simpler your environment and the more tolerant you are of performance degradation and outages, the fewer tools you’ll need to keep things running. Basic metrics will probably work fine for you.

    Conversely, more complex environments that have to be up and running at all times or need to be fixed as quickly as possible will require a mix of tools that answer more than the question of, “Is it broken?” Metrics and logs will be base requirements.

    And finally, if your environment consists of a lot of intricately interwoven pieces, then adding traces will save you effort when it comes to birddogging problems. If you’re not there yet, but see containers and microservices playing a bigger role in your future, you’ll probably want to start getting familiar with trace tools today.

    Keep in mind that each of these tools requires storage considerations. Yes, there are options to support multiple tools from the same repository, but you’ll want to consider those needs as you build out your monitoring repertoire. And ultimately, you’ll want analysis and alerting tools that can span the environments, so keep that in mind as you consider the options.

    Together metrics, logs and traces make up the golden triangle of observability and will help you stay on top of the ever-churning IT world driving business today.

    — Apurva Dave