devopsy.kinsta.cloud

Tag: Distributed Tracing

A History of Distributed Tracing
Organizations are increasingly using distributed tracing to monitor their complex, microservice-based architectures. Distributed tracing has become essential in microservice applications, cloud-native and distributed systems. Microservices and serverless applications can grow exponentially, which makes observing them at scale very challenging. The traditional logging method becomes expensive, and if there’s an issue, time-series data can reveal symptoms but not the root cause(s). With modern usage patterns of Kubernetes service mesh like Istio and Envoy, for example, there can be billions of microservices calls daily.

What is Distributed Tracing?

Traces describe the progress of a transaction or workflow through a system. Distributed tracing provides visibility into service dependencies and their interdependencies by providing end-to-end visibility. Visualizing transactions in their entirety allows you to compare non-normal traces with normal ones to determine differences in behavior, structure and timing. Through distributed tracing, you can identify bottlenecks in your system. As a traced request flows through a network, it can be tracked.

As requests flow from frontend to backend and to databases, traces provide diagnostic techniques that reveal how a set of services coordinate to handle an application request. Using distributed tracing, developers can troubleshoot requests with errors and high latency. A single trace typically shows the activity of a request or transaction within an application. A trace shows what has changed at each step. As you aggregate the traces, you can see which backend service or database has the most significant impact on performance.

The History of Distributed Tracing

Dapper, a large-scale distributed systems tracing infrastructure, was introduced by Google in 2010. Two years after Dapper was made public, Twitter open sourced Zipkin, which was designed for application performance tuning. Zipkin was the first open source distributed tracing project. Zipkin trace data can be collected and visualized using a UI. In 2015, Uber launched Jaeger, which was inspired by Dapper and named after the German word for hunter. Later in 2017, Uber published a blog post, called Evolving Distributed Tracing at Uber Engineering, explaining the reasons for the architectural choices in Jaeger. In addition to creating Jaeger, its author, Yuri Shkuro, wrote a book about distributed tracing called Mastering Distributed Tracing.

In 2016 Ben Sigelman, founder of Lightstep, wrote a blog post called Toward Turnkey Distributed Tracing, describing OpenTracing as a single standard. Some people refer to this as the OpenTracing Manifesto. OpenTracing allows developers of application code, open source packages and open source services to instrument their code without binding themselves to any particular tracing solution. The goal of OpenTracing was to solve a standardization problem. Trace context must pass through all the components, including application code, dependent libraries, standalone open source services (Nginx, MySQL) and other vendor-specific libraries and services, to collect a complete distributed trace. Collecting a full path without a standard API to define the collection and passing of trace context is difficult. OpenTracing aims to solve this problem by defining a standard API that can be implemented by components from different tracing solutions, allowing the collection of end-to-end tracing data. The Cloud Native Computing Foundation (CNCF) accepted OpenTracing as its third project in October 2016. Two months later, OpenTracing 1.0 was released. After OpenTracing, Jaeger joined the CNCF.

W3C tracing context specification was proposed in November 2019, bringing distributed tracing closer to standardization. Over ten years, distributed tracing evolved from one paper to an active community. With standardization on all the layers, it is moving from just tracing to overall observability, ranging from latency optimization to root cause analysis and application performance management. It is moving from a single backend system to an end-to-end solution that spans multiple systems.

OpenTracing and OpenTelemetry merged in 2019. Using OpenTelemetry, distributed tracing can be implemented end-to-end. It released version 1.0 in 2021. CNCF’s OpenTelemetry project is one of the fast-growing projects, and OpenTelemetry has become the de facto standard for tracing, metrics and logging.

How it Works

Traces in distributed tracing consist of tagged time intervals called spans. It is possible to think of a span as a unit of work. Traces are directed acyclic graphs (DAGs) composed of segments known as spans. According to OpenTelemetry, There are four major components of a trace:
- traceID
- - A unique 16-byte array to identify the trace that a span is associated with
- spanID
- - Hex-encoded 8-byte array to identify the current span
- Trace Flags
- - Provides more details about the trace, such as if it is sampled
- Trace State
- - Provides more vendor-specific information for tracing across multiple distributed systems. Please refer to W3C Trace Context for further explanation.
The spans in a trace consist of contiguous segments of work that are named and timed. There are start and end times for spans, as well as metadata such as tags or logs that help classify them. Parent-child relationships may be present between spans to show how particular transactions traverse the application’s numerous services and components.

A trace represents an end-to-end request; it can contain one or more spans. Spans indicate work performed by a single service with associated time intervals and metadata. The purpose of traces is to provide a request-centric view of spans through tags. Despite microservices enabling teams to work independently, distributed tracing offers a central resource that allows all teams to understand issues from a user’s perspective.

Conclusion

In distributed systems, response latency can have a significant commercial impact. Identifying bottlenecks and understanding how a request moves through a complex system is complicated. Other techniques, like logging and monitoring metrics, don’t provide insight into distributed applications such as those created with microservices architecture. Several open standards and tools are emerging within the distributed tracing space, like OpenTracing and commercial tools that could compete with existing APM solutions. Implementing distributed tracing for modern cloud-native services poses several challenges. OpenTelemetry tracing is necessary for processing high volumes of trace data while generating meaningful insight.
December 6, 2022
Logz.io Embraces Jaeger for Distributed Tracing

Logz.io announced at its online ScaleUp! 2020 conference announced today it has added support for open source Jaeger distributed tracing software to its observability platform based on a curated instance of open source Elasticsearch, Logstash and Kibana (ELK) software.

Logz.io CTO Jonah Kowall said support for Jaeger extends the reach of the open source platform by enabling the platform to consume data generated by Jaeger agent software, which is now being deployed more widely to instrument complex distributed computing applications.

Jaeger, which is being developed under the auspices of the Cloud Native Computing Foundation (CNCF), is finding traction among DevOps teams who are deploying microservices-based applications based on containers, which tend to have a lot of dependencies that are challenging to monitor and troubleshoot.

A survey of more than 1,000 DevOps and software development engineers conducted by Logz.io finds 50% of respondents have half their applications or more running in containers, with 38% using service meshes and 33% employing serverless computing frameworks. The survey also finds 87% of respondents view their main observability challenges to be Kubernetes, microservices and serverless frameworks. The most common Kubernetes difficulties cited are monitoring and troubleshooting (44%), followed by security (35%), networking (34%) and cluster management (30%).

More than a quarter of respondents (26%) said they already make use of distributed tracing, with 68% reporting they expect distributed tracing to play an important role in their observability strategy. Nearly two-thirds said they expect to implement distributed tracing in the next two years, with 91% of respondents reporting they already employ two or more observability tools. The most commonly used tools are log management and analysis (90%) and infrastructure monitoring (78%).

Just over half of respondents (56%) also noted that more than half their infrastructure now resides in the cloud, with 42% noting that cloud now accounts for more than 75% of their infrastructure. Amazon Web Services (AWS) is the most widely used cloud (66%), followed by Microsoft Azure (12%) and Google Cloud Platform (11%).

Kowall said that while interest in employing ELK along with visualization tools such as Grafana is rising to reduce the overall cost of observability, IT organizations are also making it clear they don’t want to integrate an ELK stack themselves. Most IT organizations would prefer to focus their efforts on building and deploying applications rather than maintaining an observability platform.

The rise of microservices clearly is forcing the observability hand of more organizations. Most application performance management (APM) tools use has been limited to mission-critical monolithic applications because of the cost of deploying proprietary agents and platforms. Open source platforms now present an opportunity to observe both microservice and monolithic applications at a time when the overall IT environment has become much more difficult to manage.

There is, of course, no shortage of options when it comes to achieving the observability of IT environments. The challenge now is determining how much observability is actually required and at what level of acceptable cost.

November 10, 2020
Instana Applies Continuous Profiling to Distributed Traces

Instana today announced it has added a continuous profiling capability to its application performance management (APM) that makes it easier for DevOps teams to leverage distributed traces to troubleshot issues.

Chris Farrell, technical director and APM Strategist at Instana, said that while distributed traces make it easier to identify bottlenecks, continuous profiling makes it possible to drill down into profiles to identify to identify, for example, specific CPU, memory and concurrency issues.

Based on technology that Instana gained when it acquired StackImpact last year, Farrell said the continuous profiling capabilities are being added to the Instana APM platform at no extra cost.

Rather than having to build, maintain and deploy a monitoring platform that can consume distributed traces, Instana is making the case for a software-as-a-service (SaaS) platform that offloads all the back-end management of an observability platform from the DevOps team, including the cost of collecting and storing distributed traces.

Farrell said the profiling capability continuously maps individual traces directly to applicable process profiles. Whenever there is an issue, those profiles are instantly available. In contrast, rival approaches to employing distributed traces seek to replicate the bottleneck issue discovered in the hopes of pinpointing where the issue is, which Farrell noted is still a hit-or-miss proposition for DevOps teams attempting to resolve an issue. However, he said, hope is not an attribute of a sound engineering process.

The Instana APM platform is optimized for applications based on microservices but it can also be applied to monolithic applications. However, as organizations deploy microservices-based applications dependencies become more complex, which in turn necessitates employing some type of observability platform. Historically, most IT organizations have limited APM platform use to their most valuable monolithic applications because of the costs associated with employing them.

Farrell said the Instana platform will consume trace data generated by its own agent software as well as by open source projects seeking to reduce the total cost of instrumenting applications. Instana automatically deploys sensors for each part of the application technology stack and traces all application requests without requiring any human configuration or application restarts. It also detects application and infrastructure changes in real-time, adjusting its own models to visualize changes and assess performance impact in seconds.

Ultimately, Farrell said the goal for adopting any APM platform should be to reduce mean time to resolution for IT teams that are increasingly tasked with managing complex applications. The challenge is that not only are there more complex applications than ever, but also organizations are more dependent than ever on applications. In the wake of the economic downturn brought on by the COVID-10 pandemic, many organizations have been forced to accelerate digital business transformation initiatives that are wholly dependent on the rate at which applications can be built, deployed and maintained successfully.

Of course, Instana is not the only provider an APM platform racing to embrace distributed tracing to make it easier to manage application environments. The issue now is determining which APM platform won’t make its customers pay extra for the privilege.

September 1, 2020
CNCF Provides Jaeger Distributed Tracing Update

The Cloud Native Computing Foundation (CNCF) has made available a report detailing the progress being made around the open source Jaeger distributed tracing software. Among other things, the report notes there are now more than 300 organizations contributing to the project.

Jaeger is designed to provide a framework for monitoring microservices-based applications in a way that captures every interaction as part of an effort to simplify troubleshooting.

Originally developed by Uber, Jaeger has almost 2,000 contributors. Developers from Uber and Red Hat make the bulk of those contributions., while other developers work at Ryanair, IBM, Ticketmaster and CloudBees.

Thus far, there have been nearly 40,000 contributions, 4,300 code commits and 3,200 pull requests for Jaeger.

Yuri Shkuro, a software engineer at Uber who created and maintains the Jaeger project, said as open source agent software in the form of OpenTelemetry, a separate sandbox-level initiative also hosted by the CNCF, gains traction, the more feasible it will become for DevOps teams to embrace distributed tracing. Currently, most DevOps teams are dependent on sampling techniques typically employed via an application performance management (APM) platform. While that approach can indicate there might be an issue, distributed tracing makes it easier to pinpoint the precise problem.

That’s critical because many DevOps teams can spend hours, if not days, trying to identify the source of an issue that often only takes a few minutes to fix. The trade-off is that distributed tracing collects a lot more data than sampling, so IT teams will need to decide how long they may want to store that data. Most organizations will only store tracing data that identifies anomalies based on a reasonable retention schedule rather than allocating storage resources to what could become massive amounts of tracing data, noted Shkuro.

Providers of commercial observability platforms have been embracing OpenTelemetry to provide distributed tracing services, so it’s not clear to what degree IT organizations will opt to deploy Jaeger on their own or access a managed Jaeger-based service.

Regardless of how they are invoked, next-generation observability platforms based on Jaeger should become more affordable at a time when microservices-based applications are being more widely deployed. Given the inherent complexity of those applications, observability platforms that are a core tenet of any best DevOps practice become a requirement. In comparison, the use of ALM platforms was typically limited to only the most mission-critical applications because of the costs associated with deploying proprietary tools.

It may be a while before every application is instrumented by default. In the meantime, Shkuro said the Jaeger project is looking to recruit additional maintainers. As is often the case with any open source project, there are plenty of contributors. IT professionals willing to volunteer to maintain a project are often more scarce.

August 10, 2020
Sumo Logic Expands Observability Platform Scope, Reach

Sumo Logic announced today it has extended the reach of its monitoring platform to provide deeper insights across application and infrastructure logs, metrics, traces and metadata in real-time.

The latest additions to the company’s suite of software-as-a-service (SaaS) offerings include Sumo Logic AWS Observability Solution, which focuses exclusively on AWS services, and additions to the Sumo Logic Software Development Observability Solution that provide support for GitHub, Jenkins and PagerDuty alongside existing support for Jira, Bitbucket and OpsGenie. The Sumo Logic Software Development Observability Solution is based on a key performance indicator (KPI) methodology developed by the DevOps Research and Assessment (DORA) organization, an arm of Google.

In addition, Sumo Logic has launched a closed beta for a Distributed Transaction Tracing service based on the OpenTelemetry standard that will integrate logs, metrics and metadata. Finally, the company has added more tracing capabilities to its existing Microservices Observability Solution.

Sumo Logic got its start by providing a cloud service for analyzing logs as an alternative to Splunk. Since then, the company has extended the scope of its services to address a wider range of observability requirements for DevOps teams.

Ben Newton, director of field evangelism for Sumo Logic, said the company is trying to engage IT organizations at any level of maturity by providing access to a range of offerings from log analytics to observability tools for various platforms including Kubernetes. Those offerings now span the application life cycle from coding to grave, he said.

Sumo Logic also differentiates a pricing model based on data tiers that enables IT teams to analyze logs, metrics and traces across a variety of use cases without monthly limits, peak provisioning or on-demand charges for data volume overages or exceeding custom metrics limits.

In the wake of the economic downturn brought on by the COVID-19 pandemic, the rate at which DevOps tools have been shifting into the cloud has accelerated. With more DevOps teams working from home, cloud-based services are more flexible. At the same time, more organizations prefer to treat IT costs as an operating expense at a time when the overall economic outlook remains uncertain. The entire IT organization needs to be able to respond to rapidly changing business conditions in a more agile manner, noted Newton.

Less clear is whether the economic downturn might also drive consolidation across what today is a spectrum of observability tools spanning log analytics to application performance management (APM) platforms used primarily to sample application traffic. It’s even less clear whether the rise of distributed tracing capabilities might reduce dependency on application sampling altogether.

What is apparent is IT teams have never had more visibility into IT environments that increasingly are more complex. The challenge and opportunity now is determining how best to apply a wide range of observability tools to turn all that insight into actionable intelligence.

August 6, 2020
New Relic Makes Revamped APM Platform More Affordable

New Relic has revamped its application performance management (APM) platform as part of an effort to make observability more accessible and affordable.

Bill Staples, chief product officer for New Relic, said New Relic One has been updated to provide a new user interface designed to make it easier to navigate and explore an observability platform that now monitors both applications and IT infrastructure.

At the same time, New Relic has reduced costs by adding a free tier with no hidden costs or credit card required, he said. It provides one free license to 100GB of data every month. The first 100 million application transactions per month free as are the first 1,000 incident events per month.

Initial pricing for the next level tier is at 25 cents per GB of ingest on a per-seat basis with applied intelligence priced per transaction or event.

Staples said New Relic is further reducing costs by leveraging open source agent software to collect data, which along with the revamped New Relic One platform will make observability ubiquitous.

The move to reduce observability costs coincides with the rise of microservices-based applications that by definition have many dependencies that need to be monitored continuously. Historically, many IT organizations have limited use of APM platforms to only their most mission-critical monolithic applications because of the costs involved. Given their inherent complexity, however, microservices-based applications require IT teams to employ an observability platform. They also drive organizations to embrace best DevOps practices, of which observability is a core tenet.

Staples said it’s still early days in terms of the adoption of observability platforms, noting in light of their absence, most of the adoption of DevOps thus far could be described as a half-measure.

To further that goal, Staples said New Relic One will also make it more cost-effective for IT organizations to embrace distributed tracing as an alternative to application sampling without having to incur massive storage costs. Distributed tracing monitors all requests between services with each segment recorded as a span. All the spans of a request are combined into a single distributed trace to provide a method to monitor an entire request.

Competition among provides of APM platforms is now nothing short of fierce. New Relic today has a significant share of the market, but as demand for observability platforms continues to expand, it has more rivals than ever. In the wake of the economic downturn brought on by the COVID-19 pandemic, IT organizations are also pressuring vendors to reduce costs. The challenge IT organizations face is that while they are trying to reduce costs, they are also rolling out more cloud-native applications based on microservices that require deeper levels of observability.

It’s unclear how the battle of the future of observability will play out. The good news, however, is the cost of proactively monitoring IT environments appears to be dropping by the day.

July 30, 2020
Traceable Secures Apps Using Distributed Tracing

Traceable this week announced it is making available under an early access program a namesake platform that combines distributed tracing and machine learning algorithms to better secure applications.

Fresh of raising $20 million in funding, Traceable CEO Jyoti Bansal said the goal is to leverage distributed traces generated via open source Hypertrace agent software to capture streaming data to which machine learning algorithms can be applied to better secure microservices.

Bansal, who is also the founder and former CEO of AppDynamics, said this approach leverages software engineering techniques and principles to address application security issues both before and after an application is deployed.

As part of this effort, Traceable has developed TraceAI, a machine learning engine that has been trained to identify application security issues.

Once those issues are discovered, developers will be able to apply policies to remediate vulnerabilities using Hypertrace agent software, he said.

In effect, Traceable is applying many of the principles of application performance management (APM) platforms to securing applications. However, rather than simply aggregating and analyzing data, Traceable has added a streaming engine to collect massive amounts of cybersecurity data in real-time that is analyzed by machine learning algorithms to determine, for example, when and how data might be exfiltrated via an application programming interface (API).

As the number of microservices that are deployed in enterprise IT environments increases, the number of API endpoints that can be compromised does as well. By adding open source Hypertrace agent software to microservices, Bansal said it becomes possible to leverage distributed tracing to better understand the context in which how business logic embedded within a microservice is being compromised. That context also serves to enable development teams to remediate that issue by either updating or replacing a microservice.

While the rise of best DevSecOps practices has increased collaboration among developers and cybersecurity teams, Bansal said application security remains largely a software engineering challenge. Application development teams need to find a way to instrument agents to address application security issues within their code instead of relying on overlays deployed by network engineers, he said.

Most cybersecurity professionals would nod their heads in agreement with that principle. However, it may take cybersecurity teams some time to get used to the idea that they might not need to be as involved with application security as they are today. In fact, the amount of time spent looking for the root cause of any vulnerability is likely to be substantially reduced.

In the meantime, it’s apparent a primordial soup of agent software, distributed traces, machine learning algorithms and streaming data engines is coming together in a way that will transform application security. It may be a little while longer before the catalyst that converts all those elements into a widely employed platform arrives. However, it appears it’s now more a question of when rather than if there will be a major advance in application security that could take DevSecOps processes to a much higher level.

July 15, 2020
New Relic Adds Support for W3C Trace Context Spec

New Relic has begun adding support for the W3C Trace Context standard within the agent software it uses to collect metrics for its application performance management (APM) platform.

W3C Trace Context is a specification that defines the format for propagating distributed tracing context between services. Distributed tracing makes it simpler for DevOps teams to track events across multiple services. Each step of a trace is correlated through an ID that is passed between services, with the specification defining a standard for the context propagation headers.

The W3C Distributed Tracing working group moved the Trace Context specification to the point where it could be consumed by platform vendors late last year. TraceContext has since been adopted by the OpenTelemetry project, an open source observability framework being developed under the auspices of the Cloud Native Computing Foundation (CNCF), along with providers of APM platforms such as New Relic, LightStep and Dynatrace. Vic Soares, director of product management for New Relic, said many of the individuals working on the TraceContext and OpenTelemetry projects come from the same organizations.

The first phase of the Trace Context project has focused on HTTP. The members of the project are also working toward defining trace context propagation for other formats, starting with AMQP and MQTT for IoT. The project team is also working on providing context propagation from clients and web browsers.

That approach solves an issue that arises when using proprietary formats: As trace context headers move across boundaries such as proxies, service meshes and messaging systems, proprietary headers tend to be dropped, which often winds up breaking traces, said Soares.

Soares said DevOps teams will need to upgrade the agent software they have in place to take advantage of Trace Context to provide more context across distributed services. Distributed tracing, however, will not replace sampling. Rather, distributed tracing provides a mechanism for identifying the root cause of issues that may first be uncovered using sampling techniques.

New Relic APM agents for Java 5.1.0 and higher, Python 5.5 and higher, Go 3.1.0 and higher and the New Relic open source Elixir agent now support the standard. Support for Trace Context support in other New Relic agents, as well as the New Relic Browser agent, are expected to be available shortly. New Relic APM agents that support W3C Trace Context can accept and emit both the W3C Trace Context header format and the New Relic header format. The new agents are also backward-compatible to enable trace context and will be propagated between services with older and newer releases of New Relic agents. IT organizations can also opt to disable the New Relic proprietary trace format once they’ve upgraded all agents to a W3C-compliant version.

Given the complexity of microservices-based applications, many organizations that deploy these applications will be among the first DevOps teams to take advantage of distributed tracing to not just troubleshoot applications but also proactively manage their application environments. Going forward, a lot more of the DevOps focus needs to be on identifying potential issues long before they can become a bigger problem.

— Mike Vizard

February 7, 2020
CNCF Advances Jaeger Distributed Tracing Project

The open source Jaeger distributed tracing platform has officially graduated into the top tier of projects being stewarded by the Cloud Native Computing Foundation (CNCF).

Designed to be employed within any DevOps environment, Jaeger is a tool for monitoring transactions across complex distributed computing environments. It consists of a backend for trace collection and analysis and multiple client libraries implementing the OpenTracing application programming interface (API) in various languages.

Developed by Uber, Jaeger now integrates with tools such as OpenTracing, OpenTelemetry, Envoy and Prometheus to make it easier to discover the root cause of issues impacting transactions in complex IT environments. For example, DevOps teams can apply data mining techniques and machine learning algorithms to analyze data from a specific cluster.

Yuri Shkuro, a software engineer for Uber who works on the Jaeger project, said organizations may rely on tools such as Prometheus to monitor environments, but once an issue is identified, it typically takes a tool such as Jaeger to identify the root causes of an issue. The primary reason for that approach is the amount of data that distributed tracing tools would store if they were to run continuously, he said.

Organizations that are already using Jaeger in production environments include GrafanaLabs, Northwestern Mutual, SeatGeek, Symantec, Ticketmaster, Uber, Weaveworks and Zenly.

Jaeger is also starting to be bundled with other platforms, such as the Red Hat OpenShift Service Mesh. As that trend continues, more organizations are likely to become exposed to Jaeger as a tool for debugging a wide variety of issues.

Although originally developed to address the inherent complexity associated with microservices, Jaeger also can be applied to monolithic applications, Shkuro said. While many of those monolithic applications eventually are likely to be deconstructed into a set of microservices, many of those applications are not going away anytime soon, he noted.

Jaeger now has 15 active maintainers from four companies, with more than 1,200 contributors and 375 authors of commits and pull requests. It also has more than 9,000 GitHub stars and has been pulled from Docker Hub more than 10 million times.

While distributed tracing as a concept has been around for a while, the rise of microservices is forcing IT teams to look for tools that make it possible to engage in deeper forensics. The dependencies between microservices coupled with the ephemeral nature of containers often make it challenging for IT organizations to discover the root cause of intermittent issues. Within IT environments that have adopted Kubernetes to run those containers, it’s also become easier to deploy Jaeger using tools as Kubernetes templates, Operator software and a Helm Chart.

As developers are held more accountable for applications, interest in programmable forensics tools is rising. Performance issues that developers once viewed as a problem to be solved by faceless IT operations staffs are now being solved much faster by developers. As a result, application experiences have improved now that developers are seeking out tools such as Jaeger to identify issues that are preventing them from focusing their efforts on writing additional code. Given the long, difficult history of IT, there are more than a few IT leaders wondering what took them so long to figure that out.

— Mike Vizard

November 8, 2019
Splunk Adds Omnition to DevOps Portfolio

Splunk followed up on its pending acquisition of SignalFx, a provider of an application performance management (APM), by also acquiring Omnition, a startup building a distributed tracing tool accessed as a software-as-a-service (SaaS) application.

Rick Fitz, senior vice president and general manager for the IT Markets Group at Splunk, said Omnition consists of a small team of developers who have been especially active in maintaining OpenCensus, an open source set of libraries for various languages that make it easier to collect application metrics and distributed traces and then transfer the data to a backend of choice in real-time; and OpenTelemetry, a project led by the Cloud Native Computing Foundation (CNCF) that merges OpenCensus and OpenTracing to standardize the instrumentation and telemetry data collection.

Distributed tracing is emerging as a requirement for APM platforms because the IT environment is becoming both more complex and ephemeral. Serverless computing frameworks are especially challenging because the only way to know an event occurred involving, for example, cloud bursting is to employ distributed tracing. The challenge is, distributed tracing requires organizations to capture, store and analyze massive amounts of data to pinpoint the root cause of any application performance issue, noted Fitz. As such, distributed tracing tends to be employed only by the most elite DevOps practitioners, as implementing distributed tracing is nontrivial, he added. On the upside, distributed tracing reduces the need to deploy a large amount of agent software across the enterprise.

Fitz said Omnition will provide a set of capabilities that over time will be added to the SignalFx platform. Long-term, Splunk hopes to reduce the costs associated with distributed tracing to the point where that capability becomes democratized, he added, noting that as organizations embrace microservices to build cloud-native applications that will run in hybrid computing environments, increased reliance on distributed tracing is now all but inevitable.

In general, Fitz said Splunk is working toward combining the best elements of DevOps with traditional IT service management (ITSM). Most organizations will be employing both approaches for years to come simply because some applications don’t need to be updated regularly, so it doesn’t make any sense to apply DevOps processes to those applications, he noted. At the same time, as DevOps processes are embraced in enterprise IT environments, appreciation is growing for IT operational challenges that stem from multiple applications running at scale which need to share a limited amount of IT infrastructure resources.

With the acquisitions of SignalFx and now Omnition, Splunk is betting that it will be able to combine IT operational analytics based on machine data generated by logs with an APM platform to correlate application performance against infrastructure events. It’s not clear how long it will be before Splunk can fully realize that vision. In the meantime, the rate at which application and infrastructure management platforms will converge will only accelerate as developers are held more accountable for applications long after they are deployed in a production environment.

— Mike Vizard

September 5, 2019