Company CEO Christine Yen said that capability eliminates the need to deploy, for example, a separate real user monitoring (RUM) tool to troubleshoot the front end of an application. That approach ultimately reduces the total cost of managing an application environment in part by leveraging open-source OpenTelemetry agents to collect a wider range of data, she added.
Honeycomb achieves that goal by making use of an OpenTelemetry wrapper to instrument the front end of an application in a way that makes it possible to collect performance data, trace and other custom attributes that DevOps teams can define.
The Honeycomb platform then analyzes that high-cardinality data alongside the back-end application to provide more holistic insights into the application environment, Yen noted. Previously, IT teams would have needed to separate RUM and application performance management (APM) platforms that they then would have to integrate themselves, noted Yen. That approach is fundamentally inefficient because both platforms are essentially collecting different types of telemetry data that should be analyzed on a single platform, she added. The Honeycomb approach is designed to eliminate the divide that exists between the management of front and back ends of applications, noted Yen.
Honeycomb is designed to make it simpler to declare which events DevOps teams want to analyze, and then pull all the raw data associated with those events into a Honeycomb cloud service where analytics is applied. That approach makes it simpler for DevOps teams to move beyond basic monitoring of a set of pre-defined metrics, in favor of an observability platform they can query to better ascertain the root cause of an issue.
It’s still early days as far as the adoption of observability platforms is concerned. However, it’s apparent that as application environments become more complex, the plethora of monitoring tools that most IT teams rely on need to be superseded by platforms that make it possible to investigate and correlate anomalous events.
The rate at which IT teams embrace observability naturally varies, but the biggest immediate obstacle might not be the platforms themselves. Understanding what queries to craft to surface the root cause of an IT issue before there is a major disruption requires a level of expertise that is often challenging to find and retain. Fortunately, as machine learning algorithms and generative artificial intelligence (AI) capabilities are added to these platforms, the level of expertise required to effectively use them continues to decline.
As IT environments continue to become more complex, it’s more a question of the degree to which organizations will be embracing observability rather than if they will do so. As a core tenet of DevOps best practices, IT teams that programmatically manage IT environments tend to be at the forefront of the adoption of these platforms. The challenge and the opportunity now is to take observability to the next level before DevOps teams become even more overwhelmed than many of them already are.
OpenTelemetry promised to be a unified standard that would make it easier for everyone to collect and correlate traces, logs and metrics from distributed systems. Sounds like a dream, right? Well, here is the hard truth: OpenTelemetry is only satisfactory. When I compare it to deeper tracing technologies such as eBPF, OpenTelemetry feels bloated, inefficient, incomplete and missing a key component of tracing. And much of this is thanks to the corporate hijacking of what should have been a lean, community-driven project.
Let’s dive into why eBPF and OpenTelemetry have a place in the world, but also why I think OpenTelemetry is only a support team player and eBPF is the real MVP.
OpenTelemetry: The Swiss Army Knife That’s Trying Too Hard
OpenTelemetry is like a Swiss Army knife I got for my birthday: It has tons of tools, but all of them are only mediocre. It tries to do everything — logs, metrics and traces, with dozens of integrations. It provides a standardized, vendor-neutral way to gather observability data. In a world full of fragmented tools and monitoring solutions, that is valuable. But its broad scope has come at a cost: It is bloated, slow and inefficient.
So, why is OpenTelemetry bloated?
1. Corporate Co-Opting and Feature Creep
OpenTelemetry started as an open-source project with a clear and focused vision. But, like many open-source projects that gain attention, large enterprises saw an opportunity. They swooped in, contributing resources and features — but not necessarily out of altruism. These companies have their agendas, adding features to ensure their platforms, cloud services and proprietary tools are covered. As more enterprises pile on, OpenTelemetry must stretch more to accommodate everyone.
What started as a simple and elegant solution, has now become a bloated beast. We ended up with a tool that has too many knobs, options and complexity. Every company wants their special requirements baked into the core of the project, leading to feature creep that bogs down OpenTelemetry’s performance. The more features we add, the more we dilute the focus and efficiency of the tool.
I’ve been around enough to see it in the wild: The complete inefficiencies of the OTel collector cause an unreasonable and absurd amount of scaling, resources and capacity just to keep up with a marginally busy environment. This becomes unworkable and unmanageable on any reasonable scale!
2. Inefficiency: Trying to be Everything for Everyone
As OpenTelemetry tries to cover everything, it excels at nothing. When I trace a request using OpenTelemetry, I witness a performance overhead that makes me wonder if it is worth it. While it is great at a high level — tracing application flow or showing where bottlenecks might be — it lacks the precision and depth you may need to dig into the nitty-gritty.
This inefficiency stems directly from OpenTelemetry’s broad mandate. It is not just trying to capture traces, it is also juggling metrics and logs, all while integrating with hundreds of other tools and systems. The overhead becomes more noticeable at scale, making it difficult to recommend for performance-critical environments. When you need precise, real-time insights, OpenTelemetry can feel like an anchor dragging behind the app.
3. Incomplete: Just ‘Good Enough’ for Enterprises to Cash In
Here is the kicker: Despite all the bloat and inefficiency, OpenTelemetry still feels incomplete. This is because large enterprises are more interested in creating an open-source product that is ‘good enough’ to get us in the door but not so good that it satisfies all our needs. If you want the full observability package — optimized performance, advanced analytics, smooth scaling — you would be nudged toward their proprietary tools and services. It is a classic bait-and-switch.
In other words, OpenTelemetry is functional enough for basic observability, but when things get complex, you are often pushed into premium enterprise add-ons. It is not a coincidence — this is part of the corporate strategy. They contribute just enough to the open-source project to make it widely adopted but hold back the best features for their paid offerings.
eBPF: The Low-Level Tracing Tool That Just Does it Better
What is the alternative to OpenTelemetry’s bloated mediocrity? eBPF — the hero we didn’t know we needed. While OpenTelemetry operates at a high level — instrumenting applications and services — eBPF (extended Berkeley packet filter) works at the kernel level. It is the secret sauce for real-time, low-overhead observability, generating insights directly from the operating system. For instance, if you wish to know exactly why your network latency is spiking, or which process is causing a performance bottleneck — eBPF has you covered.
Here is why eBPF is the real MVP.
1. Lightweight and Fast
Unlike OpenTelemetry, which can feel bloated and sluggish, eBPF is lightweight and incredibly efficient. It doesn’t try to do everything — it only generates raw, real-time data from the kernel. eBPF allows you to observe and manipulate low-level system behavior, from I/O operations to network traffic, without a significant overhead.
In environments where performance matters, eBPF is a go-to tool. It provides deep visibility into how the system is behaving without the burden of the extra layers and complexity that come with OpenTelemetry.
2. Granular, Deep Insights
eBPF provides visibility that OpenTelemetry can’t match. While OpenTelemetry is great for tracing requests across services, eBPF gives insights at the kernel level. If you need to know why your app is consuming too much CPU or why I/O operations are slowing down, eBPF allows you to see the exact system events that are causing the issue. It is the difference between listening to the weather forecast and being able to measure the air pressure, wind speed and humidity in the backyard.
While OpenTelemetry gives us a 10,000-foot view, eBPF lets us zoom in to a molecular level.
3. Not Co-Opted by Enterprises (Yet)
eBPF remains a relatively niche tool, which means it hasn’t yet been co-opted by enterprises in the way OpenTelemetry has. It is a powerful, kernel-level technology that has yet to be tainted by corporate agendas. It still does what it was designed to do: Provide deep, granular insight into the system with minimal overhead. It hasn’t been bloated with unnecessary features yet and is not trying to funnel us toward a paid solution.
OpenTelemetry is Great for the Big Picture, But eBPF Unlocks the Depths
At the end of the day, OpenTelemetry and eBPF have their place in the observability stack. OpenTelemetry, with its broad adoption and standardized approach, excels at providing high-level visibility across distributed systems. It is the go-to solution for distributed tracing, offering a cohesive view of how services interact in complex environments. For anyone trying to make sense of a multi-service architecture, OpenTelemetry provides a much-needed map.
However, when it comes to deeper, more granular insights — to understand what is happening within the system at the kernel level — eBPF is a must-have. Its strength lies in its ability to capture low-level data with minimal overhead, offering raw and detailed visibility into system performance and behavior. It is perfect for digging into performance bottlenecks, resource contention or network issues that might not be visible through high-level telemetry.
Rather than seeing these tools as competing solutions, consider them as complementary. OpenTelemetry offers you a big-picture view, while eBPF allows you to zoom in when you need to troubleshoot with precision. If you are serious about observability, combining OpenTelemetry’s ease of use and standardized distributed tracing with eBPF’s power and depth will be the key to a comprehensive understanding of your system.
Today’s IT application environment is complex and evolving quickly, especially with the integration of artificial intelligence (AI). Modern development processes and collaborative approaches like open-source code have streamlined the creation of sophisticated applications, enabling development teams to rapidly build complex systems. The proliferation of data, driven by AI, allows these apps to source vast amounts of information, further contributing to the development of advanced and impressive applications. However, as applications get more complicated, managing and monitoring them becomes increasingly challenging. One solution the industry is turning to is OpenTelemetry (OTel), an open-source framework designed to provide a consistent and comprehensive approach to collecting and understanding data.
DevOps and other modern development processes have significantly advanced how we build applications. Containerization technologies such as Docker and orchestration platforms like Kubernetes support these processes by enabling consistent, scalable and portable application environments. These modern practices enhance efficiency, reliability and scalability, allowing organizations to quickly adapt to demands and deliver high-quality applications.
Additionally, open-source code — a code that anyone can access, utilize or modify — plays a pivotal role in modern application development. As a collaborative approach, it has been instrumental in creating complex, scalable and innovative solutions that might have been otherwise impossible within a closed framework. Open-source code has enabled DevOps teams to achieve amazing feats in the world of application development. Democratizing a source code allows users anywhere to share and access useful software, but because the code is governed by its users, it also creates an environment where users can build off each other, making application advancement hypothetically infinite.
The ability to utilize open-source code and, therefore, pull from different open-source repositories has led to an explosion of progress in application development. However, significantly more advanced applications have created increasingly complex application environments, which makes it harder for teams to monitor their systems from an IT operations standpoint. In addition, AI-driven systems have added new layers of complexity. AI workflows involve real-time processing and unprecedented amounts of data, necessitating robust monitoring systems capable of managing it all.
OTel Presents a Promising Solution
Pulling from the same open-source belief, there is a solution to this challenge: OTel. Just as open-source code has fostered the creation of advanced and innovative apps, OTel, an open-source solution, exists to better manage and monitor them. OTel addresses these pain points in its ability to provide a standardized approach to collecting and analyzing telemetry data, ensuring that organizations can effectively monitor and manage the performance of these fast-moving and intricate AI-driven systems. The OTel framework supports the real-time monitoring and observability needs of AI applications, helping teams maintain the health, performance and reliability of their systems in the face of massive data volumes and convoluted processing tasks.
OTel helps manage large amounts of data generated by AI systems and enhances AI’s ability to predict and detect issues. AI systems run by processing data gathered from various sources, including telemetry data managed by OTel. When that rich telemetry data is fed into AI-driven analytics platforms, it fosters predictive maintenance, issue detection and automated remediation. In other words, by easily accessing high-quality and organized data, AI systems can learn over time what is normal for a particular IT environment and then automatically adjust and improve its responses to debug issues and maintain system performance. As a result, OTel contributes to the process of helping systems identify problems early and take action without needing constant human intervention, thereby improving efficiency and effectiveness overall.
Imagine a system that not only monitors but also predicts anomalies, diagnoses root causes autonomously and adapts to changing conditions in real-time — all powered by AI algorithms trained on vast amounts of telemetry data. With OTel, AI-driven systems can rapidly reveal and analyze billions of insights, contributing not only to an increased level of observability but to an environment where issues can be solved before ever becoming an issue.
We live in a world where modern application development processes have enabled teams to create incredibly advanced, helpful and even transformational applications. As applications become increasingly complex, the demand for sophisticated observability solutions grows in tandem. The future of observability hinges on the advancement of technologies such as open-source software and AI. While open-source software has significantly improved application development processes, it has created an environment where applications have become extremely difficult to monitor.
The combination of modern AI-driven observability solutions with OTel presents a promising solution. By standardizing the collection and analysis of telemetry data, OTel ensures that organizations can achieve robust observability, enabling them to harness the full potential of AI and to keep developing advanced applications while also being able to manage them. Due to this, through enhanced observability, OTel is going to do for IT operations what open-source software originally did for application development. It is going to transform it.
vFunction, the AI-driven architectural observability platform today added real-time observability and GenAI assistant.
vFunction today added an ability to observe IT architectures in real-time to a platform that automatically converts monolithic application code into a set of microservices.
In addition, vFunction has added a generative artificial intelligence (AI) assistant that identifies issues and surfaces recommendations as changes are based to application environments. Based on the ChatGPT-4 service created by OpenAI, the vFunction Assistant is designed to make it simpler to remediate and refactor applications.
Leveraging OpenTelemetry
vFunction CEO Moti Rafalin said these capabilities all leverage OpenTelemetry, open-source agent software being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) to collect telemetry data.
It’s not clear how many applications are instrumented using OpenTelemetry, however, a survey of 1,000 software engineering professionals conducted by vFunction suggests that as organizations look to streamline the management of application environments there is a clear need for increased visibility. A full 80% said these capabilities are either extremely or very valuable, with nearly four in 10 (39%) acknowledging they lack visibility into their IT architecture. A total of 40% said they advocate for a “shift left” approach to proactively address resiliency concerns earlier in the development lifecycle.
A similar percentage (41%) also plans to leverage generative AI to improve application performance and scalability, the survey finds.
The survey also finds more than half (51%) of respondents work for organizations that dedicate more than a quarter of their annual IT and engineering budgets to remediating applications, including refactoring and re-architecting efforts. More than three-quarters (77%) have implemented enterprise-wide initiatives to address technical debt.
Among those respondents, 57% work for organizations allocating more than a quarter of their IT budget to technical debt remediation involving monolithic applications, compared to 49% for microservices architectures. More than half 53% also noted their organization delayed major technology migrations or platform upgrades due to productivity concerns involving microservices. The survey also finds that 44% noted monolithic application complexity that results in a lack of clear domain boundaries and decreasing modularity is a major challenge.
Most application environments will be made up of monolithic and microservices-based applications for the foreseeable future. The challenge that organizations encounter is that as changes are made to application environments the level of drift away from the original architecture starts to increase.
Apply AI to Reduce Drift
At the same time, organizations accumulate technical debt as issues that should be remediated are left to be addressed at some future date. Unfortunately, many of those issues are never resolved, simply because new application development projects require DevOps teams to divert limited resources. However, as more applications are instrumented it becomes feasible to apply AI to reduce both drift and technical debt.
Each organization must determine to what degree they wish to modernize existing applications versus replacing or retiring them altogether. However, there are plenty of instances where legacy applications are still far too critical to be replaced. The challenge then becomes to what degree to update these applications, in a way that makes them more resilient and flexible as business requirements continue to evolve.
Embrace revealed today it is adding support for open source OpenTelemetry agent software to its software development kits (SDKs) that enable observability of iOS and Android applications.
Andrew Tunall, head of product for Embrace, said that as more DevOps teams standardize on OpenTelemetry to collect telemetry data from applications, there is a growing requirement for observability platforms to support it.
DevOps teams can now use the Embrace SDKs to export telemetry data from mobile applications to any backend observability platform that supports OpenTelemetry, he added. Initially, Embrace is providing support for logs and traces, with support for metrics to soon follow.
Embrace developed an observability platform initially designed specifically for engineering teams working on mobile applications. Now many of those teams are collaborating with DevOps teams as more organizations take a mobile-first approach to application development, noted Tunall. Embrace customers include Adidas Runtastic, Cameo, GOAT, Hatch, Ibotta, Marriott, The New York Times, Warby Parker, Yahoo! and Wildlife Studios.
OpenTelemetry Gaining Traction
Being advanced under the auspices of the Cloud Native Computing Foundation (CNCF), OpenTelemetry defines an instrumentation format that makes it simpler to centrally manage the collection metrics, logs and traces using open source software. It is gaining traction as more DevOps teams look to rationalize the amount of commercial agent software they need to deploy, update and manage.
Too many organizations, however, are still relying on legacy real user monitoring (RUM), error and bug tracking, and crash reporting tools to manage mobile applications that don’t provide the level of insight required to identify the root cause of a performance issue said Tunall. Observability platforms, in contrast, make it possible for DevOps teams to collect telemetry data they can query versus tracking a few simplistic metrics, he added. In the case of Embrace, the data includes signals such as crashes, errors, application not responding (ANRs), performance traces, memory issues and full user sessions.
Cost-Efficient to Centralize the Management of Applications
Mobile applications require that capability because they can be especially challenging to troubleshoot when, for example, trying to account for network latency as DevOps teams seek to achieve and maintain specific service levels to ensure a high-quality application experience is consistently delivered, noted Tunall. Most end users are not especially forgiving whenever they experience an issue because it’s often relatively trivial for them to switch to another application, he added.
It’s not clear to what degree teams that were once dedicated to managing mobile applications are being folded into DevOps teams but just about every organization that builds software now supports at least one or more mobile applications. While mobile applications have unique requirements, it’s ultimately going to be more cost-efficient to centralize the management of applications regardless of what platform they run on.
In the meantime, DevOps teams should expect the overall size of the mobile application portfolio to only continue to grow. The challenge now is melding the tools currently used to manage those applications with the DevOps platforms currently being used to manage every other type of application.
Achieving full observability across enterprise applications is challenging. While third-party services like New Relic or Datadog make end-to-end observability easier, as your application grows in complexity, so does the telemetry data, leading to increased costs. Migrating to an open source stack gives you control over telemetry data and reduces observability costs, despite challenges with existing service provider commitments.
Migrating to open source solutions is straightforward with simple architecture and tech stacks. However, transitioning the entire observability stack to open source demands meticulous planning, tools/frameworks assessments, testing, risk analysis, and cross-team communication, especially given the number of microservices, heavy cloud reliance, and diverse languages and frameworks to take into account.
A cloud observability platform usually absorbs 20%-30% of overall infrastructure spending, but in some cases, it can reach 50%-60%. If your observability expenses exceed 50%, then transitioning to open source solutions would be a sensible option to mitigate tech infrastructure costs.
For the past two years, I have been working on migrating the observability platform from a proprietary solution provider to an open source stack. Let’s dive into the key steps necessary for such a migration.
Finalize the key telemetry data and systems
Managed observability platforms offer comprehensive insights into system health, necessitating a significant volume of high-quality telemetry data. By default, the agents tasked with capturing and transmitting telemetry data are configured to gather as much information as possible, facilitating the creation of extensive dashboards and reports. However, this extensive data collection contributes to the overall cost.
Select which systems you actually need to monitor. A typical enterprise application includes databases, caching databases, container orchestrators, and numerous cloud services. Engineers often monitor more services than necessary, which leads to unnecessary complexities and costs. It’s often sufficient to streamline the monitoring scope to essential systems.
In my experience, the quantity of telemetry data required is significantly lower than what third-party services typically retrieve. You can easily divide those data collections into two buckets: must-have and nice-to-have. The “must-have” data aligns with the well-established four golden signals:
Latency: the amount of time it takes to serve a request
Traffic: the volume of requests that a system is currently handling
The error rate: the number of requests that fail or return unexpected responses
Resource saturation: the percentage of available resources being consumed
These fundamental signals serve as the foundation for key dashboards and alerts. Observability platforms built on these signals cover 80% of typical use cases.
Understanding each signal’s metadata requirements is crucial. The extent of metadata captured directly impacts the observability platform’s complexity and cost. For example, when monitoring service-to-service latency, consider the necessity for service IP addresses, instance IDs, or header information.
Select the relevant stack
Managed observability platforms such as New Relic and Datadog provide comprehensive monitoring for various enterprise components. However, when you migrate to an open source stack, you must assess and integrate the tool stacks to meet diverse monitoring requirements.
One important aspect to consider is scaling: how to manage the huge amount of data generated every minute across all those systems. Focus on two fronts: selecting the stack for storing and processing telemetry data (logs, metrics and traces) and devising methods for capturing and pushing telemetry data from diverse systems.
Telemetry signal stack selections
Stack for logs
You require a stack that can efficiently and economically process and store the substantial volume of logs generated by the system. In the past, I used the ELK stack for storing and searching logs, but it is a generic solution and not tailored specifically for logs.
I recommend Grafana’s Loki for its effective management of large log volumes and LogQL, a language akin to PromQL. If you are familiar with PromQL, navigating through extensive log data with LogQL is easy and intuitive.
Stack for Metrics
Prometheus is a popular database for storing time-series metrics data. However, it has scaling-related limitations; it cannot scale horizontally. Alternatives like Thanos, Grafana’s Mimir, and VictoriaMetrics offer better out-of-the-box solutions for horizontal scaling.
I conducted some research and ultimately chose Grafana’s Mimir to store metrics in a similar migration project. This decision was based on Mimir’s remote storage capability, as well as its scalable and highly available architecture.
Stack of Tracing
Distributed tracing is a must in a microservices architecture, for easily identifying bottlenecks in latency and for resolving performance-related bugs. Grafana’s Tempo could be a good option for several reasons:
It can seamlessly integrate as a backend for Grafana’s dashboard. Consolidating dashboards in Grafana helps avoid navigating between multiple applications. Switching platforms just for tracing while using Grafana for other aspects of observability would not be practical.
Tempo provides a similar experience to TraceQL when interacting with the database, streamlining the learning process and significantly reducing the learning curve.
Tempo is compatible with popular open source tracing protocols such as Zipkin and Jaeger. If any team already uses these protocols, transitioning to Grafana’s Tempo would be smoother.
Capturing and pushing telemetry data
You have to install an agent on the system to capture and push the telemetry data. Those steps vary based on the system being monitored. Typical enterprise application setups involve monitoring cloud integrations, process monitoring, infrastructure hosts, Kubernetes clusters and application monitoring. Among the possibilities:
Cloud integrations: This includes monitoring for cloud services such as SQS, SNS, EMR and EC2.
Process monitoring: This involves monitoring processes running on bare-metal machines. With the advent of dockerization and Kubernetes, there is now a standardized way of starting the application. In the past, there was no fixed mechanism. For example, to run a Java application, you could use a Java application, the ‘java -jar’ command, Tomcat, or OS systemctl. In the case of node, it could be npm or PM2. Each team or service may have had its own way of starting the process.
Infrastructure host: This entails monitoring the machine itself for metrics such as CPU usage, memory, disk I/O and Network IO or checking if the machine is offline.
Kubernetes monitoring: You regularly need to monitor a Kubernetes cluster, such as instances where Kubernetes is unable to schedule a pod due to insufficient resources.
Application monitoring: This monitoring focuses on overseeing the services the team created. Each service may differ in its development approach and choice of tech stack; but from an observability perspective, they usually are treated the same.
Browser and mobile monitoring: These metrics ensure optimal performance and user experience across different platforms. Browser monitoring includes tracking page load times, rendering performance, JavaScript errors and resource usage. For mobile, it includes monitoring app crashes, latency, battery usage, network requests and device-specific metrics.
Before the advent of OpenTelemetry, there was no standardized method for monitoring applications. OpenTelemetry emerged from the merger of two earlier projects, OpenTracing and OpenCensus. It stands as a vendor- and tool-agnostic framework to instrument applications or systems irrespective of language, infrastructure or runtime environment. OpenTelemetry represents a significant community effort, and its popularity and stability are steadily growing.
Validate observability stack on application architecture
Applications vary in architecture and development stages. To address this diversity, you need multiple proofs-of-concept (POCs) across different systems, such as infrastructure, cloud service and backend and frontend applications.
Migrating an observability platform from one tech stack to another is simple when the application architecture is a monolith. However, when the underlying architecture is microservices, it becomes difficult.
While the microservices architecture offers flexibility in choosing different tech stacks for building services, devising a standard solution for capturing telemetry data becomes challenging due to the diverse range of tech stacks involved.
Conducting POCs for various tech combinations is necessary to build a self-service tool that individual teams can follow to migrate their services easily. For instance, if your service uses Java 11, Spring Boot 3.x.x and PostgreSQL, these POCs can provide standardized steps to enable application monitoring.
Migrate core components
Setting up the observability platform involves configuring the stack for metrics, logs and traces, as well as installing the necessary agents. One advantage of migrating the observability platform is that there’s no need to transfer old telemetry data. During the testing phase, you will have an overlapping period to accumulate enough useful data to create specific dashboards and alerts.
Migrating alerts and dashboards to the new system is essential, even when data migration is not necessary. This is because changes in the underlying techniques for capturing metrics may modify the names of the metrics and subsequently change all query expressions used in alerts and dashboards. Some metrics may not be available in certain scenarios, especially derived metrics that can be recreated from underlying metrics.
While migrating query expressions manually is an option, a manual process is error-prone and time-consuming for a large number of alerts. I had experienced a similar situation with alert numbers ranging from 100 to 400.
To streamline the migration, we developed a Node.js script that programmatically converted New Relic’s query expressions to Prometheus expressions. The script performed the following high-level steps:
Connected to the New Relic API server and fetched all the alerts configured for cloud integration
Converted New Relic alert expressions into PromQL alert expressions
Wrote all the Prometheus alert expressions into the YAML file
Implementing this technique allowed us to successfully migrate all alerts and dashboards in four or five days, a process that would normally demand a month’s manual labor. Additionally, scripting significantly minimized the risk of human error.
Test the migration
Verify each alert and dashboard. Run both old and new platforms during testing. Parallel operation permits thorough testing of the metrics.
However, running both systems concurrently may lead to a performance hit due to services sending telemetry data to two locations. Additionally, there may be a slight increase in costs until the complete transition.
Running both systems in parallel provides an ideal setup to ensure the migrated dashboards and alerts are functioning correctly.
Testing alerts that trigger during this process is straightforward, but evaluating those that have not yet triggered poses a challenge. In one project, I reduced the threshold value of each alert and tested its functionalities. Approximately 90% of the alerts generated through custom scripts worked seamlessly, with only 10% requiring some manual tweaking.
Migrate other related components
Other systems depend on the data generated by the observability platform. For example:
Notification systems like email, Slack channel, etc., that alert in case of an incident
Incident management tools like PagerDuty provide a streamlined way to handle incidents.
These systems rely on the alert’s payload to work properly. Their assets should be updated, as a change in the observability platform leads to a change in the alert payload. Update the templates of notification systems to ensure they seamlessly integrate with the new alert payload.
Similarly, incident management tools like PagerDuty require modifications to routing rules, escalation policies, and scheduling. Fortunately, readymade open-source migration tools are available to facilitate easy migration from PagerDuty to Grafana’s OnCall. However, in other solutions, you may not have access to out-of-the-box migration tools. In such cases, manual migration or script writing may be required.
Timeframe for the Migration
Planning a migration involves finalizing key telemetry data and systems, selecting relevant stacks. and validating the observability stack on the application architecture. It typically takes around three weeks.
The project duration depends on factors such as the number of teams involved and the quantity of active services. For instance, a migration to open source with 100 microservices and involvement from 10 different teams may take four months to complete.
Migration Complete!
Transitioning your observability platform to an open-source stack offers a promising avenue for cost reduction and increased control over telemetry data. However, accomplishing this migration demands meticulous planning and execution, encompassing essential steps like feature prioritization, stack selection, POCs, core component migration, testing, and the migration of related systems. Despite the challenges posed by diverse architectures and technologies, a systematic approach, comprehensive assessment, and collaboration among teams can facilitate a smooth migration process and guarantee its success.
This week at the KubeCon + CloudNativeCon Europe conference, the maintainers of the open source OpenTelemetry observability agent software project being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) revealed they have added profiling capabilities to enable DevOps teams to identify the root cause of issues down to a specific line of code.
Austin Parker, director of open source for Honeycomb and a maintainer of OpenTelemetry, said while several providers of IT observability platforms have in the past year added code profiling capabilities, it makes more sense to provide this as a capability surfaced by OpenTelemetry in a way that any IT platform can easily consume.
The overall goal is to reduce the amount of time it takes to troubleshoot IT issues by identifying not just the code at issue but also the development team responsible for it, noted Parker.
OpenTelemetry defines a format that makes it simpler to centrally manage the collection metrics, logs and traces using open source software. In general, OpenTelemetry continues to gain traction as DevOps teams look to rationalize the amount of commercial agent software they need to deploy, update and manage.
In effect, OpenTelemetry is aggregating agent software that, in terms of observability, provides no differentiated value, said Parker. That’s critical because as application environments become more complex, it’s all but impossible to manage them without being able to broadly apply the agent software required to instrument them.
Ultimately, as OpenTelemetry becomes more widely employed, it makes it significantly easier for developers to understand how their application is performing, noted Parker. Over time, armed with those insights, developers will build and deliver higher-quality applications, he added.
It’s still early days as far as adoption of OpenTelemetry is concerned, but it’s clearly well on its way to becoming a de facto standard for enabling observability. The number of developers who have access to not just metrics and logs but also traces should steadily increase as the cost of instrumenting code declines.
Of course, finding a way to collect that data inexpensively will also be crucial for applying artificial intelligence (AI) to DevOps workflows. After all, before an AI model can be trained, it needs access to verified data.
In the meantime, DevOps teams might want to consider at what rate they can replace existing commercial agents deployed across the enterprise. Much of that agent software is more mature than OpenTelemetry, but over time, the economic arguments for using open source agent software become compelling. Many of the providers of observability platforms have already made support for OpenTelemerty available by default.
Regardless of approach, the need for observability has always been apparent. The issue is that there is a world of difference between monitoring a pre-defined set of metrics and being able to launch queries to identify the root cause of an issue that metrics alone are not going to be able to surface.
The biggest compliment you can pay a technology standard is to say that it is “boring.”
Transport layer standards like TCP and UDP are boring. They are the underlying protocols powering pretty much everything we do on the internet. REST is boring. It brokers trillions of API transactions across the world’s distributed microservices.
Infrastructure standards get boring when they become common infrastructure plumbing that “just works.” They give open source creators and maintainers common foundations to innovate on top of. They give users confidence that their technology choices aren’t leading to dead ends. Look at any major evolution in computing, and it’s always propped up by boring standards.
When OpenTelemetry debuted at KubeCon in 2019, telemetry data for applications in distributed systems was far from boring. Two competing projects—OpenTracing and OpenCensus—were trying to encapsulate best practices around the still nascent distributed tracing discipline. They were “very similar in that they aim[ed] to unify app instrumentation and make observability a built-in feature in every modern application” (Merging OpenTracing and OpenCensus: Goals and Non-Goals). But having two fragmented communities around these two projects made it hard for distributed tracing to move forward—so it was a big deal when the two projects merged and OpenTelemetry was born.
Let’s take a look at how the OpenTelemetry standard is making this all possible.
Standard Semantic Conventions and a Single ‘Mental Model’
Semantic conventions are important to standardize how we describe common operations so that we have consistent telemetry for similar actions. Developers instrumenting applications need to be able to relate to clear semantic definitions to understand how to instrument everything from database drivers to SQL statements and how to consume that telemetry data for analysis.
For example, OpenTelemetry semantic conventions define where in the metadata the process name should be stored, but not what the process name should be. Similarly, if I’m deploying my application on Kubernetes on a cloud provider somewhere, I’d expect my telemetry data to have the Kubernetes pod and namespace it originates from, as well as the cluster and region.
OpenTelemetry also standardizes metadata for labels that span languages. So if I have an HTTP service and request, it doesn’t matter if it is in Java, Python or any other language — with OpenTelemetry, I can just start querying the data without having to think about the language behind it.
It brings a consistent mental model to how the different telemetry data types (traces, metrics, logs) are related: OpenTelemetry gives them the same base, so understanding how a span is modeled provides knowledge that can be reused when understanding metrics and logs. It can be used by any microservice and any HTTP- or gRPC-connected service or device. For any network device, OpenTelemetry gives developers semantic conventions that bring consistency to reporting telemetry data on any type of event. So, any engineer working with OpenTelemetry can compare latencies and queries between services—even if they are written in different languages—because of the semantic conventions.
Protecting Users From Vendor Lock-In
A decade ago, monitoring and logging vendors provided agents to their customers. They didn’t call it observability back then. But they told the market that if you want the best telemetry data, you need to use our agent. Every vendor had its own agent.
Adding these agents was risky and took a lot of time. Once a vendor got its agents into a customer’s environment, they were hard to change and you had to either pay the vendor or your team of engineers to remove that instrumentation in favor of another.
OpenTelemetry really commoditized the telemetry definition in such a way that we don’t need to use agents from vendors anymore. This means users can decide which vendor to use for observability tooling and have the flexibility to switch to different vendors down the line. With OpenTelemetry, they no longer have to instrument applications, restart applications and get into risky operations cycles to change the destination of their telemetry data.
Economies of Community – Scale for Drivers and Integrations
OpenTelemetry has achieved a very large and diverse set of contributors in a relatively short period of time. It’s become the second most popular CNCF project, behind only Kubernetes.
The project is drawing contributions both on the instrumentation side of things, as well as the collection side of things. Specifications are another big area where the project is evolving, including the API/SDK spec, the data model and the OTel protocol (OTLP). Accelerating all of this is the fact that OpenTelemetry has broad participation from the logging, monitoring and observability vendor ecosystem, which comprises many of its largest contributors.
We still have a ways to go. But in the last year, it’s become obvious that OpenTelemetry is so commonly regarded as the standard for tracing data that the language, library and database creators and maintainers are instrumenting for OpenTelemetry. This is the ultimate sign of maturity for any project or standard—when its popularity has created pull from maintainers designing around the spec and using it natively.
What’s Next For OpenTelemetry?
Looking to the future, I think we’re going to see an unintended consequence where OpenTelemetry moves the industry from operational data to more data intelligence. OpenTelemetry makes it much easier to instrument how users are interacting with systems and enables developers to add business metrics and compare those against actual user behavior.
OpenTelemetry also has a great potential marriage with eBPF and related projects like Cilium. These abstractions that make the Linux kernel “programmable” add a pool of much finer-grained telemetry data that can be extracted into logs from applications during run-time.
This all feeds into the concept of “Profiles” that OpenTelemetry is implementing. While distributed tracing emphasizes the horizontal cut of how a specific transaction is running or all the services that are tied through a single request, a profile is a different vertical cut that emphasizes the state of resources like memory, CPU and more at a point in time. Those profiles are collections that are stored continuously and another great slice of telemetry data that will be encapsulated by the OpenTelemetry standard.
It was a rainy day in Seattle at KubeCon + CloudNativeCon North America in December 2018 when I first encountered the term ‘OpenTelemetry.’ At that time, I was an active member of a working group focused on developing W3C Trace Context, a standard now extensively employed for context propagation in distributed systems. It is likely no coincidence that some key individuals in this working group also played a pivotal role in orchestrating the merger between two prominent open source telemetry products: OpenCensus and OpenTracing. I never would’ve guessed that, in 2023, OpenTelemetry would become the second-largest project in the CNCF, sitting just behind Kubernetes.
As is the case with any popular new technology like OpenTelemetry, the hype can make it difficult to discern signal from noise. During my years working in observability, I’ve had the chance to chat with many organizations that were thinking about using OpenTelemetry or already actively use it. Through those conversations, I’ve heard many reasons for adopting OpenTelemetry, some good and some bad. In this post, I will share examples of the best and worst reasons to adopt OpenTelemetry to help you cut through the noise.
Poor Reason One: Following the Crowd
It’s never a good idea to adopt something solely because others are doing it. Just because you saw a conference talk praising a new technology as the greatest innovation since sliced bread doesn’t necessarily mean that technology is suitable for your specific use case.
Speaking of which, have you clearly defined your use case? To start, ask yourself the following three questions:
1. Are you able to determine at any given moment whether your systems are performing as expected, such as meeting their service level objectives (SLOs)?
2. When something goes wrong, can you easily pinpoint the root cause?
3. Are you able to drill into previously unidentified problems, also known as “unknown unknowns”?
If you answered these questions with “no,” you certainly have observability gaps worth tackling, and OpenTelemetry may help close some of them.
Poor Reason Two: Building Your Own Observability Product
Gathering telemetry data can be a challenge, and with OpenTelemetry now handling essential signals like metrics, traces and logs, you might feel the urge to save your company some cash by building your own system. As a developer myself, I totally get that feeling, but I also know how easy it is to underestimate the effort involved by just focusing on the fun parts when kicking off the project. No joke, I’ve actually seen organizations assign teams of 50 engineers to work on their observability stack, even though the company’s core business is something else entirely.
Keep in mind that data collection is just a small part of what observability tools do these days. The real challenge lies in data ingestion, retention, storage and, ultimately, delivering valuable insights from your data at scale. Chances are, a homegrown solution won’t give you a return on investment that your company would appreciate.
However, whether you build your own system or use an existing product, OpenTelemetry may be a good fit for your observability stack.
Poor Reason Three: ‘Vendor Neutrality’
People I chat with often cite vendor neutrality as a major reason for their interest in OpenTelemetry. While there’s some truth to this, which I’ll touch on shortly, it’s important to take this argument with a grain of salt to avoid over-promising your management.
If you’re a developer, you’ve likely worked with database abstraction libraries and have probably mentioned to someone that they allow you to swap out the database at any time–using MySQL today and Oracle tomorrow, for instance. While that’s the promise, I’ve never seen this happen in my career. There’s more to consider other than the SQL dialect when switching a database backend.
In the same vein, the idea of vendor neutrality suggests that you can easily switch your observability backend. While this might be true in theory, in practice, vendor lock-in often occurs on the backend, where people and processes become closely tied to the tool currently in use.
So, if you believe you’ve achieved vendor neutrality by going all-in with OpenTelemetry, I’d encourage you to think twice and consider building a stronger case. Vendor neutrality should not be your only reason.
With all that said, it’s time to focus on the good reasons to use OpenTelemetry today. Trust me, there are plenty, and they’re definitely worth considering.
Good Reason One: Bridging Gaps With Manual Instrumentation
Nearly every observability tool out there today relies on some degree of auto-instrumentation. This means once you’ve added their respective language SDK to your code, the library will utilize a technique called “monkey patching” to insert observability code that runs automatically.
At the very least, auto-instrumentation will search for recognized libraries and APIs and then add some code to indicate the start and end of well-known function calls. Additionally, auto-instrumentation takes care of capturing the current context from incoming requests and forwarding it to downstream requests.
However, auto-instrumentation falls short when it comes to pure userland code, particularly for dynamic languages like JavaScript. It won’t catch your blocking Fibonacci function that’s slowing down all your requests by 50%. In some cases, you might also have to add your own context-propagation logic to RPC calls not captured by the auto-instrumentation.
That’s where manual instrumentation comes in. In the past, vendors provided an API for their SDK, allowing developers to manually add instrumentation code to segments that weren’t auto-instrumented. This often resulted in code being littered with vendor-specific instructions. This brings us back to the topic of vendor neutrality. While adding a dependency to your project is usually no big deal, it becomes cumbersome when 20% of your codebase is made up of vendor-specific observability instrumentation.
OpenTelemetry addresses this issue by offering both auto-instrumentation and a vendor-neutral API for manual instrumentation. This allows observability vendors to access the data while maintaining vendor neutrality within your codebase.
Good Reason Two: Sending Data to Multiple Vendors
I’ve yet to come across a large organization that relies on just one observability tool. Different parts of the organization have unique needs and perspectives when it comes to data, and it’s rare that a tool designed for SREs satisfies all use cases developers have and vice versa.
At the same time, it’s not practical to instrument code specifically for each vendor. Doing so would lead to performance issues and other problems, such as double-tracing. For example, if multiple vendors modify an outbound HTTP request to add context, which one takes precedence?
OpenTelemetry addresses this by offering a unified way to instrument code and multiple integration points where vendors can capture the collected data and send it to their respective backends. As a result, a single application can send data to multiple observability backends with a minimal added performance impact.
Good Reason Three: Gaining Insight Into Third-Party Libraries and Services
In an era where many applications depend on third-party libraries and services, gaining insight into all your application dependencies has become increasingly difficult. It’s also challenging for observability vendors to keep up with every new library and service that’s released.
Traditionally, vendors would try to reverse-engineer these services and apply the aforementioned monkey patching to add in their code. However, since every vendor must do this for each service, it ends up being a lot of redundant work. A key goal of the OpenTelemetry project is to make it easy for infrastructure and library vendors to add instrumentation hooks to their code. This way, their offerings can be monitored without the need for reverse engineering. The observability community is gradually making progress on this goal. One successful example is Istio, which emits OpenTelemetry data that can be easily consumed by observability backends to understand what’s happening inside the service mesh.
Unsurprisingly, cloud vendors like AWS, Microsoft and Google are among the founding members of the OpenTelemetry project. They share a strong interest in defining a format that allows observability vendors to seamlessly consume data from cloud services. OpenTelemetry Protocol, or OTLP, is set to become the standard for emitting observability data from these services that would otherwise be black boxes.
Wrapping it Up
“If all you have is a hammer, every problem looks like a nail.” This statement holds especially true when selecting tools in the tech industry. While there are many good reasons to adopt OpenTelemetry, it’s crucial to approach technology choices pragmatically.
Avoid jumping on the bandwagon for its own sake; instead, carefully evaluate your use case and assess the available tools. OpenTelemetry is undoubtedly here to stay, and if you’re new to the topic, taking a closer look at the project and adding it to your toolbelt makes perfect sense.
And don’t forget: OpenTelemetry is open source. Your contribution is always appreciated.
The software landscape has shifted significantly in recent years as companies digitize their operations and adopt cloud and microservices technologies. The complexity of modern software systems has led to an increasing need for real-time visibility into their performance—resulting in an explosion of telemetry data that is stretching IT budgets and outpacing engineering teams’ ability to find value in that data. In fact, data volume is growing at 23% yearly, while IT budgets are only growing at 5%.
Managing all this data at increasing cost is further compounded by unpredictable software failures and ever-evolving security threats. As a result, companies are forced to pick which parts of their software to instrument and monitor. To overcome this challenge, a different approach to managing data is required.
Observability pipelines are emerging as a way to address this problem. They centralize telemetry data (i.e., logs, metrics and traces) from many sources, transform that data to fit downstream needs, and route it to various destinations for analysis. With complete control of the data, companies can sort through large amounts and prioritize what is essential, allowing them to act swiftly to avoid disruptions while reducing costs by only storing the data they need.
Research indicates 62% of organizations currently have an observability practice. This means 38% operate with limited awareness of what’s happening in their software. Even those with an observability practice can only afford to store some data and have to choose which applications to instrument. This leads to blind spots and, ultimately, issues that are time-consuming to fix due to a lack of data.
Managing Limited Resources With Observability Pipelines
Engineers face more demands in today’s fast-paced business environment than ever before. Despite an increase in their responsibilities, the size of engineering teams has yet to grow accordingly. Given this, there has been a rise in full-stack engineers responsible for technologies from the front end to the back end. Short on time and resources, it’s difficult for these engineers to instrument and monitor software.
Observability pipelines help control the amount of telemetry data using various processors such as sampling, throttling, filtering and parsing, and only forward valuable data to the downstream systems. The rest of the data can either be discarded or stored in a low-cost system, such as Amazon S3. This reduces costs and lets engineers focus on analyzing relevant data to identify issues.
Engineers can also use observability pipelines to consolidate data from various tools. Many teams use different platforms or open source tools, like Prometheus or Jaeger. The fragmented data makes it hard to solve issues and understand application health. Observability pipelines combine all this data to make it easier to act on.
It’s also important to remember that real people work on these technical challenges. Engineers face burnout working long hours to keep up with software development demands. Constantly switching between tools and manually moving data adds to the pressure. Moreover, the number of data sources, observability tools, and platforms that need that data are constantly increasing. Embracing open standards such as OpenTelemetry can ease this integration burden. Engineers can swap out tools without extensive testing and integration. The increased control and easy integration enable engineers to focus on delivering business value.
Making Data More Usable
Teams across organizations want access to data but struggle to use it effectively. The difficulty stems from the lack of data usability: Telemetry data is unstructured; varying formats make it hard to use; data preparation is time-consuming and sensitive data in logs may lead to compliance violations.
By most estimates, 80%-90% of data is unstructured, which makes it hard to store and analyze because it doesn’t adhere to traditional data models. Observability pipelines make sense of unstructured data before it reaches its final destination. This is achieved using data processors that shape and transform data to make it more actionable. These processors include advanced parsers that identify and extract relevant information, transforming data into consumable formats that make it easier to work with and analyze.
The advantage of performing these operations within the pipeline is that the same data can be prepared to fit different use cases downstream. For instance, while one team may require data optimized for visualization and trend analysis, another may require complete data for threat hunting. These transformations are managed from a single control point, eliminating the need to maintain separate data streams.
Observability pipelines can also help companies meet data compliance requirements. For example, teams can scrub, mask or redact PII data based on a defined key structure before the data reaches a SIEM or audit platform.
Taking Immediate Steps to Make the Most of Telemetry Data
Organizations can take immediate action to make the most of their telemetry data. First, they need to adopt a visibility-first rather than a cost-first approach. They should assess what software they’ve instrumented. This will identify gaps in telemetry, which can be used to develop a plan to instrument all software. Next, they need to create a strategy for managing their telemetry data, being particularly thoughtful about how and why the data is stored and analyzed.
Once a company has taken these two steps, it can decide which tools to use. One of the great things about OpenTelemetry is that it provides flexibility for future decisions about what tools to use.
As companies invest more resources into building observability practices, they’ll encounter problems processing an extraordinary amount of telemetry data for any meaningful action. Observability pipelines provide a competitive advantage of prioritizing the essential data that will enable companies to make better decisions faster.
