I recently returned from KubeCon EU where I participated in the first-ever OpenTofu Day and an Evolution of Infrastructure-as-Code panel discussion. During the conference, dozens of engineers and tech execs asked me questions about the OpenTofu project. They wanted to ascertain if it made sense for them to switch to OpenTofu, whether for testing purposes or to make the IaC management tool a permanent fixture in their tech stack.
Many people shared similar questions and concerns which, notably, were still the same as those expressed during the GA release. Now, on the heels of the OpenTofu v1.7 release, I’m taking the time to summarize my answers to those frequently asked questions and to share my (understandably biased) insights on the advantages of adopting it.
What Does “Drop-In” Replacement Mean?
In our communications, we commonly refer to OpenTofu as a “drop-in replacement” for Terraform. Unsurprisingly, the most common question I was asked was, “What does ‘a drop-in replacement’ really mean?”
As a developer, I understand where such questions come from. “Drop-in replacement” sounds more like a marketing promise than a technical specification. With that in mind, let me clarify what we mean.
Use the tofu command to execute your previously terraform commands (e.g.,tofu plan instead of terraform plan, tofu apply instead of terraform apply, and so forth).
And that’s it.
You don’t need to change a single line of code in your Terraform configuration. You can continue using all your favorite providers and modules, public or private. They will all just work.
By “drop in” we mean literally the minimal possible amount of effort with the fewest possible adjustments.
What’s the Best Time to Make the Switch?
Another question I encountered, often implied rather than asked directly, was, “Should I start with OpenTofu now or wait a while?”
Setting aside my personal bias, I can sum up my answer with a simple, undeniable fact: No future scenario is likely to make the switch easier than it is right now.
Let me explain.
The ease of transitioning between OpenTofu and Terraform comes from their complete interoperability in 1.6 stable versions. The ability to switch between Terraform and its open-source successor with a simple command alias means there’s effectively no cost to trying OpenTofu.
To me, this suggests that if you’re considering giving OpenTofu a try—whether sooner or later—the best time to do so is now.
Will the Projects Diverge?
Inevitably they will. However, there’s more to consider.
Ease of migration is a top priority for us on the OpenTofu team, and it is equally important to our community. This commitment ensures that we will manage any divergence between versions carefully.
Take OpenTofu v1.7 as an example. This version introduces several features, such as the much-anticipated state file encryption. But importantly, all these additions are opt-in. This approach exemplifies our commitment to not locking users into OpenTofu against their preferences.
Starting with OpenTofu is essentially a low-effort trial. The worst-case scenario is that you end up with a free, open-source version of Terraform that includes some additional, optional features to explore.
This was the mindset of several people I spoke with at KubeCon who were adopting OpenTofu as a future-proofing strategy. Doing so would let them keep their options open and help them avoid being locked into any specific vendor-ecosystem or IaC management tool.
What About Community Support?
Support for the project is gaining momentum. Key players including Alpine, Brew, and GitLab have already moved away from BSL-licensed Terraform binaries to ensure long-term open-source interoperability and to address their unique needs.
These are just a few examples. The shift in HashiCorp’s licensing terms, compounded by its recent acquisition by IBM, has sparked concerns about long-term viability and compatibility. These are unlikely to go away anytime soon. For many people in the community, this positions OpenTofu as a logical and straightforward path to return to business as usual.
Moreover, adopting OpenTofu also serves as a way to tap into the wellspring of community contribution that dwindled for Terraform following the licensing change.
As a vendor-neutral Linux Foundation project, OpenTofu remains true to Terraform’s original open-source ethos, drawing on the collective wisdom and diversity of the FOSS community.
This commitment to openness, reinforced by the project’s transparent governance through a multi-vendor technical steering committee, ensures that community innovation can flourish.
But don’t just take my word for it. Form your own opinion by joining our Slack community (2,100 members and growing), or follow the discussions on our GitHub repo. You can also keep up with our project updates and engage in open discussions by following us on Twitter or LinkedIn, and by joining our office hours.
Looking Forward
Switching to OpenTofu isn’t the right move for everyone. For example, if you’re deeply embedded in Terraform Cloud, it may be more practical to stick with HashiCorp.
However, the undeniable fact is that currently, the effort to experiment with OpenTofu is as low as it’ll ever be.
So, if you are looking for a true open-source project backed by a vibrant and enthusiastic community, or you just foresee a future where OpenTofu would fit into your tech stack, the best time to dip your toes in the water is right now.
Oracle today revealed it is developing a generative artificial intelligence assistant that leverages multiple large language models to create Java and SQL code, in a way that is tuned to an organization’s existing code bases and best practices that an organization employs.
Vijay Kumar, vice president of product marketing for application development services and developer relations for Oracle, said Oracle Code Assist will also enable DevOps teams to create code for provisioning IT infrastructure resources hosted on the Oracle Cloud Infrastructure (OCI) service.
Scheduled to become available at an unspecified future date, Oracle Code Assist will also be designed to update, upgrade and refactor code written in most modern programming languages. In addition, upgrading from older versions of programming languages will become more automated.
Pinpointing Bugs
Deployed as a plugin for integrated development environments (IDEs) such as JetBrains, IntelliJ IDEA, or Microsoft Visual Studio Code, Oracle Code Assist will be specifically trained to provide expert, opinionated feedback such as code snippets, dependency analysis, error mitigation alternatives, test cases, annotation, summarization and documentation that will make it simpler to pinpoint the precise location of bugs within an application, noted Kumar.
Benefits will include code suggestions based on learned context and patterns to reduce the time spent writing boilerplate code, summarizations of existing code, increased text coverage and faster code reviews.
Finally, any code generated will not be sourced from examples lacking non-permissive licenses, to minimize potential compliance issues.
In effect, Oracle is committing to training LLMs to generate code using samples it has vetted to eliminate hallucinations that occur when general-purpose LLMs such as ChatGPT are invoked. A general-purpose LLM is trained using samples of code of varying quality, pulled from across the web. As a result, they are more likely to create code that, for example, contains known vulnerabilities.
One immediate benefit of Oracle Code Assist should be an acceleration to a more recent version of Java. Oracle most recently made available Java 22, but many organizations are still running older, slower versions of Java that are less secure. Conversely, however, generative AI is also being used to accelerate the conversion of Java into other programming languages.
Ultimately, DevOps teams will be able to take advantage of Oracle Code Assist to create an automated feedback loop, noted Kumar. Additionally, he added, onboarding developers to existing application development projects will become much easier.
It’s not clear whether senior or junior developers will benefit most from generative AI. In theory, junior developers should be able to take on more complex tasks. However, senior developers may find that many of the manual tasks that today conspire to reduce their productivity are being eliminated.
One way or another, the overall pace at which applications are being developed and deployed is thanks to the rise of generative AI is now being accelerated. The only thing left to determine is how best to ensure the quality of the code being generated meets the requirements of enterprise IT organizations that today depend more on software than ever before.
devopsy.kinsta.cloud provides a weekly DevOps jobs report highlighting opportunities for DevOps professionals. This effort aims to better serve our audience and make it easier for DevOps professionals to advance their careers in these challenging economic times.
Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role, it tends to create opportunities for others.
The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment, and, naturally, the pay scale being offered.
We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration.
Announced at a POST/CON 2024 conference, Postbot is included with version 11 of the Postman platform which also adds workspaces that make it simpler for software engineering teams to collaborate.
Postman CEO Abhinav Asthana said the overall goal is to improve the quality of the APIs being developed while at the same time reducing the level of friction that occurs as APIs are developed and updated.
Developers are already making extensive use of AI tools to write code, but Postman is using a combination of open-source large language models (LLMs) and LLMs from Open AI and Microsoft to automatically generate tests and documentation based on the data its platform collects. In addition to making it easier to create API tests using a conversational interface, the overall quality of those tests will improve using LLMs specifically trained using Postman data, noted Asthana.
Developers Need to Rely on AI Tools
That’s critical because the ratio between developers and the number of services that expose APIs is much too high, said Asthana. Developers will need to rely more on AI tools to create and manage APIs as the number of services they are expected to manage continues to steadily increase.
It’s not always clear who within organizations is managing APIs. Developers undoubtedly create APIs, but the management of APIs might be assigned to a DevOps team. As more organizations embrace platform engineering as a methodology for managing DevOps workflows at scale the more likely it becomes APIs will be managed centrally.
Ultimately, the rise of microservices-based applications has dramatically increased the number of APIs that need to be managed. APIs play a crucial role in making data available externally to other applications. However, the bulk of APIs being created today are for integrating all the microservices that make up a modern application environment. Those largely internal APIs are increasing at a pace that makes it challenging for individual developers to manage at scale. As a result, many organizations now have a significant API change management challenge, noted Asthana.
Each organization will need to determine to what degree the management of microservices is sustainable. Rather than getting caught up in philosophical debates, many organizations are deploying a mix of microservices alongside monolithic applications that also expose an API. In effect, monolithic applications are becoming large microservices. Regardless of approach, the number of APIs, including rogue and zombie APIs that organizations either didn’t realize were built or have simply been forgotten, is dizzying. In many ways, APIs in the absence of a management framework quickly become too much of a good thing.
There are multiple ways to manage APIs but the one thing that is critical to remember is the approach taken needs to integrate well with the tools used to create those APIs in the first place.
Today, Snyk made available an edition of its application security posture management (ASPM) tool for assessing application risks that provides more context into how code has been written and its role within the application environment.
Manoj Nair, chief product officer for Snyk, said Snyk AppRisk Pro leverages artificial intelligence (AI) and machine learning to provide deeper insights into how applications have been constructed. Snyk AppRisk Pro, for example, can trace insecure portions of deployed applications all the way back to the specific code components.
Armed with those insights it then becomes simpler for DevSecOps teams to prioritize their remediation efforts, said Nair.
Snyk originally released Snyk AppRisk late last year and subsequently acquired Helios, a provider of tools for collecting security data from runtime environments via integrations with third-party tools and platforms from Dynatrace, SentinelOne and Sysdig, last January. That capability has now been integrated into Snyk AppRisk Pro.
In addition, Snyk AppRisk Pro also adds the ability to scan for secrets in code that cybercriminals might later discover.
Snyk Working With Google
Snyk has been working closely with Google to apply AI to DevSecOps. Most recently, Snyk announced its support for Gemini Code Assist, a tool for generating code using the Gemini large language model (LLM) created by Google. The integration with Snyk helps identify security vulnerabilities in code before applications are deployed. That’s critical because LLMs are trained using examples of code of varying quality from across the Web. As more code is created using generative AI tools, the probability of vulnerabilities in that code is high because the code used to train an LLM is often flawed. In effect, DevSecOps teams need tools that use AI models to identify vulnerabilities created by other AI models.
Hopefully, as generative AI continues to advance the overall quality of the generated code will continue to improve. The next generation of LLMs is being trained using code that has been vetted for quality. That’s crucial because most of the issues that cybersecurity teams usually need to resolve start with mistakes made by developers that cybersecurity teams then need to convince developers to allocate time to fix. Thanks to the rise of DevSecOps best practices, the overall security of software supply chains is improving. The simpler it becomes for developers to identify issues as they write code the more secure applications will become.
Of course, writing more secure code is only one element of the overall equation. DevSecOps teams will also need to continue to scan code after it’s been added to a build and as updates are made to applications already deployed in production environments. It’s now only a matter of time before more stringent regulations are implemented that will require organizations to attain and maintain higher levels of application security.
The challenge, as always, is modernizing legacy DevOps workflows that typically were designed to build and deploy code as fast as possible rather than ensuring application security requirements are being met.
Pia Nilsson, senior director of engineering for Spotify, said Spotify Portal for Backstage, available as a private beta today, provides platform engineering teams with low-code/no-code framework for setting up an IDP.
In addition, Spotify is now making enterprise support available for organizations that adopted Backstage, along with additional plug-ins that simplify integrations with tools and platforms from third-party vendors such as New Relic, Atlassian, Snyk and Datadog.
Finally, Spotify is adding capabilities to manage hackathons and surface usage insights. A forthcoming plug-in also promises to make it possible to add data entities to a software catalog residing in Backstage.
Backstage has been gaining traction as an IDP mainly because it provides an extensible framework for centralizing the management of developer environments.
Spotify Portal Implementation
Some organizations have found it challenging to build and deploy Backstage. So Spotify is now making available a more opinionated Spotify Portal implementation that is simpler to deploy, configure and manage using templates, noted Nilsson.
As applications have become more challenging to build there is more focus on improving developer productivity. At the same time, IDPs enable platform engineering teams to streamline the number of tools being employed in a way that serves to reduce the total cost of software development.
Many developers want to add tools as they best see fit, so DevOps teams need to find a way to strike a balance between reducing the cognitive load developers encounter and what could become an overly heavy-handed approach to centralizing DevOps workflows. If talented developers start heading for the exits because they don’t like the experience being provided, IT leaders will find it difficult to replace them.
Giving DevOps Teams More time
The hope is that IDPs, when used within the context of platform engineering, will give developers more time to focus on writing business logic. However, application development is as much art as it is science. Just because developers have more time it does not necessarily follow that the insights and inspiration needed to write code automatically ensue. IDPs and platform engineering, at the very least, however, create the opportunity for developers to write more business logic faster. The degree to which that occurs naturally varies from one organization to the next.
Each organization will need to decide to what degree to embrace platform engineering, but regardless of their level of commitment IDPs can play a critical role in making it simpler to onboard developers to new projects, noted Nillson. In addition, an IDP makes it simpler for DevOps teams to track metrics such as the rate at which code is being updated, she said.
IDPs, of course, are not necessarily ideal but with the rise of Backstage there is clearly now more standardization, as DevOps teams continue to seek to eliminate as much friction as possible.
Supply-chain vulnerabilities loom large on the cybersecurity landscape, with threats and attacks such as SolarWinds, 3CX, Log4Shell and now XZ Utils underscoring the potentially devastating impact of these security breaches. The latter examples of Open Source Software (OSS) attacks are a growing attack vector. Capterra’s research reveals that software supply-chain attacks impacted almost two-thirds (61%) of all U.S. businesses from April 2022 to April 2023.
Expect attacks on the open source software supply chain to accelerate, with attackers automating attacks in common open source software projects and package managers. Many CISOs and DevSecOps teams are unprepared to implement controls in their existing build systems to mitigate these threats. In 2024, DevSecOps teams will migrate away from shift-left security models in favor of “shifting down” by using AI to automate security out of the developers’ workflows.
Here are the factors fueling the increase in software supply-chain attacks and the role of AI in helping developers work more efficiently while creating more secure code.
Attacks on the Open Source Software Supply Chain Will Accelerate
Open source libraries and languages are the foundation of over 90% of the world’s software. In a U.S. survey of nearly 300 IT and IT security professionals, 94% reported their companies use open source software, and 57% employ multiple open source platforms. Exactly half of the respondents say the threat level is “high” or “extreme,” while another 41% view it as “moderate.” At the time of writing, details of the backdoor implanted in the XZ library and several other OSS packages have just been published. The ubiquity of open source worldwide is one key factor fueling the rise of supply chain attacks.
Data Governance and the Data Supply Chain Will Become Critical Issues
Security pros must also consider how security vulnerabilities extend to their data supply chains. Although organizations typically integrate externally developed software through their software supply chains, their data supply chains often need clearer mechanisms for understanding or contextualizing data. In contrast to software’s structured systems or functions, data is unstructured or semi-structured and faces a wide array of regulatory standards.
Many companies are building AI or ML systems on top of enormous data pools with heterogeneous sources. ML models on model zoos are published with minimal understanding of the code and content used to produce the models. Software engineers need to handle these models and data just as carefully as they do the code going into the software they’re creating, with attention to its provenance.
DevSecOps teams must assess the liabilities of using data, especially when building LLMs to train AI tools. That demands careful data management within models to prevent the accidental transmission of sensitive data to third parties like OpenAI.
Organizations should adopt strict policies outlining the approved usage of AI-generated code, and when incorporating third-party platforms for AI, conduct a thorough due diligence assessment ensuring that their data will not be used for AI/ML model training and fine-tuning
AI Security Automation Will Help Organizations Transition From ‘Shift-Left’ to ‘Shift-Down’
The industry adopted the shift-left concept a decade ago to address security flaws early in the software development lifecycle and to enhance developer workflows. Defenders of systems have long been at a disadvantage. AI has the potential to level the playing field. As DevSecOps teams navigate the intricacies of data governance, they must also assess the impact of the evolving shift-left paradigm on their organizations’ security postures.
Companies will begin moving beyond shift-left to embrace AI to fully automate security processes and remove them from the developer’s workflow. This is called “shifting-down,” because it pushes security into automated and lower-level functions in the tech stack instead of burdening developers with complicated and often difficult decisions.
GitLab’s Global DevSecOps Report: The State of AI in Software Development found that developers only spend 25% of their time on code generation. AI can elevate their output by optimizing the remaining 75% of their workload. That’s one way to leverage AI’s capacity to solve specific technical issues and improve the efficiency and productivity of the entire software development life cycle.
Expect to look back on 2024 as the year that the escalating threats OSS ecosystems adversely affecting global software supply chains, catalyzed substantial changes in cybersecurity strategies, including a heightened dependence on AI to safeguard digital infrastructures. The cybersecurity landscape is already transforming, with a growing focus on mitigating supply chain vulnerabilities, enforcing data governance and incorporating AI into security measures. This transformation promises to steer DevSecOps teams toward software development processes with efficiency and security at the forefront.
The emergence of low/no-code platforms is challenging traditional notions of coding expertise. Gone are the days when coding was an exclusive skill set reserved for a knowledgeable few. Low/no-code platforms have democratized software development. They empower individuals from non-IT or technical backgrounds to translate their business ideas into applications without the need to master complex programming languages.
These low/no-code platforms, with their user-friendly drag-and-drop interfaces and pre-built components, have simplified application development. Imagine building a smart home application with just a few clicks and drags — low/no-code platforms are making it possible.
Add generative artificial intelligence (Gen AI) into the mix, and the new development practices change the field remarkably.
The Impact of Gen AI on Low-Code Platforms
Expect low-code platforms to evolve further, becoming more intelligent, adaptable and seamlessly integrated into the software development lifecycle. There will be a shift towards platforms that use Gen AI to understand user intent, automate routine tasks and generate complex code snippets.
Gartner predicts that, by 2026, over 80% of enterprises will use GenAI APIs and models or deploy GenAI-enabled applications in production environments, up from less than 5% in early 2023.
Combining low-code and Gen AI will empower developers by automating routine tasks and allowing them to focus on high-level logic and creativity. Gen AI acts as a multiplier that can accelerate and optimize developmental operations.
McKinsey estimates that generative AI could add $2.6 trillion to $4.4 trillion annually to the global economy by 2040.
Another Tool: AI Copilots and Code Generators
Another promising application for AI large language models (LLMs) is AI-powered assistants. Enter copilots and code generators — AI assistants that take the hassle out of coding.
While low/no-code platforms have transformed application development, copilots and code generators add an extra layer of efficiency.
AI copilots use a chat interface to simplify coding through a chat interface, where developers articulate their code requirements in natural language prompts, and the copilot generates the code accordingly. These assistants provide real-time assistance, offering contextual suggestions and code snippets, and they can anticipate potential bugs. Moreover, they can explain code blocks, generate unit tests and propose bug fixes, which enhances the coding process.
AI copilots are like trusted assistants, ready to aid you with specific coding tasks or generate complex code – and they do it swiftly.
The Big Question
As a result of low/no-code platforms gaining traction, the skill sets required in the tech industry will transform. Gone are the days when proficiency in multiple programming languages was a must-have.
With low/no-code tools taking center stage, does this mean coding is becoming obsolete?
A big no.
While basic programming knowledge remains valuable, it’s no longer the sole requirement for building innovative applications. With low/no-code platforms democratizing application development, individuals from diverse backgrounds can leverage technology to create impactful solutions.
Do GenAI, copilots, and code generators render low/no-code platforms obsolete?
Again, no.
While GenAI, copilots, and code generators excel at certain specific tasks, low/no-code platforms offer a holistic solution, combining visual interface design, database integration and more. They’re the Swiss army knives of development, catering to a wide range of needs beyond just code generation.
Gen AI, copilots and code generators are not here to replace low/no-code platforms or traditional coding practices. Instead, they are positioned to complement and enhance them. The fusion of these technologies holds great promise in simplifying software development, boosting efficiency, and unlocking new avenues for innovation.
The Final Word
With better code suggestions and automated testing capabilities, low/no-code platforms deliver high-quality solutions that meet the demands of today’s tech-savvy users.
With the thrilling combination of low/no-code platforms and GenAI, the future of software development is promising. While these advancements may disrupt traditional coding, they also present opportunities for innovation and creativity.
It’s time for us to embrace this change and build a brighter tomorrow, one line of code — or lack thereof — at a time.
Achieving full observability across enterprise applications is challenging. While third-party services like New Relic or Datadog make end-to-end observability easier, as your application grows in complexity, so does the telemetry data, leading to increased costs. Migrating to an open source stack gives you control over telemetry data and reduces observability costs, despite challenges with existing service provider commitments.
Migrating to open source solutions is straightforward with simple architecture and tech stacks. However, transitioning the entire observability stack to open source demands meticulous planning, tools/frameworks assessments, testing, risk analysis, and cross-team communication, especially given the number of microservices, heavy cloud reliance, and diverse languages and frameworks to take into account.
A cloud observability platform usually absorbs 20%-30% of overall infrastructure spending, but in some cases, it can reach 50%-60%. If your observability expenses exceed 50%, then transitioning to open source solutions would be a sensible option to mitigate tech infrastructure costs.
For the past two years, I have been working on migrating the observability platform from a proprietary solution provider to an open source stack. Let’s dive into the key steps necessary for such a migration.
Finalize the key telemetry data and systems
Managed observability platforms offer comprehensive insights into system health, necessitating a significant volume of high-quality telemetry data. By default, the agents tasked with capturing and transmitting telemetry data are configured to gather as much information as possible, facilitating the creation of extensive dashboards and reports. However, this extensive data collection contributes to the overall cost.
Select which systems you actually need to monitor. A typical enterprise application includes databases, caching databases, container orchestrators, and numerous cloud services. Engineers often monitor more services than necessary, which leads to unnecessary complexities and costs. It’s often sufficient to streamline the monitoring scope to essential systems.
In my experience, the quantity of telemetry data required is significantly lower than what third-party services typically retrieve. You can easily divide those data collections into two buckets: must-have and nice-to-have. The “must-have” data aligns with the well-established four golden signals:
Latency: the amount of time it takes to serve a request
Traffic: the volume of requests that a system is currently handling
The error rate: the number of requests that fail or return unexpected responses
Resource saturation: the percentage of available resources being consumed
These fundamental signals serve as the foundation for key dashboards and alerts. Observability platforms built on these signals cover 80% of typical use cases.
Understanding each signal’s metadata requirements is crucial. The extent of metadata captured directly impacts the observability platform’s complexity and cost. For example, when monitoring service-to-service latency, consider the necessity for service IP addresses, instance IDs, or header information.
Select the relevant stack
Managed observability platforms such as New Relic and Datadog provide comprehensive monitoring for various enterprise components. However, when you migrate to an open source stack, you must assess and integrate the tool stacks to meet diverse monitoring requirements.
One important aspect to consider is scaling: how to manage the huge amount of data generated every minute across all those systems. Focus on two fronts: selecting the stack for storing and processing telemetry data (logs, metrics and traces) and devising methods for capturing and pushing telemetry data from diverse systems.
Telemetry signal stack selections
Stack for logs
You require a stack that can efficiently and economically process and store the substantial volume of logs generated by the system. In the past, I used the ELK stack for storing and searching logs, but it is a generic solution and not tailored specifically for logs.
I recommend Grafana’s Loki for its effective management of large log volumes and LogQL, a language akin to PromQL. If you are familiar with PromQL, navigating through extensive log data with LogQL is easy and intuitive.
Stack for Metrics
Prometheus is a popular database for storing time-series metrics data. However, it has scaling-related limitations; it cannot scale horizontally. Alternatives like Thanos, Grafana’s Mimir, and VictoriaMetrics offer better out-of-the-box solutions for horizontal scaling.
I conducted some research and ultimately chose Grafana’s Mimir to store metrics in a similar migration project. This decision was based on Mimir’s remote storage capability, as well as its scalable and highly available architecture.
Stack of Tracing
Distributed tracing is a must in a microservices architecture, for easily identifying bottlenecks in latency and for resolving performance-related bugs. Grafana’s Tempo could be a good option for several reasons:
It can seamlessly integrate as a backend for Grafana’s dashboard. Consolidating dashboards in Grafana helps avoid navigating between multiple applications. Switching platforms just for tracing while using Grafana for other aspects of observability would not be practical.
Tempo provides a similar experience to TraceQL when interacting with the database, streamlining the learning process and significantly reducing the learning curve.
Tempo is compatible with popular open source tracing protocols such as Zipkin and Jaeger. If any team already uses these protocols, transitioning to Grafana’s Tempo would be smoother.
Capturing and pushing telemetry data
You have to install an agent on the system to capture and push the telemetry data. Those steps vary based on the system being monitored. Typical enterprise application setups involve monitoring cloud integrations, process monitoring, infrastructure hosts, Kubernetes clusters and application monitoring. Among the possibilities:
Cloud integrations: This includes monitoring for cloud services such as SQS, SNS, EMR and EC2.
Process monitoring: This involves monitoring processes running on bare-metal machines. With the advent of dockerization and Kubernetes, there is now a standardized way of starting the application. In the past, there was no fixed mechanism. For example, to run a Java application, you could use a Java application, the ‘java -jar’ command, Tomcat, or OS systemctl. In the case of node, it could be npm or PM2. Each team or service may have had its own way of starting the process.
Infrastructure host: This entails monitoring the machine itself for metrics such as CPU usage, memory, disk I/O and Network IO or checking if the machine is offline.
Kubernetes monitoring: You regularly need to monitor a Kubernetes cluster, such as instances where Kubernetes is unable to schedule a pod due to insufficient resources.
Application monitoring: This monitoring focuses on overseeing the services the team created. Each service may differ in its development approach and choice of tech stack; but from an observability perspective, they usually are treated the same.
Browser and mobile monitoring: These metrics ensure optimal performance and user experience across different platforms. Browser monitoring includes tracking page load times, rendering performance, JavaScript errors and resource usage. For mobile, it includes monitoring app crashes, latency, battery usage, network requests and device-specific metrics.
Before the advent of OpenTelemetry, there was no standardized method for monitoring applications. OpenTelemetry emerged from the merger of two earlier projects, OpenTracing and OpenCensus. It stands as a vendor- and tool-agnostic framework to instrument applications or systems irrespective of language, infrastructure or runtime environment. OpenTelemetry represents a significant community effort, and its popularity and stability are steadily growing.
Validate observability stack on application architecture
Applications vary in architecture and development stages. To address this diversity, you need multiple proofs-of-concept (POCs) across different systems, such as infrastructure, cloud service and backend and frontend applications.
Migrating an observability platform from one tech stack to another is simple when the application architecture is a monolith. However, when the underlying architecture is microservices, it becomes difficult.
While the microservices architecture offers flexibility in choosing different tech stacks for building services, devising a standard solution for capturing telemetry data becomes challenging due to the diverse range of tech stacks involved.
Conducting POCs for various tech combinations is necessary to build a self-service tool that individual teams can follow to migrate their services easily. For instance, if your service uses Java 11, Spring Boot 3.x.x and PostgreSQL, these POCs can provide standardized steps to enable application monitoring.
Migrate core components
Setting up the observability platform involves configuring the stack for metrics, logs and traces, as well as installing the necessary agents. One advantage of migrating the observability platform is that there’s no need to transfer old telemetry data. During the testing phase, you will have an overlapping period to accumulate enough useful data to create specific dashboards and alerts.
Migrating alerts and dashboards to the new system is essential, even when data migration is not necessary. This is because changes in the underlying techniques for capturing metrics may modify the names of the metrics and subsequently change all query expressions used in alerts and dashboards. Some metrics may not be available in certain scenarios, especially derived metrics that can be recreated from underlying metrics.
While migrating query expressions manually is an option, a manual process is error-prone and time-consuming for a large number of alerts. I had experienced a similar situation with alert numbers ranging from 100 to 400.
To streamline the migration, we developed a Node.js script that programmatically converted New Relic’s query expressions to Prometheus expressions. The script performed the following high-level steps:
Connected to the New Relic API server and fetched all the alerts configured for cloud integration
Converted New Relic alert expressions into PromQL alert expressions
Wrote all the Prometheus alert expressions into the YAML file
Implementing this technique allowed us to successfully migrate all alerts and dashboards in four or five days, a process that would normally demand a month’s manual labor. Additionally, scripting significantly minimized the risk of human error.
Test the migration
Verify each alert and dashboard. Run both old and new platforms during testing. Parallel operation permits thorough testing of the metrics.
However, running both systems concurrently may lead to a performance hit due to services sending telemetry data to two locations. Additionally, there may be a slight increase in costs until the complete transition.
Running both systems in parallel provides an ideal setup to ensure the migrated dashboards and alerts are functioning correctly.
Testing alerts that trigger during this process is straightforward, but evaluating those that have not yet triggered poses a challenge. In one project, I reduced the threshold value of each alert and tested its functionalities. Approximately 90% of the alerts generated through custom scripts worked seamlessly, with only 10% requiring some manual tweaking.
Migrate other related components
Other systems depend on the data generated by the observability platform. For example:
Notification systems like email, Slack channel, etc., that alert in case of an incident
Incident management tools like PagerDuty provide a streamlined way to handle incidents.
These systems rely on the alert’s payload to work properly. Their assets should be updated, as a change in the observability platform leads to a change in the alert payload. Update the templates of notification systems to ensure they seamlessly integrate with the new alert payload.
Similarly, incident management tools like PagerDuty require modifications to routing rules, escalation policies, and scheduling. Fortunately, readymade open-source migration tools are available to facilitate easy migration from PagerDuty to Grafana’s OnCall. However, in other solutions, you may not have access to out-of-the-box migration tools. In such cases, manual migration or script writing may be required.
Timeframe for the Migration
Planning a migration involves finalizing key telemetry data and systems, selecting relevant stacks. and validating the observability stack on the application architecture. It typically takes around three weeks.
The project duration depends on factors such as the number of teams involved and the quantity of active services. For instance, a migration to open source with 100 microservices and involvement from 10 different teams may take four months to complete.
Migration Complete!
Transitioning your observability platform to an open-source stack offers a promising avenue for cost reduction and increased control over telemetry data. However, accomplishing this migration demands meticulous planning and execution, encompassing essential steps like feature prioritization, stack selection, POCs, core component migration, testing, and the migration of related systems. Despite the challenges posed by diverse architectures and technologies, a systematic approach, comprehensive assessment, and collaboration among teams can facilitate a smooth migration process and guarantee its success.
