Wrangling infrastructure while innovating quickly and keeping developers happy is a constant tug-of-war for many technology leaders, including myself. To streamline some of these efforts, two prominent approaches have emerged in recent years, centers of excellence (CoE) and platform engineering. While both aim to boost productivity and collaboration, they differ significantly in their core principles and implementation methods.
Let’s delve into the key aspects of each approach, examining their strengths and weaknesses to help you determine which might be best for your organization.
Centers of Excellence
The center of excellence approach starts with the idea of establishing a centralized team of experts who possess specialized knowledge and skills in a particular domain. Usually, a CoE is created to formally address an organization’s specific knowledge deficit. This team acts as a hub of knowledge, best practices, and guidance for the entire organization.
Among the CoE benefits:
● More expertise and knowledge sharing: A CoE brings together subject matter experts who provide guidance, mentorship, and training to other teams. This ensures that knowledge is shared across the organization, leading to improved efficiency and standardization.
● Better quality control and governance: A CoE establishes standards, frameworks and governance processes to ensure consistency and quality across projects. In other words, everyone gets on the same page. This helps to reduce errors, improve code quality, and minimize technical debt.
However, a CoE also has potential drawbacks due to centralized decision-making, limited scalability and adaptability. It creates a dependency on a small core group of experts.
Several industry experts are trying to shift the conversation from a “center of excellence” to a “center of enablement” approach. In our recent API trends panel, expert James Higginbotham said, “There’s a lot of automation enablement we need to do for our developers before we even talk about a platform. Let’s shift the conversation to focus on API enablement with a look at centers for enablement.”
Platform Engineering
As software scales out and becomes more complex, DevOps helps to wrangle infrastructure into a usable state for developers to build upon. And what has evolved from those conversations is the world of platform engineering.
Platform engineering focuses on building a robust and scalable technology platform that serves as a foundation for various teams and projects within an organization, with the main goal being to improve developer experience across the board. The PE approach often combats DevOps challenges, including increasing automation and dealing with siloed teams at scale. Other benefits of a PE approach include:
● An infrastructure and tooling focus: Platform Engineering emphasizes the development and maintenance of a shared infrastructure and tooling ecosystem. This enables teams to take advantage of pre-built components, services and frameworks, which reduces duplication of effort and accelerates development cycles.
● Increased self-service and autonomy: Self-service capabilities empowers teams to independently deploy, scale and manage their applications. This reduces dependencies on specialized teams and enables faster time-to-market.
● Flexibility and adaptability: This approach allows teams to experiment, iterate and innovate without being constrained by rigid processes or technologies.
On the weaknesses side, PE practices often require a significant amount of initial investment and setup costs upfront, which smaller organizations and teams may not have the budget or resources to support. They also require continuous maintenance and updates.
The biggest hurdle, however, can often be the potential resistance to change from existing teams. At the end of the day, cultural change can be the greatest obstacle to overcome, and it can take months to fully shift a team to a new approach.
So Which do I Choose?
While a CoE can provide centralized expertise and knowledge sharing, a platform engineering approach can empower teams with self-service capabilities and flexibility. A hybrid approach may be the best route. Use a CoE to drive innovation, provide training and develop best practices. And while they’re focused on that, maximize platform engineering to ensure these practices are scalable and integrated into the development process, providing the tools and environments to keep your developers happy.
When a CoE makes sense
Let’s say you need your team to concentrate and develop expertise in a specific area or your team is focused on innovation and adopting new technologies rather than laying down foundational aspects. In these scenarios, a CoE would benefit you more. Likewise, if your projects require specialized knowledge and close guidance, a CoE would make the most sense.
When PE makes sense
Larger enterprises or teams with multiple development groups working towards similar goals would benefit most from this approach. If your biggest need is to standardize development and deployment processes above all else, Platform Engineering will get you there. Companies with the proper time and resources to invest in building and maintaining a robust platform will flourish with a PE approach.
Regardless of approach, both are focused on streamlining innovation infrastructure and ensuring the happiness of your engineering teams. Both of these things can be solved by one simple golden rule: Always listen to your developers.
Your biggest asset is always your team, and its biggest asset is always leadership that supports those people with the resources they need. So, it’s less about a tug-of-war and a choice of CoE or PE, but more about knowing which helps your company to create innovative software. I’ll leave you with the most important step you can take towards either approach, and that starts with one thing: A conversation with your teams.
In the rapidly evolving landscape of software development and IT operations, new methodologies and frameworks frequently emerge to address the challenges of efficiency, scalability and reliability. Among these, platform engineering and DevOps have garnered significant attention.
In my prior blog Measuring the Value of DevOps-as-a-Service I said “In high-level terms, DaaS is DevOps capabilities made available to the users of those capabilities through portals and APIs.”
This blog will argue that platform engineering is essentially an evolution of DevOps into a more structured and service-oriented model, providing a standardized and scalable approach to implementing DevOps practices across organizations.
Understanding DevOps
DevOps Principles: DevOps is a cultural and methodological approach emphasizing collaboration between development and operations teams. Its core principles include:
Collaboration and Communication: Breaking down silos between development and operations to foster a culture of continuous collaboration. Continuous Integration and Continuous Delivery (CI/CD): Automating the integration and deployment processes to ensure rapid and reliable software delivery. Infrastructure as Code (IaC): Managing and provisioning computing infrastructure through code rather than manual processes. Monitoring and Observability: Continuously monitoring applications and infrastructure to ensure performance, reliability and availability.
Challenges in Implementing DevOps:
While the benefits of DevOps are clear, many organizations struggle with its implementation. Common challenges include:
Tool Integration: Integrating various tools and technologies required for CI/CD, monitoring and IaC can be complex and time-consuming. Skill Gaps: Organizations often lack the necessary skills and expertise to implement and manage DevOps practices effectively. Cultural Change: Shifting the organizational culture to embrace DevOps principles can be difficult and slow.
The Rise of Platform Engineering
What is Platform Engineering?
Platform engineering focuses on building and maintaining integrated platforms that provide the necessary tools, environments, and services to support the entire software development lifecycle. These platforms are designed to streamline development, testing, deployment, and operations by offering a cohesive and standardized set of services.
Key Components of Platform Engineering:
Self-Service Interfaces: Providing developers easy access to the necessary resources, such as environments, databases and CI/CD pipelines.
Automation: Implementing automation to manage infrastructure, deployments and monitoring, reducing manual effort and error.
Standardization: Creating standardized environments and processes to ensure consistency and reliability across projects and teams.
Scalability: Designing platforms that scale with the organization’s needs, supporting multiple teams and diverse projects.
Platform Engineering as DevOps-as-a-Service
Defining DevOps-as-a-Service:
DevOps-as-a-service (DaaS) can be understood as a service-oriented approach to delivering DevOps capabilities. It involves providing a standardized and scalable set of tools and practices as a managed service, allowing organizations to leverage DevOps principles without the need to build and maintain the underlying infrastructure themselves.
Similarities Between Platform Engineering and DevOps-as-a-Service:
1. Standardization and Automation:
Both platform engineering and DaaS emphasize standardization and automation. By providing standardized tools and environments, platform engineering simplifies the integration and deployment processes, similar to how DaaS delivers a unified set of DevOps capabilities as a service.
2. Self-Service Capabilities:
Platform engineering provides self-service interfaces that allow developers to access the resources they need without waiting for operations teams. This aligns with the DaaS model, where services are readily available to users through a self-service portal.
3. Focus on Efficiency and Scalability:
Both approaches aim to improve efficiency and scalability. Platform engineering builds scalable platforms that can support multiple teams and projects, while DaaS offers scalable DevOps capabilities that can grow with the organization’s needs.
4. Reduced Complexity and Skill Gaps:
By offering a managed platform with integrated tools, platform engineering reduces the complexity of implementing DevOps practices and addresses skill gaps within organizations. Similarly, DaaS provides a turnkey solution that simplifies DevOps adoption and management.
5. Enhanced Collaboration and Integration:
Platform engineering fosters collaboration by providing a unified platform that integrates various tools and services, promoting seamless team communication and coordination. This mirrors the collaborative nature of DaaS, which integrates and orchestrates DevOps tools to support continuous delivery and operations.
Benefits of Platform Engineering as DevOps-as-a-Service 1. Accelerated Adoption: Platform Engineering accelerates the adoption of DevOps practices by providing a ready-to-use platform that incorporates best practices and tools. Organizations can quickly onboard teams and start to leverage DevOps capabilities without extensive setup and configuration.
2. Consistency and Reliability: Standardized platforms ensure consistency and reliability across projects and teams. This reduces the risk of errors and discrepancies arising from ad hoc tool integrations and manual processes.
3. Focus on Core Competencies: By offloading the responsibility of building and maintaining the DevOps infrastructure to a centralized platform team, organizations can allow development and operations teams to focus on their core competencies — developing and delivering high-quality software.
4. Improved Developer Experience: Self-service interfaces and automated processes enhance the developer experience by providing quick and easy access to necessary resources. This reduces friction and accelerates development cycles.
5. Scalability and Flexibility: Scalable platforms can accommodate the growing needs of organizations, supporting larger and more complex projects. This flexibility ensures that the platform can evolve with the organization’s requirements.
Challenges and Considerations
1. Initial Investment: Building a comprehensive platform requires an initial investment in time and resources. Organizations need to plan and allocate resources carefully to ensure successful implementation.
2. Continuous Improvement: Platforms must be continuously improved and updated to incorporate new tools, technologies and best practices. This requires ongoing effort and commitment from the platform team.
3. Balancing Standardization and Flexibility: While standardization is beneficial, it is essential to strike a balance between standardization and flexibility. Platforms should be adaptable to meet the diverse needs of different teams and projects.
Conclusion
Platform engineering represents the evolution of DevOps into a more structured and service-oriented model, effectively embodying the principles of DevOps-as-a-service. By providing standardized, scalable platforms with integrated tools and services, platform engineering simplifies the adoption and management of DevOps practices, addressing common challenges such as tool integration, skill gaps and cultural change.
As organizations seek ways to improve efficiency, scalability and reliability in their software development and IT operations, platform engineering as DevOps-as-a-service offers a compelling solution. By accelerating adoption, ensuring consistency, and enhancing the developer experience, platform engineering enables organizations to fully realize the benefits of DevOps and drive continuous innovation and improvement.
In conclusion, platform engineering is not a replacement for DevOps but an evolution that provides a structured, scalable approach to delivering DevOps capabilities as a service. This perspective highlights the complementary nature of platform engineering and DevOps, emphasizing the importance of achieving successful digital transformation.
Shreds.AI today unveiled a namesake generative artificial intelligence (AI) platform based on a large language model (LLM) it trained to specifically automate software engineering tasks.
Available in beta, the Shreds.AI platform can assign tasks to as many as eight other LLMs by invoking the application programming interfaces (APIs) they expose.
Shreds.AI CEO Soufiane Amar said rather than producing small amounts of code the platform has been trained to create the tens of thousands of lines and files of code needed to drive complex software engineering workflows. The Shreds.AI platform will also accurately orchestrate the integration of various software components to build an application because it has been trained using tools that developers regularly employ, he added.
A developer inputs a simple natural language description of the software that they want Shreds.AI to create, and the platform then generates architectural diagrams and the code for independent and isolated features called shreds. DevOps teams only need to validate the code before employing it, a process Shreds.AI makes simpler via a network of independent developers that organizations can contract to review code, said Amar.
Shreds.AI is a Meta AI, in that in addition to generating code and reasoning across processes, is able to rank third-party LLMs based on their ability to perform specific tasks, he noted.
The Shreds.AI platform is already being tested by the automotive conglomerate Stellantis and Réseau de Transport d’Électricité (RTE), the electricity transmission system operator of France. Shreds.AI estimates that an application that previously might have cost $1 million can now be created for less than $30,000. The company claims Shreds.AI slashes the time to market for software, along with team sizes and costs, by over 80% compared to traditional software development methods. By enabling automatic maintenance, it also solves the software obsolescence problem. It increases software lifespan by more than 60% by, for example, making it easier to convert the programming language used to build that application into another language that more developers know.
It’s still early days so far as the incorporation of AI into DevOps workflows is concerned and the code generated by these platforms still needs to be managed. The challenge is that thanks to the rise of AI it’s expected that the amount of software that will be built and deployed in the next two years will exceed the amount of software that has been deployed in the past two decades. The only way to keep pace with that rate of development will be to also apply AI to the management of DevOps workflows as well.
In the hope of eliminating as much existing toil as possible, DevOps teams should be identifying which manual processes they regularly perform today, with an eye towards soon applying AI to them tomorrow. After all, the whole point of embracing DevOps in the first place was to ruthlessly automate as many software engineering processes as possible, to enable more applications to be built and deployed as quickly as possible.
Catchpoint’s value proposition is simple to understand. It monitors website and application performance beyond your own infrastructure to provide a “user’s-eye view.”
I have been familiar with Catchpoint for several years, as we covered them here on devopsy.kinsta.cloud and other Techstrong sites for some time. On the second day of the #AppDev Tech Field Day in Santa Clara, Catchpoint CEO Mehdi Daoudi presented the Catchpoint story, explaining the difference between garden-variety application performance management (APM) and what Catchpoint does, which it calls Internet Performance Monitoring (IPM).
I couldn’t help but think back to the dot-com days. Several of my friends – Rajat Bhargava, Tom Higley and Neil Robertson – started a company called Service Metrics. Again, the premise was simple: understand how a website was performing from various places throughout the internet.
A funny thing happened with Service Metrics. Less than 18 months after founding the company – even before the product was fully baked – it was bought by Exodus Communications, one of that era’s largest data center providers. In typical dot-com craziness, Exodus stock split several times and went through the roof from when the deal was announced to the time it closed. What was a $280 million dollar acquisition became over a billion dollar deal by the time of closing. Such was life in the dotcom days.
That was over 25 years ago, but the basic premise—monitoring Internet performance—is similar to Catchpoint’s goals now. Of course, today’s world and network are much more complicated. But ultimately, we all want to know how our websites and applications perform for end users.
The interesting thing about Catchpoint is that it has matured to the point where the monitoring services can flow directly to the CI/CD pipeline via feedback loops. Using industry-standard tools such as Open Telemetry, Catchpoint works with tools, too. These integrations and standards-based engineering allow Catchpoint to play nicely with the rest of an organization’s tools, help with service-level objectives (SLOs) and take advantage of existing IT investments.
What really excited me, though, was Mehdi’s response to my question about adding automated responses to what Catchpoint monitors. AI can really be a game changer. The company has something like 15 years’ worth of monitoring data, which would be a mother lode for training an AI so it could add actionable, automated responses to what Catchpoint sees as issues.
Daoudi foresees a future where an SRE can wake up, check a Catchpoint “sonar” dashboard, and receive a full report of what Catchpoint saw as sub-optimal performance and what steps were taken to remediate them. And of course, in this scenario, the fixes and findings were incorporated into the next version of the code and tests to check it prior to deployment were already prepared.
Utopian? Maybe. But that is the kind of stuff that every network person I know could only dream of back in the Service Metrics day.
By the way, I loved my first experience as a Tech Field Day delegate, and I’m looking forward to the next one. Perhaps it’ll be in November, in Salt Lake City, around the Kubecon North America conference.
The emergence of low/no-code platforms is challenging traditional notions of coding expertise. Gone are the days when coding was an exclusive skill set reserved for a knowledgeable few. Low/no-code platforms have democratized software development. They empower individuals from non-IT or technical backgrounds to translate their business ideas into applications without the need to master complex programming languages.
These low/no-code platforms, with their user-friendly drag-and-drop interfaces and pre-built components, have simplified application development. Imagine building a smart home application with just a few clicks and drags — low/no-code platforms are making it possible.
Add generative artificial intelligence (Gen AI) into the mix, and the new development practices change the field remarkably.
The Impact of Gen AI on Low-Code Platforms
Expect low-code platforms to evolve further, becoming more intelligent, adaptable and seamlessly integrated into the software development lifecycle. There will be a shift towards platforms that use Gen AI to understand user intent, automate routine tasks and generate complex code snippets.
Gartner predicts that, by 2026, over 80% of enterprises will use GenAI APIs and models or deploy GenAI-enabled applications in production environments, up from less than 5% in early 2023.
Combining low-code and Gen AI will empower developers by automating routine tasks and allowing them to focus on high-level logic and creativity. Gen AI acts as a multiplier that can accelerate and optimize developmental operations.
McKinsey estimates that generative AI could add $2.6 trillion to $4.4 trillion annually to the global economy by 2040.
Another Tool: AI Copilots and Code Generators
Another promising application for AI large language models (LLMs) is AI-powered assistants. Enter copilots and code generators — AI assistants that take the hassle out of coding.
While low/no-code platforms have transformed application development, copilots and code generators add an extra layer of efficiency.
AI copilots use a chat interface to simplify coding through a chat interface, where developers articulate their code requirements in natural language prompts, and the copilot generates the code accordingly. These assistants provide real-time assistance, offering contextual suggestions and code snippets, and they can anticipate potential bugs. Moreover, they can explain code blocks, generate unit tests and propose bug fixes, which enhances the coding process.
AI copilots are like trusted assistants, ready to aid you with specific coding tasks or generate complex code – and they do it swiftly.
The Big Question
As a result of low/no-code platforms gaining traction, the skill sets required in the tech industry will transform. Gone are the days when proficiency in multiple programming languages was a must-have.
With low/no-code tools taking center stage, does this mean coding is becoming obsolete?
A big no.
While basic programming knowledge remains valuable, it’s no longer the sole requirement for building innovative applications. With low/no-code platforms democratizing application development, individuals from diverse backgrounds can leverage technology to create impactful solutions.
Do GenAI, copilots, and code generators render low/no-code platforms obsolete?
Again, no.
While GenAI, copilots, and code generators excel at certain specific tasks, low/no-code platforms offer a holistic solution, combining visual interface design, database integration and more. They’re the Swiss army knives of development, catering to a wide range of needs beyond just code generation.
Gen AI, copilots and code generators are not here to replace low/no-code platforms or traditional coding practices. Instead, they are positioned to complement and enhance them. The fusion of these technologies holds great promise in simplifying software development, boosting efficiency, and unlocking new avenues for innovation.
The Final Word
With better code suggestions and automated testing capabilities, low/no-code platforms deliver high-quality solutions that meet the demands of today’s tech-savvy users.
With the thrilling combination of low/no-code platforms and GenAI, the future of software development is promising. While these advancements may disrupt traditional coding, they also present opportunities for innovation and creativity.
It’s time for us to embrace this change and build a brighter tomorrow, one line of code — or lack thereof — at a time.
Achieving full observability across enterprise applications is challenging. While third-party services like New Relic or Datadog make end-to-end observability easier, as your application grows in complexity, so does the telemetry data, leading to increased costs. Migrating to an open source stack gives you control over telemetry data and reduces observability costs, despite challenges with existing service provider commitments.
Migrating to open source solutions is straightforward with simple architecture and tech stacks. However, transitioning the entire observability stack to open source demands meticulous planning, tools/frameworks assessments, testing, risk analysis, and cross-team communication, especially given the number of microservices, heavy cloud reliance, and diverse languages and frameworks to take into account.
A cloud observability platform usually absorbs 20%-30% of overall infrastructure spending, but in some cases, it can reach 50%-60%. If your observability expenses exceed 50%, then transitioning to open source solutions would be a sensible option to mitigate tech infrastructure costs.
For the past two years, I have been working on migrating the observability platform from a proprietary solution provider to an open source stack. Let’s dive into the key steps necessary for such a migration.
Finalize the key telemetry data and systems
Managed observability platforms offer comprehensive insights into system health, necessitating a significant volume of high-quality telemetry data. By default, the agents tasked with capturing and transmitting telemetry data are configured to gather as much information as possible, facilitating the creation of extensive dashboards and reports. However, this extensive data collection contributes to the overall cost.
Select which systems you actually need to monitor. A typical enterprise application includes databases, caching databases, container orchestrators, and numerous cloud services. Engineers often monitor more services than necessary, which leads to unnecessary complexities and costs. It’s often sufficient to streamline the monitoring scope to essential systems.
In my experience, the quantity of telemetry data required is significantly lower than what third-party services typically retrieve. You can easily divide those data collections into two buckets: must-have and nice-to-have. The “must-have” data aligns with the well-established four golden signals:
Latency: the amount of time it takes to serve a request
Traffic: the volume of requests that a system is currently handling
The error rate: the number of requests that fail or return unexpected responses
Resource saturation: the percentage of available resources being consumed
These fundamental signals serve as the foundation for key dashboards and alerts. Observability platforms built on these signals cover 80% of typical use cases.
Understanding each signal’s metadata requirements is crucial. The extent of metadata captured directly impacts the observability platform’s complexity and cost. For example, when monitoring service-to-service latency, consider the necessity for service IP addresses, instance IDs, or header information.
Select the relevant stack
Managed observability platforms such as New Relic and Datadog provide comprehensive monitoring for various enterprise components. However, when you migrate to an open source stack, you must assess and integrate the tool stacks to meet diverse monitoring requirements.
One important aspect to consider is scaling: how to manage the huge amount of data generated every minute across all those systems. Focus on two fronts: selecting the stack for storing and processing telemetry data (logs, metrics and traces) and devising methods for capturing and pushing telemetry data from diverse systems.
Telemetry signal stack selections
Stack for logs
You require a stack that can efficiently and economically process and store the substantial volume of logs generated by the system. In the past, I used the ELK stack for storing and searching logs, but it is a generic solution and not tailored specifically for logs.
I recommend Grafana’s Loki for its effective management of large log volumes and LogQL, a language akin to PromQL. If you are familiar with PromQL, navigating through extensive log data with LogQL is easy and intuitive.
Stack for Metrics
Prometheus is a popular database for storing time-series metrics data. However, it has scaling-related limitations; it cannot scale horizontally. Alternatives like Thanos, Grafana’s Mimir, and VictoriaMetrics offer better out-of-the-box solutions for horizontal scaling.
I conducted some research and ultimately chose Grafana’s Mimir to store metrics in a similar migration project. This decision was based on Mimir’s remote storage capability, as well as its scalable and highly available architecture.
Stack of Tracing
Distributed tracing is a must in a microservices architecture, for easily identifying bottlenecks in latency and for resolving performance-related bugs. Grafana’s Tempo could be a good option for several reasons:
It can seamlessly integrate as a backend for Grafana’s dashboard. Consolidating dashboards in Grafana helps avoid navigating between multiple applications. Switching platforms just for tracing while using Grafana for other aspects of observability would not be practical.
Tempo provides a similar experience to TraceQL when interacting with the database, streamlining the learning process and significantly reducing the learning curve.
Tempo is compatible with popular open source tracing protocols such as Zipkin and Jaeger. If any team already uses these protocols, transitioning to Grafana’s Tempo would be smoother.
Capturing and pushing telemetry data
You have to install an agent on the system to capture and push the telemetry data. Those steps vary based on the system being monitored. Typical enterprise application setups involve monitoring cloud integrations, process monitoring, infrastructure hosts, Kubernetes clusters and application monitoring. Among the possibilities:
Cloud integrations: This includes monitoring for cloud services such as SQS, SNS, EMR and EC2.
Process monitoring: This involves monitoring processes running on bare-metal machines. With the advent of dockerization and Kubernetes, there is now a standardized way of starting the application. In the past, there was no fixed mechanism. For example, to run a Java application, you could use a Java application, the ‘java -jar’ command, Tomcat, or OS systemctl. In the case of node, it could be npm or PM2. Each team or service may have had its own way of starting the process.
Infrastructure host: This entails monitoring the machine itself for metrics such as CPU usage, memory, disk I/O and Network IO or checking if the machine is offline.
Kubernetes monitoring: You regularly need to monitor a Kubernetes cluster, such as instances where Kubernetes is unable to schedule a pod due to insufficient resources.
Application monitoring: This monitoring focuses on overseeing the services the team created. Each service may differ in its development approach and choice of tech stack; but from an observability perspective, they usually are treated the same.
Browser and mobile monitoring: These metrics ensure optimal performance and user experience across different platforms. Browser monitoring includes tracking page load times, rendering performance, JavaScript errors and resource usage. For mobile, it includes monitoring app crashes, latency, battery usage, network requests and device-specific metrics.
Before the advent of OpenTelemetry, there was no standardized method for monitoring applications. OpenTelemetry emerged from the merger of two earlier projects, OpenTracing and OpenCensus. It stands as a vendor- and tool-agnostic framework to instrument applications or systems irrespective of language, infrastructure or runtime environment. OpenTelemetry represents a significant community effort, and its popularity and stability are steadily growing.
Validate observability stack on application architecture
Applications vary in architecture and development stages. To address this diversity, you need multiple proofs-of-concept (POCs) across different systems, such as infrastructure, cloud service and backend and frontend applications.
Migrating an observability platform from one tech stack to another is simple when the application architecture is a monolith. However, when the underlying architecture is microservices, it becomes difficult.
While the microservices architecture offers flexibility in choosing different tech stacks for building services, devising a standard solution for capturing telemetry data becomes challenging due to the diverse range of tech stacks involved.
Conducting POCs for various tech combinations is necessary to build a self-service tool that individual teams can follow to migrate their services easily. For instance, if your service uses Java 11, Spring Boot 3.x.x and PostgreSQL, these POCs can provide standardized steps to enable application monitoring.
Migrate core components
Setting up the observability platform involves configuring the stack for metrics, logs and traces, as well as installing the necessary agents. One advantage of migrating the observability platform is that there’s no need to transfer old telemetry data. During the testing phase, you will have an overlapping period to accumulate enough useful data to create specific dashboards and alerts.
Migrating alerts and dashboards to the new system is essential, even when data migration is not necessary. This is because changes in the underlying techniques for capturing metrics may modify the names of the metrics and subsequently change all query expressions used in alerts and dashboards. Some metrics may not be available in certain scenarios, especially derived metrics that can be recreated from underlying metrics.
While migrating query expressions manually is an option, a manual process is error-prone and time-consuming for a large number of alerts. I had experienced a similar situation with alert numbers ranging from 100 to 400.
To streamline the migration, we developed a Node.js script that programmatically converted New Relic’s query expressions to Prometheus expressions. The script performed the following high-level steps:
Connected to the New Relic API server and fetched all the alerts configured for cloud integration
Converted New Relic alert expressions into PromQL alert expressions
Wrote all the Prometheus alert expressions into the YAML file
Implementing this technique allowed us to successfully migrate all alerts and dashboards in four or five days, a process that would normally demand a month’s manual labor. Additionally, scripting significantly minimized the risk of human error.
Test the migration
Verify each alert and dashboard. Run both old and new platforms during testing. Parallel operation permits thorough testing of the metrics.
However, running both systems concurrently may lead to a performance hit due to services sending telemetry data to two locations. Additionally, there may be a slight increase in costs until the complete transition.
Running both systems in parallel provides an ideal setup to ensure the migrated dashboards and alerts are functioning correctly.
Testing alerts that trigger during this process is straightforward, but evaluating those that have not yet triggered poses a challenge. In one project, I reduced the threshold value of each alert and tested its functionalities. Approximately 90% of the alerts generated through custom scripts worked seamlessly, with only 10% requiring some manual tweaking.
Migrate other related components
Other systems depend on the data generated by the observability platform. For example:
Notification systems like email, Slack channel, etc., that alert in case of an incident
Incident management tools like PagerDuty provide a streamlined way to handle incidents.
These systems rely on the alert’s payload to work properly. Their assets should be updated, as a change in the observability platform leads to a change in the alert payload. Update the templates of notification systems to ensure they seamlessly integrate with the new alert payload.
Similarly, incident management tools like PagerDuty require modifications to routing rules, escalation policies, and scheduling. Fortunately, readymade open-source migration tools are available to facilitate easy migration from PagerDuty to Grafana’s OnCall. However, in other solutions, you may not have access to out-of-the-box migration tools. In such cases, manual migration or script writing may be required.
Timeframe for the Migration
Planning a migration involves finalizing key telemetry data and systems, selecting relevant stacks. and validating the observability stack on the application architecture. It typically takes around three weeks.
The project duration depends on factors such as the number of teams involved and the quantity of active services. For instance, a migration to open source with 100 microservices and involvement from 10 different teams may take four months to complete.
Migration Complete!
Transitioning your observability platform to an open-source stack offers a promising avenue for cost reduction and increased control over telemetry data. However, accomplishing this migration demands meticulous planning and execution, encompassing essential steps like feature prioritization, stack selection, POCs, core component migration, testing, and the migration of related systems. Despite the challenges posed by diverse architectures and technologies, a systematic approach, comprehensive assessment, and collaboration among teams can facilitate a smooth migration process and guarantee its success.
The history of DevOps is worth reading about, and “The Phoenix Project,” self-characterized as “a novel of IT and DevOps,” is often mentioned as a must-read. Yet for practitioners like myself, a more hands-on account is “The DevOps Handbook” (by the same author, Gene Kim, and others), which recounts some of the watershed moments around the evolution of software engineering and illustrates it with technology that provides good references around implementation. This book describes how to replicate the transformation explained in the Phoenix Project and provides case studies.
In this brief article, I will use my notes on that book to outline a concise history of DevOps, add my personal experience and opinion and establish a link to cloud development environments (CDEs), i.e., the technology category at the center of the platform that we have been developing at my company, Strong Network.
In particular, I explain how a cloud development environment finalizes the effort of bringing DevOps “fully online.” This notion of completeness, i.e., how CDEs mark the end of moving software development online, is my main contribution to this brief article. To clarify the link between DevOps and CDEs, let’s first dig into the chain of events and technical contributions that led to today’s main methodology for delivering software.
The Agile Manifesto
The creation of the Agile Manifesto in 2001 sets forth values and principles as a response to more cumbersome software development methodologies like Waterfall and the Rational Unified Process (RUP).
One of the manifesto’s core principles emphasizes the importance of delivering working software frequently, ranging from a few weeks to a couple of months, with a preference for shorter timescales. The Agile movement’s influence expanded in 2008 during the Agile Conference in Toronto, where Andrew Shafer suggested applying Agile principles to IT infrastructure rather than just application code.
This idea was further propelled by a 2009 presentation at the Velocity Conference, where a paper from Flickr demonstrated the impressive feat of “10 deployments a day” using Dev and Ops collaboration. Inspired by these developments, Patrick Debois organized the first DevOps Days in Belgium, effectively coining “DevOps.” This marked a significant milestone in the evolution of software development and operational practices, blending Agile’s swift adaptability with a more inclusive approach to the entire IT infrastructure.
The Three Ways of DevOps and the Principles of Flow
All the concepts discussed so far are incarnated into the “Three Ways of DevOps,” which are the foundational principles that guide the practices and processes in DevOps. In brief, these principles focus on:
1. Improving the flow of work (first way) by eliminating bottlenecks, reducing batch sizes, and accelerating workflow from development to production.
2. Amplifying feedback loops (second way) by quickly and accurately receiving information about any issues or inefficiencies in the system.
3. Fostering a culture of continuous learning and experimentation (third way) by encouraging a culture of continuous learning, experimentation and taking risks.
Following the lead of lean manufacturing and Agile, it is easy to understand what led to the definition of the above three principles. I delve more deeply into each of these principles in other articles. For the current discussion of how DevOps history connects to cloud development environments, we must look at the first way, the principle of flow, to understand the causative link.
Chapter nine of the DevOps Handbook explains that version control and containerization are central to implementing DevOps flows and establishing a reliable and consistent development process.
First, incorporating all production artifacts into version control serves as a single source of truth, enabling the recreation of the entire production environment in a repeatable and documented fashion. This ensures that production-like code development environments can be automatically generated and entirely self-serviced without requiring manual intervention from operations.
The significance of this approach becomes evident at release time, which is often the first instance where an application’s behavior is observed in a production-like setting, complete with realistic load and production data sets. To reduce the likelihood of issues, developers are encouraged to operate production-like environments on their workstations, created on-demand and self-serviced through mechanisms such as virtual images or containers, utilizing tools like Vagrant or Docker. Putting these environments under version control allows the entire pre-production and build processes to be recreated.
From Developer Workstations to a CDE Platform
The notion of self-service is already emphasized in this 2016 book as a key enabler of the principle of flow. Looking back at the TPS, the notion of self-service aims to empower Toyota’s workers to be more effective in controlling the development process. Using 2016 technology, this notion is realized by downloading environments to the developers’ workstations from a registry (an online portal) that provides pre-configured, production-like environments.
Through this operation, developers, in effect, (1) Copy files with infrastructure information to their machines, (2) add source code to it, and (3) build the application using their workstation computing resources.
Once the application works correctly, the source code is sent (pushed) to a central code repository, and the application is built and deployed using online or cloud-based resources.
The three steps listed above are, in effect, the only operations in addition to authoring source code using a “local” IDE. In other words, they use workstations’ physical storage and computing resources. All the rest of the DevOps operations are performed using web-based applications used as-a-service by development (even when these applications are self-hosted by the organization). The goal of cloud development environments is to move these operations online as well.
To do that, the CDE platform, in essence, provides the following basic services:
● Manage development environments online, as containers or virtual machines such that developers can access them fully built and configured, replacing step (1) above.
● Provide a mechanism for authoring source code directly online, such as inside the development environment using an IDE or a terminal, replacing step (2).
● Provide a way to execute build commands inside the development environment (via the IDE or terminal), replacing step (3).
Note that the replacement of step (2) can be done in several ways: For example, the IDE can be browser-based (in a cloud IDE), or a locally installed IDE can implement a way to remotely author the code in the remote environment. It is possible to use a console text editor via a terminal such as vim.
I cannot conclude this discussion without taking a recursive step to rethink the developer’s workflow. Often, multiple containerized environments are used for local testing, particularly in combination with the main containerized development environment. Hence, cloud IDE platforms need to provide the ability to run containerized environments inside the cloud development environment (itself a containerized environment). If this becomes a bit complicated, no worries. We have reached the end of the discussion, and we can move to the conclusion.
What Comes Out of Using Cloud Development Environments in DevOps
A good way to conclude this discussion is to summarize the benefits of moving development environments from the developers’ workstations online. For example, the use of CDEs for DevOps leads to the following advantages:
● Streamlined Workflow: CDEs enhance the workflow by removing data from the developer’s workstation and decoupling the hardware from the development process. This ensures the development environment is consistent and not limited by local hardware constraints.
● Version Control Integration: With CDEs, version control becomes more robust as it can uniformize the environment definition and all the tools attached to the workflow, leading to a standardized development process across the organization.
● Self-Service Environments: The self-service aspect is improved by centralizing production, maintenance and evolution of environments based on distributed development activities. This allows developers to quickly access and manage their environments without the need for operations manual work.
● Consistency Across Teams: The use of CDEs leads to uniform environments and tooling, which opens the possibility of harmonizing the entire development workflow, making it easier to onboard new team members and ensure that all developers are using the same configurations and tools.
● Reduced Local Resource Dependence: Migrating the consumption of computing resources from local hardware to centralized and shared cloud resources lightens the load on local machines and leads to more efficient use of organizational resources and potential cost savings.
● Improved Collaboration: Ubiquitous access to development environments, secured by embedded security measures in the access mechanisms, allows organizations to cater to a diverse group of developers, including internal, external and temporary workers, fostering collaboration across various teams and geographies.
● Scalability and Flexibility: CDEs offer scalable cloud resources that can be adjusted to project demands, facilitating the management of multiple containerized environments for testing and development, thus supporting the distributed nature of modern software development teams.
● Enhanced Security and Observability: Centralizing development environments in the cloud bolsters security and provides immediate observability due to their online nature, allowing for real-time monitoring and management of development activities.
By integrating these aspects, CDEs become a comprehensive solution for modern software development. They align with the principles of DevOps to improve flow, feedback and continuous learning, while also addressing the practical needs of development teams for security, accessibility and resource efficiency.
Dispelling a Few Myths About DevOps
For starters, “The DevOps Handbook” aims to dispel common misconceptions about DevOps and how it is applicable across diverse business environments, including non-software product delivery. That last point was one of my first moments of personal enlightenment when reading the book about the general applicability of the method. That led me to suggest to a friend creating a business of globally designed yet locally manufactured home furniture that designers post their models via a platform for furniture pieces to be built by local (relative to the delivery address) woodshops and that his manufacturing process perfectly fits a DevOps canvas.
Furthermore, the book refutes the notion that DevOps means eliminating IT operations. Instead, it highlights the importance of collaboration and integration between development and operations teams. On that point, it also strikes me that the mention of DevOps still has a strong attachment to the IT operations of building and deploying software rather than to the design and coding phases of that process. In other words, some managers fail to recognize that developers are indeed “doing DevOps” in the same way that IT operators are. Platform engineering aims to further reduce the gap between developers and IT operations by empowering developers (and other roles) to perform complex IT operations in a self-service environment.
Lastly, the idea that DevOps is solely about “infrastructure as code” and automation is also challenged in the book. While automation is a crucial component of many DevOps patterns, the book stresses that DevOps also requires a cultural shift and an architecture that supports shared goals across the IT value stream.
The RUP was the main software engineering methodology that IBM used to structure its development process and the ones of its customers, at the time I was part of the IBM Research division at T.J. Watson, New York in the 2000s. We sold RUP across every industry, including automotive manufacturers (a nice feedback loop), when they started to embed gigabytes of software in cars.
At IBM Research, we even evolved the RUP into a methodology to fit the needs of software-laden product development from a system perspective, dubbed model-driven systems development. This is important because software in manufacturing needs to communicate and actuate with independently-built hardware. I have taught that concept in Japan at Keio University (along with Prof. Nishimura) from 2006 to the mid-2010s.
Enterprises are under constant pressure to ensure their IT infrastructure and applications are available throughout the year. The complexity of modern architectures (containers, hybrid cloud, SOA, microservices, etc.) is constantly growing, generating vast volumes of unmanageable logs. We need intelligent application performance management (APM) and observability tools to achieve production excellence and meet the availability and uptime goals. These include analyzing application health, performance and user experience. Adopting machine learning techniques to identify anomalies and behavior patterns will help detect the root cause early and meet customer service level agreements (SLA).
The APM and observability tools market is undoubtedly hot. These tools ingest multiple telemetry data feeds and are powerful analytics platforms providing critical insights into application and infrastructure health, including system performance. The software development teams that adopt observability are much better equipped to release their application code iteratively. As per “MarketsandMarkets” research, the market size of observability tools and platforms is expected to grow from $2.4 billion in 2023 to $4+ billion by 2028 at a compound annual growth rate (CAGR) of 11.7%.
What is Observability?
Observability is the ability to collect data about the distributed applications, infrastructure and communication among its internal and external components and services, allowing teams to debug their systems diligently. It enables the site reliability engineering (SRE), software engineering and operations teams to analyze the customer impact and triage a service outage. Observability and monitoring are sometimes used interchangeably. Observability (proactive) makes the data accessible and allows you to ask any system question to know how the code behaves more deeply. Monitoring (reactive) is the task of collecting and displaying that data and the ability to determine the system’s overall state.
Observability can be further broken down into three key pillars: Logs, traces and metrics, which are essential for SRE observability.
• Logs help us diagnose the issues and tell us why they happened.
• Traces help us isolate issues and tell us where they happened.
• Metrics help us detect the issues and tell us what happened.
Market Tools, Capabilities and Challenges
Gartner’s magic quadrant for APM and observability has identified 20+ vendor products offering APM and observability capabilities, including self-hosted, vendor-managed or SaaS deployments. These products provide multiple features, including application performance metrics, monitoring and alerting on events, traceability, anomaly detection and vulnerabilities, etc.
An enterprise business application comprises homegrown applications (such as .NET, Java, Python, SQL, NoSQL DB, etc.), third-party/off-the-shelf products (such as Salesforce, HubSpot, etc.), and integrations (such as Stripe, PayPal, etc.). Homegrown applications are hosted in an on-premises data center or by cloud vendors like AWS, GCP, or Azure. The off-the-shelf products are SaaS-based or integrated via APIs. There are highly distributed applications spanning tens and hundreds of nodes, services and instances.
• Too Many Tools: Enterprise applications use various tools for monitoring application health and performance (such as New Relic, Data Dog, etc.), Error logging (such as Splunk), and cloud vendor-provided tools (such as CloudWatch). These products overlap in functionality and maintaining and managing these tools (procurement, learning curve, etc.) can be cumbersome.
• Unpredictable Data Volume: Imagine the volume of observability data (logs, traces, metrics) collected based on the application traffic, usage, dependency on external products, etc. The amount of data storage required to consolidate these data feeds can quickly grow out of control.
• Pricing is Complex: These vendor products also offer different pricing models such as charge per host (such as Splunk, Data Dog, Dynatrace), per-user (such as New Relic), per-ingestion (such as SumoLogic, AppDynamics). The complexity of pricing models makes it challenging to compare total cost of ownership (TCO) among vendors and determine the right tool that fits your requirements and budget.
Why Choose Open Source Observability Platform?
Open source-based observability tools aim to provide a standard, vendor-agnostic approach for ingesting, transforming and sending data to an observability backend. The open source observability tools can be an alternative to save on licensing costs and consolidate multiple APM tools with the tool that fits your requirements and budget.
However, maintaining open-source systems may require efforts to set up and maintain and will add to your initial operational cost. But in the long term, you will save on licensing fees and avoid vendor lock-in and contractual agreements.
Gartner predicts that, by 2025, 70% of new cloud-native application monitoring will use open-source instrumentation rather than vendor-specific agents for improved interoperability, and 70% of new cloud-native applications will adopt OpenTelemetry for observability rather than vendor-specific agents and software development kits (SDKs).
Scale Observability Using the Open Source Ecosystem
The open source landscape for observability is quite dynamic. There are multiple Cloud Native Computing Foundation (CNCF) open source tools for observability and monitoring. This post will primarily focus on the OpenTelemetry framework and LGTM technology stack.
OpenTelemetry:
The “too many tools” challenge described above brings up a new struggle in telemetry data collection. Each tool vendor has its own APIs, SDKs, agents and collectors for logs, metrics and traces. We need a unified telemetry collection using the OpenTelemetry framework to create and manage telemetry data such as logs, traces and metrics.
The OTEL project under the auspices of the CNCF offers a unified set of vendor-agnostic APIs, SDKs, and tools for generating and collecting telemetry data and exporting it to various analysis tools. You get one API and SDK per programming language to extract your application’s observability data, a standard collector, a transmission protocol (OTLP) and more.
LGTM:
The most popular open source-based observability and monitoring is implemented using the LGTM technology stack.
In the LGTM stack, we leverage:
• Loki for log aggregation
• Grafana dashboards for telemetry visualizations
• Tempo (or Jaeger) for trace aggregation
• Managed Prometheus for metrics aggregation
Conclusion
Observability is about complete visibility across your systems and tying business metrics with technical data. Monitoring is about understanding if things are working correctly, and AIOps is about getting meaning from that visibility. Observability and monitoring are critical to ensure the smooth functioning of applications and meet customer SLAs. In conclusion, by investing in open source OTel frameworks and LGTM tools, SRE teams can effectively monitor their applications and gain insights into system behavior and potential issues. These tools offer cost-effectiveness and customization to suit specific requirements. It promotes vendor neutrality, which can be critical for avoiding vendor lock-in.
A global survey of 500 application development and deployment professionals finds that while 43% work for organizations that have had a platform team in place for three to five years, there doesn’t appear to be much consistency regarding where that team reports within an organization.
The survey, conducted by Puppet by Perforce, a provider of an automation framework, found that among organizations that have platform teams, well over half (58%) operate within the context of a larger DevOps or infrastructure management organization, compared to 40% that have their own dedicated leadership.
David Sandilands, a principal solutions architect for Puppet by Perforce, said the survey makes it clear that platform engineering, in general, is not a new concept, but it’s clear more organizations are trying to reduce the level of toil currently associated with building and deploying applications.
Regardless of where platform teams reside in an organization, nearly two-thirds of respondents (65%) said they are important to their organizations and would receive continued funding. The primary purpose for forming these teams is to increase productivity (58%) and automate standardized processes (51%), the survey finds. Top goals for platform engineering teams are to solve problems (30%), enforce security processes (27%) and accelerate transitions to cloud-native computing environments (26%). A full 70% said security was built into their platforms from the start.
That suggests that, in addition to assuming responsibility for traditional DevOps workflows, many platform teams are now also being asked to ensure DevSecOps best practices are followed at a time when application security requirements are becoming more stringent, noted Sandilands.
More than three quarters (76%) also reported that they had deployed two or more self-service portals, with 27% having deployed five or more.
Only 22%, however, said they have deployed applications on Kubernetes in a production environment, with just under half (46%) reporting they have no current plans to deploy a cloud-native computing platform, which are widely seen to be challenging to deploy and maintain.
Overall, the survey makes it clear the primary focus is to ensure platforms continue to evolve as developer requirements continue to evolve, said Sandilands. Platform teams are not necessarily trying to dictate what tools should be used; instead, they are finding ways to automate processes that are increasing the level of friction developers encounter, he added.
In the longer term, advances in artificial intelligence (AI) will make it simpler to achieve that goal using natural language interfaces that DevOps teams can more easily invoke, noted Sandilands.
Each organization will need to decide for itself whether platform engineering makes sense as a methodology for managing DevOps at scale, but as application development and deployment continues to evolve, more nuanced approaches will be required. In general, organizations are trying to strike a balance between empowering developers and the need to streamline backend processes to increase productivity while also simplifying compliance and improving security.
The challenge, of course, is enticing developers to buy into that concept rather than using their expertise to resist platform teams that might, from their perspective, limit their prerogatives in ways that ultimately stifle innovation.
A global survey of 500 IT professionals suggests organizations are not making a lot of progress in their ability to truly observe application environments, especially as they become more complex with each passing day.
The survey, conducted by Logz.io, a provider of an observability platform, found only one in 10 respondents reported they have full observability into their application environments.
Logz.io CTO Asaf Yigal said while more DevOps teams than ever are collecting logs, metrics and traces, most of them have yet to determine how to operationalize all the data being collected. As data is ingested, it doesn’t just need to be stored; it also needs to be correlated to the various services that make up an application, he noted.
The more organizations deploy cloud-native applications in production environments the more pressing this issue becomes. All the microservices that make up those applications are now generating a massive amount of telemetry data that results in more alerts than ever being generated.
Not surprisingly, the biggest challenges organizations encounter when managing Kubernetes clusters in production environments are monitoring/troubleshooting (40%), followed closely by security (37%) and networking (33%).
Many organizations simply don’t have the skills required to manage cloud-native applications. Nearly half of respondents (48%) specifically cited a lack of knowledge as the biggest challenge they encountered when trying to observe these types of applications. From an IT management perspective, most microservices all use the same basic template, so it’s difficult for DevOps teams to identify which microservices are likely to have the biggest impact on service level objectives (SLOs) and service level agreements (SLAs) in the event of a disruption, noted Yigal.
In the absence of the ability to determine the actual root cause of an issue, alert storms conspire to increase fatigue—and that eventually results in higher levels of burnout across the DevOps team, noted Yigal. In fact, 82% of respondents said their mean-time-to-resolution (MTTR) during production incidents was more than an hour.
The survey also noted that more than half of respondents (52%) worked for organizations that are simultaneously trying to rein in monitoring costs. More than three-quarters (76%) of respondents also noted that open source OpenTelemetry (OTEL) or OTEL-centric tooling was at least somewhat important to their overall observability strategy.
In addition, 87% of respondents said their organization is already using some form of platform engineering to manage DevOps workflows at scale.
It’s not clear how quickly organizations are adopting observability tools and platforms, but too many DevOps teams lack the visibility needed to identify the root cause of an issue. As a result, variations of the same problems often keep manifesting themselves because previous remediation efforts simply didn’t go deep enough to resolve the core issue.
Of course, there may come a day when machine learning algorithms, along with other forms of artificial intelligence (AI), will make it simpler to surface these issues. The challenge, in the meantime, is laying down the observability foundation today to provide access to the data that will be required to train those AI models.
