devopsy.kinsta.cloud

Tag: devops

  • AI, the New Hero of Software Development … or Anti-Hero?

    AI, the New Hero of Software Development … or Anti-Hero?

    Artificial intelligence (AI) can be a hero or an anti-hero in software development, depending on your implementation approach. According to the 2024 DORA Accelerate State of DevOps Report, AI adoption shows promise with 75% of respondents reporting positive productivity gains, but also reveals unexpected challenges, including a 7.2% reduction in software delivery stability.

    The key is to position AI as a supportive partner. AI should be utilized to enhance human expertise rather than replace it.

    AI Is Promising, but Let’s Stay Suspicious

    AI has been rapidly infiltrating software development for almost two years now. Coding assistants, automated testing tools, and AI-powered analytics promise to transform how we build applications, and some early adopters have shown impressive results. When AI first entered development workflows, it tackled well-defined problems such as code completion, simple test generation, commenting, and draft documentation with reasonable success.

    But that initial enthusiasm has given way to more complex realities. Modern software development involves intricate systems, multi-cloud deployments, complex security requirements, and accelerating delivery expectations. AI has shown both remarkable capabilities and concerning limitations in this environment.

    After hearing customer stories about dozens of AI implementations, I’ve found the story isn’t as binary as “hero” versus “villain”—it’s more like “hero” versus “erratically-troubled-if-ultimately-victorious anti-hero.” The reality of AI is nuanced—and much more interesting, depending very much on the implementation approach, governance scheme, and where organizations choose to apply AI technologies.

    The Toil Problem Is Real and Measurable

    Every developer knows the frustration of “toil”—those manual, repetitive tasks that consume time without adding much value. Toil isn’t just a minor annoyance. The 2024 DORA report confirms that this challenge remains persistent, with developers continuing to struggle with time allocation between valuable work and repetitive tasks.

    Recently, I had the chance to discuss AI heroism and anti-heroism with the Chief Product Officer of CloudBees, Shawn Ahmed. CloudBees is an enterprise DevOps solutions provider and was co-founded by Jenkins creator Kohsuke Kawaguchi, so it has been in a good position to observe the thousand ways AI has been injected into software development. Their internal research indicates developers spend about half of their time triaging test failures, waiting on build times or addressing security issues—and as little as 20–30% of their time writing code. As Ahmed pointed out to me, this inefficiency seems like it wouldn’t be tolerated in most other industries; it is like accepting a manufacturing line that’s only 20–30% effective.

    An example of CloudBees’ approach to incorporating AI productively and helpfully is their continuous integration (CI) capability. It analyzes historical pipeline data to identify patterns in successful deployments and uses AI to provide suggestions for future deployments. This eliminates hours of configuration work, thanks to AI’s ability to base pattern recognition on thousands, if not millions, of data points.

    From CloudBees’ perspective, AI in software development isn’t a replacement technology in dev but rather what Ahmed characterizes as a “companion.” AI’s best role is handling routine tasks to preserve and expand the human developer’s time performing creative and strategic work.

    Three Implementation Models, Three Different Outcomes

    It’s clear that not all AI implementations are created equal. I see three distinct approaches emerging, each with dramatically different results:

    1. The Replacers: Some organizations approach AI to reduce headcount by automating development tasks. This model typically delivers poor results, fundamentally misunderstanding both AI capabilities and the creative nature of software development. The DORA research supports this concern, showing that while AI adoption is widespread, 39.2% of respondents reported having little or no trust in AI-generated code, indicating that successful implementations require human oversight. 
    2. Tools First!: These organizations adopt individual AI tools without a cohesive strategy or governance framework. This approach delivers mixed results—productivity improves in specific areas, but in many cases at the cost of quality, security, or long-term maintainability. The DORA findings highlight this challenge, showing that despite AI’s positive impact on individual productivity and code quality, it can lead to decreased software delivery performance.
    3. The Augmenters: The most successful approach treats AI as an amplifier of human capabilities rather than a replacement. The DORA research validates this model, showing that AI adoption leads to meaningful improvements: when AI adoption increases by 25%, individual productivity increases by 2.1%, flow improves by 2.6%, and job satisfaction rises by 2.2%.

    CloudBees is built on this augmentation philosophy. According to Ahmed, AI handles pattern recognition and repetitive tasks while humans make decisions on strategic matters. Their DevSecOps solution demonstrates this approach by using AI to identify security vulnerabilities while preserving human authority over remediation strategies. CloudBees’ Value Stream Analytics uses AI to spot workflow inefficiencies while leaving process improvement decisions to the teams who understand the organizational context.

    Making AI the Hero of Your App Dev Story

    Here are five critical success factors that determine whether AI becomes a hero or an anti-hero for you:

    1. Target high-toil activities first. Successful implementations start with well-defined, repetitive tasks where AI can deliver immediate value. CloudBees’ recommended targets include test generation, pipeline optimization, and code reviews, including areas where AI can significantly reduce manual effort.
    2. Establish governance first, deploy AI second. Organizations seeing positive results implement governance frameworks before widespread AI adoption, defining, for example, appropriate use cases, verification requirements, or clear boundaries for AI autonomy.
    3. Invest in developer education. Contrary to some narratives, successful AI implementation requires more skilled developers. Top-performing organizations invest in training programs that help developers understand both AI capabilities and limitations.
    4. Integrate rather than isolate. Rather than adopting disconnected point solutions, effective organizations implement AI capabilities that integrate throughout their delivery pipeline. CloudBees’ platform takes this approach, connecting AI capabilities from requirements analysis through production monitoring.
    5. Measure meaningful outcomes. The most effective implementations have moved beyond measuring code generation metrics. Instead, they focus on quality improvements, security posture enhancements, and business impact indicators that reflect the true value of AI adoption.

    AI Is Your Enthusiastic Partner, Not Your Replacement

    Successful AI implementations in app dev seem to share the common characteristic of using it as a complementary force instead of as a replacement. AI is adept at the mundane, repetitive tasks that drain motivation and attention among developers and increase their toil. It lets humans focus on innovation, customer experience, and ultimately, business value.

    According to Ahmed, CloudBees envisions a future where AI becomes increasingly personalized to individual developers. He predicts that within two years, developers will likely work with multiple AI companions tailored to their specific preferences and working styles. This personalization aspect represents an important evolution beyond today’s more standardized AI tools. 

    A particularly noteworthy perspective I got from research and vendor discussions is the relationship between AI adoption and team scaling. While some organizations initially approach AI to reduce headcount, the data suggests the opposite effect. As Ahmed noted in our discussion, the more effective approach isn’t using AI to reduce a 10-person team to 5, but rather leveraging AI to assist that 10-person team in achieving what would have previously required 15 people. This is the proactive, business-building “scale up, not down” perspective that can help propel an organization to its goals.

    I’ve concluded that today, the hero of software development isn’t AI. Not AI alone, but it is no longer purely human expertise either. The true hero is the partnership between them, working in concert to create outcomes neither could achieve independently.

  • From Noise to Narrative: Rethinking Observability for AI-Augmented DevOps Pipelines 

    From Noise to Narrative: Rethinking Observability for AI-Augmented DevOps Pipelines 

    Because DevOps teams deal with so much data, they may feel like they are drowning, but rarely find the insights they need. There is more observability data now than ever, yet it often stays separate, disorderly, inaccessible or confusing. As AI enters more organizations’ processes, observability moves from just monitoring to gathering early insight through meaningful narratives. It is not only about having new tools but also changing how development is done and organized. Supporting today’s DevOps — where AI is involved — requires developers to switch from seeing observability as a solely technical aspect to thinking of it as a way to explain what happens in the system: One that links unrelated incidents, highlights how each one is related, and warns users ahead of any failure. 

    The Telemetry Tsunami 

    Legacy monitoring was created for stable, predictable systems, including static servers, stable networks and well-known failure points. Nowadays, architectures made of ephemeral containers, dynamic microservices and AI-driven automation result in high-volume and high-velocity telemetry data that older monitoring systems cannot handle. 

    AI models add more factors to consider in analyzing systems – model drift, low data quality, inference delays and unclear ways AI decides make things difficult to observe. Logs and alerts are insufficient to make sense of what is happening in such systems. Observing these complex, AI-driven systems calls for tracing and feedback that can reveal valuable insights with a lot of noise around them. The result? As a result, most teams experience so much data that it’s hard to find the reason behind any performance, regression or security problems. 

    Observability as Narrative 

    Moving forward, DevOps teams need to treat observability as a way to construct coherent stories. DevOps teams should focus on building stories from the signals they gather that help explain what happened and what to do next. This shift involves three fundamental changes: 

    Correlated Context Over Raw Data: Systems should first add metadata like deployment IDs, feature flags, user behavior and model versions to metrics, traces and logs to describe their life cycle independently (refer to figure 1). 

    Declarative Telemetry Pipelines: Bringing AI/ML into the system will allow systems to describe their activities using words humans can easily understand. A pipeline should ensure that the effects of an AI model update on latency can be noted and followed, not simply added later on. 

    Semantic Layering with AI Assistance: Instead of being used only in the system, AI/ML can simplify understanding what is happening. With the help of AI/ML, observability tools are becoming easier for humans to understand. 

     

    Figure 1. The three pillars of observability that are essential in an AI-augmented DevOps pipeline. 

    From Dashboards to Dialogues 

    Most current observability tools rely solely on the user to interpret data. However, more practitioners are starting to wonder: Why can’t the system simply offer explanations? It all boils down to how observability tools combine AI to help sum up what’s happening, pinpoint regressions and provide possible explanations. With the help of AI, systems can now describe issues, indicate regression points and identify probable problems in the data. An AI-augmented observability system could tell you: “Starting at 12:06, service X saw a 30% increase in latency because many cold starts from model Y, which had just been redeployed recently. Service level objectives (SLOS) did not meet expectations by 12:12. Consider raising the pool size of warm data or re-adjusting the model’s time frame for batch processing”. 

    This is no longer science fiction but something that is becoming true. Several open-source and commercial solutions integrate machine learning into observability stacks to help find root causes, group anomalies and make suitable suggestions for resolving issues. 

    Designing for Explainability 

    AI models used in DevOps also require monitoring for issues, as they can perform worse than expected for reasons that might not be obvious in tests. Unlike standard software, changes in AI models can be challenging to detect until they appear in production. Sometimes, a model that looks good during testing can have problems in the real world, caused by small or unusual changes in the input data. 

    Explainability is key. AI observability must track: 

    • Input/output distributions over time 
    • Feature importance changes 
    • Data pipeline lineage 
    • Confidence scores and decision paths 

    When correlated with infrastructure and application data, this information will tell teams if the incident was caused by a system problem, an issue with the pipeline code, or faulty model behavior. 

    Practitioner-Centric Observability 

    For observability to progress, it must be designed for all system users, not just for central teams. This means that not only platform teams but also the mid-level workers responsible for trying to understand user complaints, triaging incidents, or fine-tuning production models need observability tools. 

    Some key principles include: 

    • Narrative-Driven Interfaces: Less reliance on dashboards; more chances to look at timelines, see how things are linked and get simple explanations from AI. 
    • Integrated Feedback Loops: Ability for engineers to fix, add notes to and correct system alerts so the information shown in the future gets better. 
    • Cross-Domain Visibility: Bringing together data about how applications are doing, how the systems are working and how well your AI is doing all together into one place to help you see your systems better. 

    Observability must evolve, so it is smarter than simply collecting more data. It will require the entire culture of the industry to change, not just new technology. People in the industry should start seeing observability as telling a story, where each log, trace and metric adds to how the system can be easily explained. This evolution means that your systems must be able to define themselves well, rather than simply based on how much data you can collect. 

  • Datadog Adds Raft of Additional AI Agents and Tools to Portfolio

    Datadog Adds Raft of Additional AI Agents and Tools to Portfolio

    Datadog at its DASH 2025 conference today previewed a raft of artificial intelligence (AI) agents that automate tasks ranging from assessing infrastructure alerts that normally require the expertise of a site reliability engineer (SRE), to fixing code and triaging cybersecurity issues using the company’s security information event management (SIEM) platform.

    In addition, Datadog is also adding Proactive App Recommendations that analyze telemetry data to suggest the next best action and APM Investigator to identify and trouble shoot bottlenecks impacting application performance.

    At the same time, Datadog is adding data observability tools, along with extensions to its service for monitoring large language models (LLMs), to add support for AI agents and previews of tools to test and validate the impact of prompt changes, model swaps or application changes on the performance of LLM applications. The company has also added an AI Agents Console to make it simpler to discover and invoke its AI agents. Datadog LLM Observability, now generally available, monitors the integrity of AI models and performs toxicity checks that look for harmful behavior across prompts and responses, while Datadog Workload Protection monitors interactions between LLMs and their host platforms. A Datadog Cloud Security and Sensitive Data Scanner (SDS) also enables organizations to meet compliance requirements.

    Datadog is also making generally available Datadog Code Security, a service that leverages AI to detect, prioritize and remediate vulnerabilities in their custom code and open-source libraries based on runtime threat activity and business impact. It has also extended the capabilities of its log management suite to run in an on-premises IT environment—meeting data sovereignty requirements in regulated industries—and to help reduce costs with a lower-cost archive tier of storage.

    Finally, Datadog has added a managed internal developer portal (IDP) that is integrated with the company’s core application performance monitoring platform. It provides a live system of record for telemetry data to surface what software is running, who is responsible for it, and how it is performing. There are also templates powered by Datadog’s App Builder and Workflow Automation to automate tasks such as creating the scaffolding for a new service along with engineering reports and scorecards to track compliance, reliability, security, observability, cost and other metrics.

    In general, Datadog is making use of a range of proprietary and open source LLMs, including one optimized for time-series data that it has developed. Over time, Datadog plans to mix and match LLMs to address specific use cases as additional AI advances are made.

    Michael Whetten, vice president of product for Datadog, said, for example, the latest Bits AI SRE, Bits AI Dev Agent and Bits AI Security Analyst agents not only extend the range of tasks being automated, but also enables them to query data, analyze anomalies or scale infrastructure using shared memory in a way makes it possible for AI agents to reuse functions. Datadog has also developed a Model Context Protocol (MCP) server to make it simpler for AI agents to consume its data and interoperate with each other. The overall goal is to not, for example, replace SREs, but rather augment their expertise in a way that makes it easier to manage complex IT environments, he added.

    It’s still early days so far as adoption of AI technologies across DevOps workflows is concerned, but a recent Futurum Research survey finds 41% of respondents expect generative AI tools and platforms will be used to generate, review and test code. In fact, arguably the challenge now is not so much deciding whether to use AI as much as it is determining where AI will have the biggest impact on improving both software quality and the speed at which applications are deployed.