Category: DevOps and Open Technologies

The conversation around AI agents has evolved remarkably quickly. Not long ago, most organisations were still experimenting with prototypes and controlled use cases, trying to understand where these systems might fit within existing operations. Today, that conversation has changed. Agents are increasingly being deployed into production environments where they can write code, interact with customers, query databases, trigger workflows and perform tasks that previously required direct human intervention. As adoption accelerates, much of the discussion continues to focus on capability.

Organisations want to know how much productivity agents can unlock, how much manual work they can remove and how quickly they can be integrated into day-to-day operations. Those are important questions, but I increasingly believe they are overshadowing a more fundamental one: whether organisations can adequately account for what these systems are doing once they are deployed.

What makes this particularly interesting is that modern engineering has spent decades solving a closely related problem. As software systems became larger and more complex, organisations invested heavily in observability. Logging, monitoring, tracing and audit trails became standard practice because understanding system behaviour was no longer optional. When something went wrong, teams needed to know what happened, how it happened and where responsibility lay. Over time, observability evolved from a useful operational enhancement into a core component of production infrastructure. Most mature engineering organisations would not dream of deploying critical systems without it. Yet as AI agents move from experimentation into production, many organisations appear to be repeating an earlier mistake by focusing on capability before building the accountability mechanisms required to govern it.

Part of the challenge lies in the way agents are being understood. The industry often talks about them as though they are simply another layer of software. In practice, they behave differently. Traditional software executes predefined instructions. Engineers can inspect the logic, trace the execution path and usually reproduce the conditions that led to a particular outcome. Agents operate with a greater degree of autonomy. They receive objectives, evaluate context, select tools and determine actions within the boundaries of the permissions they have been granted. The distinction may appear subtle, but it changes the nature of accountability entirely. When an agent updates a customer record, executes a transaction, modifies a configuration or interacts with another system, the most important question is often not what action occurred, but why that action was chosen in the first place.

This is where many organisations begin to encounter a visibility problem. Existing observability practices are largely designed to capture actions and outcomes. They can tell us which API was called, which workflow was triggered and whether a transaction completed successfully. They are far less effective at capturing the reasoning that led to those actions. An organisation may be able to see that an agent performed a task while remaining unable to explain what information was available at the time, what alternatives were considered or why one course of action was selected over another. That distinction rarely attracts attention when systems are behaving as expected. It becomes far more significant when something goes wrong.

The issue is not hypothetical. As agents gain access to increasingly sensitive systems, the potential consequences of poor decisions become more material. An agent can follow every technical rule available to it and still produce an undesirable outcome. The infrastructure can remain healthy. No service may fail. No alert may trigger. Every system involved can behave exactly as designed while the final decision remains flawed. This is one of the reasons agent-related incidents often feel different from traditional software failures. The challenge is no longer confined to identifying what happened. It increasingly involves reconstructing why it happened, and that becomes difficult when the decision-making process itself was never captured.

The regulatory environment is beginning to expose this gap more clearly. Much of the discussion around frameworks such as the EU AI Act focuses on compliance obligations, but beneath those requirements sits a broader reality. Regulators are asking organisations to demonstrate traceability, oversight and accountability for automated decisions. They are effectively asking questions that many organisations cannot yet answer consistently. What informed this decision? Who was responsible for oversight? What authority was the system operating under at the time? How can the decision be reconstructed after the fact? These are not purely legal questions. They are operational questions that depend on the existence of technical infrastructure capable of producing reliable answers.

What strikes me most is how closely this resembles earlier stages in the evolution of software operations. There was a time when observability itself was considered a secondary concern, something to be addressed after deployment rather than designed into systems from the outset. As digital infrastructure became more critical, that approach became unsustainable. Organisations discovered that the ability to understand behaviour was just as important as the ability to deliver functionality. I suspect accountability is following a similar trajectory. As AI agents become a more permanent part of enterprise architecture, the organisations that succeed will not simply be those deploying the most capable systems. They will be the ones building the strongest mechanisms for understanding, governing and explaining those systems once they are operating at scale.

The next phase of enterprise AI will almost certainly deliver more capable agents, more sophisticated models and greater levels of autonomy. Those developments are likely to dominate headlines and investment discussions for some time. Yet capability alone rarely determines whether a technology becomes sustainable. Trust does.

Organisations can only place meaningful trust in systems they can understand, investigate and govern. The challenge facing many enterprises today is that they are scaling agent capability much faster than they are scaling agent accountability. Eventually, that imbalance will need to be corrected. The organisations that address it early will find themselves in a stronger position not only to satisfy regulators, but to deploy AI with the confidence that comes from knowing they can explain what their systems are doing long after those systems have acted.