Five incidents in one month is a lot for any platform, but it’s especially notable when that platform hosts most of the world’s code review, CI/CD, and now AI coding agents. GitHub’s August 2026 availability report reads less like a list of unrelated hiccups and more like a pattern: the company’s own infrastructure is straining to keep up with how fast usage of Actions and Copilot has grown.
The report opens with a line that doubles as GitHub’s operating philosophy right now: “Availability, then capacity, then features.” That ordering matters, because most of August’s incidents trace back to capacity limits, not bugs in new features.
The first hit came August 6, when a routine deployment reduced pod capacity in one datacenter. That triggered service mesh saturation and cascading failures across multiple clusters, knocking out Actions (both hosted and self-hosted runners), Copilot’s coding agent, code review, Pages builds, Dependabot, and repository migrations for roughly nine hours. GitHub’s own postmortem put it bluntly: “The affected actions services were running close to their capacity and concurrency limits.” The fix was a rollback, expanded capacity, and a fix for a latent bug that had let runners pick up jobs they couldn’t actually run.
Eleven days later, on August 17, a traffic peak exceeded datacenter load balancer limits. A service-mesh sidecar failed to scale, network flow limits got exhausted, and the shared authentication path degraded. The blast radius was wide: issues, pull requests, both APIs, Actions, Copilot, authentication, and webhooks all took a hit. Peak front-door failure rate reached 56%, and about 29,000 organizations saw errors across roughly 4.8 million requests. GitHub also flagged a compounding factor familiar to anyone who has debugged a distributed system under load: “A latent client retry bug sharply amplified traffic to one internal authentication endpoint.” Retry storms make bad days worse, and this was no exception.
August 20 brought a different kind of failure, tied to Copilot’s cloud agent. An upstream managed database had a regional outage, and a storage configuration issue slowed the failover. Tasks kept running, but status and results visibility lagged for up to 90 minutes at more than 54 organizations over nearly 11 hours.
Mitch Ashley, vice president and practice lead for CIO & Technology Buyers and Software Lifecycle Engineering at The Futurum Group, said that the incident points to a gap most engineering teams haven’t closed yet. “Ordering availability ahead of features is the right call,” Ashley said. “The Copilot agent incident showed why: tasks kept running while status lagged, so teams could not tell what had finished. Engineering leaders should require agent status checks they own, outside any vendor’s UI.”
Then came back-to-back incidents on August 26 and 27. The first was a database saturation problem tied to Actions run starts, Pages deployments, and Copilot code review, with GitHub owning the underlying issue directly: “Our shared infrastructure services have not kept up with our month-over-month actions growth and peak load.” At least 24 organizations saw run-start failures, and 386 organizations felt some impact before manual throttling brought things back under control. The next day, a separate issue hit only Copilot requests routed to the Kimi K3 model, with 63% of those requests failing due to a problem at the upstream model provider. Other models were unaffected, but it’s a useful data point on how much of Copilot’s reliability now depends on providers GitHub doesn’t fully control.
For platform teams, the throughline across all five incidents is the same: growth in Actions usage and AI-assisted workflows is outpacing the shared infrastructure meant to support it.
Ashley sees that as more than an infrastructure problem. “GitHub is bidding to become the surface agents run on, which changes what availability means,” he said. “With Actions and Copilot in the delivery path, an outage there is a production incident for every team downstream.”
GitHub isn’t ignoring the pattern. Alongside the incident writeups, the company detailed parallel infrastructure work: moving MySQL primaries to Azure, cutting roughly a million queries per second from its database load, routing a third of Actions jobs to spare capacity, and expanding pull request isolation to authenticated reads. None of that is glamorous, and none of it will show up in a product announcement. But it’s the plumbing that determines whether a platform this central to software delivery can keep scaling without buckling every few weeks.
That’s the real takeaway for engineering leaders who depend on GitHub as critical infrastructure, not just a code host. Actions and Copilot have become part of the software delivery pipeline itself, which means an outage there isn’t an inconvenience — it’s a production incident for every team whose builds, deployments, and reviews run through it. Teams that haven’t already should be asking what happens to their pipeline when GitHub has a bad day: whether critical deployments have a manual fallback, whether long-running agent tasks have their own status checks outside GitHub’s UI, and whether retry logic on their end could make an incident worse instead of better.
GitHub deserves credit for its transparency here. Five detailed postmortems in one report, each naming a specific root cause, is more candor than most vendors offer. But candor doesn’t replace capacity. The next few months of availability reports will say a lot about whether GitHub’s infrastructure investments are keeping pace with a platform that’s becoming more central to how software gets built.
Enterprise AI agents need more than stronger models. They need durable execution environments that can coordinate multi-step workflows, survive failures, pause for human review and resume reliably after disconnects or delays.
AI Agents Have Moved Beyond Chat Demos
AI agents work beautifully in demos. A user asks a question, a large language model generates a response and everyone sees possibility. Production systems are different.
Consider what these systems are now asked to do. An agent assessing a change request may pull deployment history, evaluate blast radius, check freeze-window policy, wait for a release manager’s sign-off and then schedule the rollout. An agent adjudicating an insurance claim may extract fields from submitted documents, cross-check them against prior claims, apply underwriting rules and route anything unusual to a human adjuster. An agent triaging a security alert may enrich indicators, correlate against past incidents, assess asset criticality and hold containment until an analyst approves it.
Those three workflows have almost nothing in common at the domain level. They share a shape: Multiple steps, several systems, independent failure modes and at least one point where a person decides. They take minutes or hours, not milliseconds.
The challenge is no longer only intelligence; it is execution. Many enterprise AI conversations focus on model selection, prompt engineering, retrieval-augmented generation and tool calling. Those are important, but they do not answer what happens after the agent begins executing long-running work across multiple systems.
The Problem: Most Agents Are Still Designed Like Request-Response Applications
Traditional AI applications assume a simple request-response model: User input goes in; a model response comes out. That model works well for Q&A, summarization, content generation and search-like experiences.
Enterprise workflows are different. A procurement review agent might parse a contract, compare its terms against a standard template, check vendor risk ratings, invoke a sanctions screening service and produce a recommendation. These steps may execute independently, fail independently and complete at different times.
Once an agent crosses the boundary from answering a question to executing a process, it starts to look less like a chatbot and more like a distributed system. It needs state, retries, coordination, observability and recovery.
Why Stateless APIs Break Down
Teams often start by implementing an agent as a synchronous API. That approach works until the first real workflow appears. What happens if the HTTP request times out? What if the user closes the browser? What if an external system is unavailable? What if a human approval arrives six hours later?
A stateless API can initiate work, but it is a poor place to manage long-running state. Developers quickly end up building custom queues, status tables, retry logic, compensation logic and ad hoc workflow tracking. The hidden cost is not the first prototype; it is operating the system after it begins handling real workloads.
The missing layer is a durable runtime: A workflow engine that can preserve progress, coordinate parallel work, wait for external events and resume execution reliably.
Figure 1: The Same Multi-Step Workflow Under Two Runtimes
A stateless service holds progress in process memory, so a host failure discards completed work and the retry begins again from the first step. A durable runtime checkpoints each completed activity, so a replacement worker replays the history and continues from where execution stopped.
The Key Design Shift: the Workflow is the Application
The most important design shift is this: The model is not the application. The workflow is the application. The model is one activity inside a larger execution graph.
In production-grade agentic systems, the orchestrator coordinates work while specialized agents perform specific tasks. The specialization is domain-specific, but the division of labor recurs: One agent gathers and enriches raw signal, another retrieves reference material, another looks for precedent among prior cases and another evaluates policy or risk conditions.
This design makes the system easier to reason about. Each agent has a focused responsibility, and the orchestration layer owns execution flow, state, retries, progress and human interaction.
Using Durable Orchestration as the Agent Runtime
A practical architecture places a durable orchestrator between the calling application and the AI execution layer. The caller starts the workflow and immediately receives an instance identifier. The orchestrator then coordinates specialized agents and records progress as the workflow advances.
Azure Durable Functions is a strong fit for this pattern because it supports stateful workflows, activity functions, external events, checkpointing and long-running orchestration. Azure AI Foundry can provide the model and agent execution layer, while Durable Functions handles the operational execution semantics around the agent.
The result is a cleaner separation: AI services reason; activity functions perform work; the durable orchestrator governs the process.
Figure 2: The Orchestrator Between the Calling Application and the Reasoning Layer
It owns sequencing, retries, timers and external events. Specialized agents run on the model layer with narrowly scoped access and every completed activity is checkpointed so the workflow can be rebuilt after a failure or a process restart.
Fan-Out/Fan-In Maps Naturally to Multi-Agent Systems
These investigations are rarely one sequential task. Many subtasks can run independently. Enrichment, reference retrieval, precedent search and policy evaluation usually have no dependency on one another, so they can execute in parallel. This is where fan-out/fan-in becomes a natural orchestration pattern.
The orchestrator fans out work to specialized agents, waits for them to complete, and then fans in the results to generate a synthesized recommendation. This pattern improves latency while keeping the workflow easier to understand than a large monolithic agent prompt.
The fan-in step is also where engineering judgment matters. The system should handle partial results, conflicting findings and low-confidence recommendations rather than assuming every agent will succeed perfectly every time.
Example Orchestration Sketch: The sketch below triages a security alert; the domain is incidental. Rename the activities and the same orchestration serves claims adjudication or change-risk review.
Three details in that sketch are worth calling out. The external event is raced against a durable timer, so an approval that never arrives escalates instead of leaving the instance parked indefinitely. The timer is then cancelled on the success path, because a durable timer that is neither cancelled nor allowed to expire keeps the instance in the Running state long after the orchestrator function has returned. Since Task.WhenAll surfaces the first exception it encounters, genuine partial-success handling means having each activity return a typed failure result rather than throwing it once its retries are exhausted.
Human-in-the-Loop is Not an Edge Case
In regulated and high-impact workflows, human review is part of the process by design. A payment exception above a threshold, a production change during a freeze window, a claim falling outside underwriting rules: None of these should be finalized automatically when confidence is low or the consequences of being wrong are large. The important thing is to make human review part of the workflow rather than a side channel.
A durable orchestration can pause while waiting for a review event, then resume when the approver submits a decision. The system does not need to keep an HTTP request open or hold compute resources while waiting. The workflow state remains available, and the final recommendation can include the reviewer’s context.
This capability is especially important because the slowest step in a production workflow is often not the model call. It is the human decision, the external dependency or the operational handoff.
Figure 3: One Instance Over Time
Parallel agents make wall-clock cost the slowest agent rather than the sum of all of them; a failed agent degrades the result instead of failing the run, and the review window consumes no compute because the orchestration is parked rather than blocked.
Operational Considerations Matter as Much as Prompts
A durable runtime does not remove the need for engineering discipline. It makes that discipline easier to apply. Production-grade agents still need observability, correlation identifiers, retry policies, cost controls, security boundaries and careful versioning.
Workflow telemetry should answer questions such as: Which stage is running? Which activity failed? How many retries occurred? How long did human review take? What evidence supported the final recommendation? These are operational questions, not prompt engineering questions.
Retries should be designed around external dependencies such as AI model calls, search systems, databases and APIs. At the same time, not every failure should fail the entire workflow. Multi-agent systems should be designed for partial success where appropriate. If the precedent agent fails but enrichment and retrieval succeed, the system may still produce a useful recommendation with clear caveats.
Three constraints tend to surface only once a workflow reaches production. Orchestrator code has to stay deterministic, because recovery works by replaying it: Wall-clock reads, random values, fresh identifiers and direct I/O belong in activities, not in the orchestrator. Every wait needs a deadline, else an approval that never arrives becomes an instance nobody ever looks at again. Versioning deserves a decision before the first production deploy because long-running instances guarantee that new orchestrator code will ship while old instances are still in flight.
Security and Governance Cannot Be Bolted on Later
AI agents often sit at the crossroads of sensitive enterprise data, operational tooling and user-facing experiences. That makes security and governance foundational design concerns.
Each specialized agent should receive only the access necessary for its responsibility. A retrieval agent may need the document corpus and nothing else. An enrichment agent may need read access to telemetry. The agent that composes the final decision may need only summarized evidence, never the raw sources behind it. Avoid creating one over-permissioned super-agent that can reach everything.
Prompts and outputs should also be auditable. Store workflow version, prompt version, model deployment, agent version, evidence references and correlation identifiers. As agents become part of operational decision-making, teams need to explain not only what the system recommended but why it recommended it.
Figure 4: Scoping Access per Agent Contains the Blast Radius
A single agent holding every credential turns one poisoned input into estate-wide exposure, while narrowly scoped agents limit any single compromise to one agent’s data, with the synthesis step working from summarized evidence.
What This Means for DevOps and Platform Engineering Teams
The future of enterprise AI is not just better prompts. It is reliable execution. As agents become responsible for operational workflows, DevOps and platform engineering teams will need to provide runtime capabilities similar to those used in distributed systems: Orchestration, observability, resiliency, governance and deployment discipline.
This resembles the evolution of microservices. Early conversations focused on APIs. Later, the industry learned that production systems also require service discovery, telemetry, deployment strategies, circuit breakers and operational practices. AI agents are entering a similar phase.
Organizations that treat agents as production workflows rather than chat interfaces will be better positioned to build systems that users trust.
Conclusion
Most enterprises already have access to powerful foundation models. The harder challenge is building systems that can execute reliably when work spans minutes, hours, tools, services and human decisions.
Long-running agents of any kind need a durable execution layer. They need to coordinate multiple specialized agents, survive transient failures, pause for human review, provide progress visibility and generate auditable outcomes.
Azure Durable Functions and Azure AI Foundry offer one practical way to approach this architecture: Durable Functions provides orchestration and stateful execution, while Azure AI Foundry provides the intelligence layer. The broader lesson applies beyond any specific platform: Production AI agents need durable runtimes. Intelligence is only half of the system. Execution is the other half.
The most expensive mistake teams are making with AI agents right now is treating them as microservices with a language model bolted on. The approach is understandable; microservices are the architecture most platform engineers know best, agents feel like services that happen to think, and the migration path looks familiar. But the analogy only holds long enough to lead you somewhere painful.
Microservices changed how software gets built. The idea was to help teams work independently, ship faster and scale efficiently. In those days, shipping software meant one giant codebase compiled together, tested together and released on 12- or 18-month cycles. Then the cloud came along, and teams needed to ship in weeks or days. The only way to do it was to split systems apart and let each piece move on its own schedule.
It worked, but the systems underneath stayed simple in one crucial way: they were deterministic. Distributed systems are hard, but the behavior was predictable. A service received a call and returned a result, and when it failed, it failed in milliseconds and threw an error you could see.
That is the assumption agents break. An agent workflow can run for hours or days, touch a dozen systems, and make a non-deterministic decision at step three that you don’t discover was wrong until step forty. Nothing threw an error, nothing lit up red on a dashboard; the process just quietly went the wrong way and kept going. Where a broken microservice pages you, a misbehaving agent sends a confident, wrong result and moves on.
This is why the microservices playbook both applies and doesn’t, and why “agents are just distributed services” is a trap. Agents do certainly build on microservice foundations, and many of the era’s lessons carry straight over. But the practices that were optional when systems were deterministic and short-lived are now mandatory, and even the ones that carry over don’t carry over cleanly. Treat agentic AI as just another microservice, and you’re making an architectural bet that will come due in the next 12 to 18 months.
Here are the lessons the agent era rewrites, and the ones it simply reinforces.
The Lessons That Get Rewritten
1. Durable execution stops being a pattern and becomes the runtime. This is the primary one, and it’s where the microservices instinct fails quietly. In a deterministic world, you could add resilience later, layering retries and queues around services that mostly completed on their own; the process finishing was the default, and durability was just another pattern. That default is gone. When an agent workflow fails five steps into a multi-day run, detecting that something broke is often harder than recovering once you know, and the resilience you meant to add later never gets added. It has to be part of the runtime from the start.
But here’s the part teams miss: it is not the same durability. The durable execution engines built for the microservices era assumed high concurrency and millisecond steps, payment flows, order processing, thousands of shorter workflows multiplexed per worker and agents violate every one of those assumptions. A single step might be a 45-second model call. Another might pause for hours while it waits on a human, and during that wait the process should be able to die and resume cleanly later, because nobody cares whether the resume takes seventeen milliseconds or two seconds. You cannot even replay a step and expect the same result, because the step is non-deterministic by nature. So the answer isn’t to adopt the durability you skipped; it’s that the durability layer itself has to be rebuilt for a workload it was never designed to carry.
2. Security and governance need a new model, not a stronger boundary. Microservices pushed authentication to the edge, a fence around the building that worked when the actors inside were fixed services you deployed. Agents can start other agents at runtime, which raises questions the boundary model simply can’t answer. Does the sub-agent inherit the same permissions? How do you trace what it did on your behalf three steps later? The model that fits is less a fence and more a series of locked doors, each requiring its own credential, with identity and cryptographic attestation following every step of every workflow, so you can prove not just who called, but what acted and what it was allowed to do. Most teams are still guarding the front door of a building that now rearranges its own rooms.
3. Observability has to reach every step, not just the endpoint. Operators need to retain control, and non-determinism makes that harder than it ever was for microservices. When something breaks in a 100-step workflow, an operator needs to see exactly where it broke and why, not just that it failed. Typically, a status indicator on a dashboard was enough when a service either returned or errored, but it is useless when the failure is a plausible-looking wrong turn ten steps back. Distributed tracing was essential for microservices. With agents, you’re no longer debugging whether a call succeeded; you are reconstructing why an autonomous system made the choice it did. That demands full visibility into every step an agent takes, as a first-class property of the platform rather than a log you grep after the fact. It’s the difference between the application performance monitoring (APM) tools of the microservices era and LLM observability tools in the agentic era.
These Three Demands Have a Name
Treated separately, recovery, per-step identity, and per-step visibility look like three infrastructure projects competing; they aren’t. They’re three faces of one capability the agent era requires, and it’s worth naming, because you can’t build toward something you can’t name.
Call it agentic durable execution: a runtime that automatically recovers from failures, carries autonomous AI agents and deterministic workflows to completion, and provides cryptographic, tamper-proof evidence of what happened. The first two answer the durability problem: the process finishes, while the third answers governance and observability. In a non-deterministic system, proving what an agent did and being able to see and trust it are the same requirement. It is not a feature you add to an agent framework; it is the layer the framework should be standing on.
The Lessons That Carry Straight Over
These three lessons didn’t change. They just matter more, because the cost of ignoring them is higher.
4. Complexity still demands a platform. Around 2010, most teams were stitching together virtual machines with Chef and Puppet, trying to build something that resembled a distributed application. Kubernetes won because it gave teams a platform layer that absorbed that complexity instead of asking every team to rebuild it. The same pattern is repeating now, as every organization races to stitch together agent frameworks and rebuild infrastructure a platform should already provide. The stakes are just higher this time, because the failures are harder to see coming.
5. Vendor lock-in is still costly. Committing to a single cloud provider’s agentic stack seems efficient until that provider has a regional outage and stalls every dependent workflow, and data sovereignty rules, especially across the EU, only add more pressure to a single-cloud strategy. The microservices lesson holds: architecture decisions made for short-term convenience almost always turn into long-term constraints.
6. Open standards still win the long game. The cloud-native projects that lasted, Kubernetes chief among them, were built in the open and governed by communities rather than single vendors, which is why they worked across environments instead of locking teams in. The agent world is already showing the same trait, with standards for how agents communicate and authenticate emerging through open governance rather than proprietary roadmaps. Betting against openness didn’t pay off for infrastructure a decade ago, and there’s little reason to expect it will now.
Why the Timeline is Shorter This Time
Organizations that approached microservices thoughtfully ended up with systems that scaled cleanly, while those that rushed are often still paying down that architectural debt years later. The same is playing out with agentic AI, except the debt comes due faster. With microservices, a bad decision surfaced as a broken deploy or a paged engineer, but with agents, non-determinism compounds silently and autonomously across long-running workflows, so the debt accrues in decisions you can’t see until the blast radius is large.
The lessons are already on the table. The teams that will win the agent era will be those who recognized which lessons carried over, which ones didn’t, and built for agentic durable execution before the debt came due.
A back-end team extracted notification delivery from a large Ruby monolith into a centralized Go service, only to discover that their shared RabbitMQ cluster could not reliably support the new workload. Here’s how they solved the problem using MongoDB as a polling queue and the trade-offs that came with it.
The Scaling Problem Inside the Monolith
Back in 2019, the engineering team at Wheely, a popular ride-hailing platform, had a coordination problem that emerged with the product. User-facing notifications such as push alerts, SMS confirmations and status updates were scattered around the large Ruby monolith. Various parts of the codebase made direct attempts to send messages without a shared logic, unified retry strategy and consistent way to track delivery outcomes. As the system continued to grow, new back-end microservices were introduced, with each one requiring its own path to the user’s device.
Push campaigns made things worse. Sending thousands of notifications at once exposed limits that routine traffic had never revealed. The monolith wasn’t designed as a notification delivery system, and scaling it that way implied additional risks to the core business logic already implemented there.
The team needed a dedicated, centralized service that could serve as a single point of entry for all outbound communications, both for the monolith and future microservices alike.
Why the Existing Message Broker Wasn’t Enough
The natural architecture for this kind of service involves a message broker. The monolith and microservices publish notification events, workers consume them and delivery happens asynchronously. The team’s infrastructure already had RabbitMQ in place, so the initial design assumed it would handle this workload.
In practice, the shared RabbitMQ cluster proved unreliable under the notification service’s workload. Unstable behavior of the cluster during large push campaigns caused several delivery failures, which were difficult to diagnose, hard to retry correctly and disruptive to other services that used the same broker.
RabbitMQ Internals
To understand why RabbitMQ turned out to be a bottleneck, it’s worth taking a quick look at its internal architecture.
RabbitMQ is built on top of the Erlang virtual machine: BEAM, where almost every broker component runs as a separate Erlang process. Queues, connections, channels and consumers are independent lightweight processes, communicating with each other via messaging.
This model provides strong fault tolerance and scalability in various scenarios. However, under a very heavy load, the overhead of managing these processes can become noticeable. Moreover, RabbitMQ is also sensitive to the type of load it receives.
The first factor is the number of queues. Each queue is a separate Erlang process with its own internal data structures. As the number of queues increases, more memory is used, the Erlang VM scheduler starts working harder and there is an additional operational overhead from operating the broker.
Second, the length of the queue matters. RabbitMQ was designed as a system to process continuous message flow, not to store very large backlogs for long periods. If producers write faster than consumers can read, the queue starts growing quickly. It results in higher memory usage, puts additional pressure on the disk for persistent messages and significantly increases delivery delay.
Finally, RabbitMQ clusters are sensitive to network conditions. Replication between nodes, synchronization of queues after recovery and network latency can all significantly impact the system’s performance and cluster recovery time after failures.
These characteristics are not flaws in RabbitMQ; they simply define the type of loads it is best suited to handle. When designing a high-load system, it is important to consider not only the broker’s capabilities but also the workload profile it will need to support.
Why RabbitMQ Was No Longer the Right Fit
RabbitMQ had handled its role well for a long period. The problem was not with the broker itself, but the change in workload that took place over time.
First, the system was increasingly dealing with large-scale push and SMS campaigns. A single campaign generated tens of thousands of individual messages, all arriving in RabbitMQ practically at the same time. This created sharp load spikes, made queues grow and increased delivery delays.
Second, some consumers were still running inside the Ruby monolith. Compared to the newer Go services, they were slower at processing messages, allowing queues to accumulate much faster than they could be drained. Simply increasing the number of consumers wasn’t a practical option. Performance of the monolith had already become a limiting factor, and scaling it would have been complicated due to the current system’s architecture.
Finally, RabbitMQ served as a shared infrastructure component for various services. Every new communication-related feature added load to the same cluster, making it an increasingly critical point of failure.
Extracting a Centralized Go Notification Service
The new service was implemented as a single, stateless API responsible for accepting send requests and handling everything else: Routing to the target provider, retrying failed deliveries and recording the outcome. The monolith and other microservices sent requests to a single endpoint without having to manage the delivery process themselves.
Go was chosen for the implementation of the new service. It was containerized and deployed using Docker on the existing AWS infrastructure, consistent with the rest of the microservices on the platform.
On the mobile side, the service integrates with the Apple Push Notification service (APNs) and Firebase Cloud Messaging (FCM). SMS delivery was handled via integration with one or more third-party providers.
Using MongoDB as a Polling Queue
Since it was not possible to rely on any broker, it was necessary to find some alternative way of buffering delivery tasks and processing them. The alternative chosen by the team was the usage of MongoDB as a polling queue. MongoDB was one of the tools used in the current stack.
When a notification request arrives, the service writes a document to a dedicated MongoDB collection. Each document represented a single delivery task and contained all the information needed to be processed independently: The recipient, the message payload, the channel, the current status and metadata to retry tracking.
A simplified document life cycle looked like this:
pending → processing → delivered
↘ retry → (back to pending, up to N attempts)
↘ failed (dead-letter)
Workers polled the collection at short intervals, querying for documents in the pending state. To process two workers trying to process the same task, MongoDB’s atomic findOneAndUpdate operation was used to transition a document from pending to processing in a single step. Only one worker succeeded in claiming any given record.
Retry policy was embedded in the document itself. Each failed attempt incremented a counter, as well as updated a retry_after timestamp using an exponential backoff strategy. Workers excluded documents whose retry_after time had not yet passed. Documents that reached the maximum retry count were moved to the failed state rather than being tried to be processed again.
Idempotency was maintained by creating a stable identifier for each delivery task when it was created. In case the same request arrived twice due to the retry on the caller’s side, the service could detect the duplicate before writing a second document.
Processed and failed records were either archived into another collection or dropped after the retention window.
Index design was critical. The collection required compound indexes covering status, channel and scheduling fields for efficient polling queries. Without the right indexes, polling at scale would have caused unacceptable read load on MongoDB.
Optimizing MongoDB for a Write-Heavy Workload
Using MongoDB as a persistent queue required careful data model design.
First, we separated the queue into a write-heavy collection, fully isolated from the rest of the application data. This way, it was possible to optimize it independently by choosing its own indexes, configuring a dedicated retention policy and ensuring that intensive writes did not affect the rest of the system.
At the same time, we attempted to implement an append-heavy data model. Messages were created once and then updated just a few times while moving through the delivery pipeline before being either archived or deleted. This life cycle aligned well with MongoDB’s strengths and helped maintain stable write performance.
Queue documents were intentionally kept compact. They included only information needed for routing, delivery and status tracking. This reduced the amount of data transferred, lowered disk and replication load and allowed more of the working set to fit in memory.
Indexes required particular attention. We deliberately limited them to those needed for efficient polling and status-based lookups. Any additional index increases the write cost, as MongoDB must update it on every insert or update. For write-heavy workloads, keeping the number of indexes minimal becomes more important than maximizing read performance.
Messages were polled in batches instead of one at a time. During each poll cycle, the worker would claim only a certain number of tasks, which significantly reduces the number of database operations, thereby making the load more predictable in case of many incoming messages.
Finally, to prevent the collection from growing indefinitely, we either periodically archived processed messages or automatically removed them using TTL indexes. This kept both the collection and its indexes relatively small, improving write speed and the efficiency of polling queries.
It was the combination of these decisions that made MongoDB an efficient and predictable persistent queue for this workload. Observability relied on tooling already in use at Wheely: The ELK stack for log aggregation and Datadog for metrics and alerting.
The Trade-Offs We Accepted
MongoDB was not built to be a message broker, and using it as one comes at a cost.
During each poll cycle, database read operations were generated regardless of whether the queue was empty. At scale, it added measurable load to MongoDB. This load was then associated directly with notifications throughput instead of being isolated in a dedicated broker. Despite a careful index design and a sensible polling interval, it was still possible to reduce but not eliminate this overhead.
Polling introduces inherent latency. A message written to the queue would not be processed until the next poll cycle. It is a reasonable decision for most notification use cases, but it was a deliberate compromise compared to the near-immediate delivery that one can expect from a properly functioning broker.
The team also had to implement, test and maintain functionality that mature message brokers typically provide out of the box: Atomic claiming, retry scheduling, dead-letter handling, duplicate suppression and observability. Each of these capabilities added complexity to the service itself.
There was also an organizational risk. Solutions described as temporary often become permanent. Teams adopting this pattern should clearly define under what conditions that would justify migrating to a dedicated broker, and they should build sufficient observability to detect when those conditions are met.
A dedicated message broker continues to be the right choice in the case of a stable infrastructure, when the team has the operational expertise to operate it and the delivery latency requirements are strict. This architectural approach was chosen based on specific sets of constraints, not as a general recommendation.
Operational Results and Lessons
After the migration, the service handled notification volumes several times higher than the previous peak without delivery failures caused by broker instability. The service achieved a much better delivery success rate and now serves as the shared communication layer for multiple back-end services across the platform.
Three lessons worth carrying forward:
First, architecture must reflect the maturity of the infrastructure it runs on. A theoretically correct design built on an operationally unreliable dependency can perform worse than a pragmatic design built on infrastructure the team already operates well.
Second, database polling can be a viable delivery mechanism, but it requires a thorough implementation. Idempotence, atomic claiming, backoff, dead-letter handling and observability are not optional additions; but are the key to making the pattern safe. Skipping even one of them leads to failure modes that are difficult to diagnose under load.
Third, extracting a shared capability from a monolith is an operational change, not just a code refactoring exercise. The migration required a careful cutover strategy to avoid interrupting live delivery. The organizational and operational work was at least as significant as the engineering work.
Most platform teams aren’t deciding whether they’ll run across multiple clouds. They already are, or they’ll be soon. The real question is how to migrate critical systems without turning on-call into a guessing game.
Observability raises the stakes more than almost any other domain. An observability control plane isn’t just a dashboard. It’s the operational authority system. It defines alert rules, routing, ownership, escalation policy, and notification endpoints. When that layer is wrong, the impact is immediate. The wrong team gets paged. The right team never hears about the incident. Your service level indicators look clean while production burns.
A typical failure pattern is painfully simple. During a migration window, an ownership change lands in one system but not the other. A routing update is processed out of order. A notification endpoint rotates, but only one store is updated. Those discrepancies can sit quietly for days. Then a real incident hits, an alert fires, and it routes to an old escalation path. At that point, you aren’t debugging the service. You’re debugging the migration.
That’s the core point. In a control plane, being slightly wrong produces operational consequences, not cosmetic discrepancies.
The strategy that holds up is built for continuous motion, not a frozen world. In practice, it comes down to two building blocks.
Continuous synchronization between the old store and the new store
A dual read service layer that shifts read traffic gradually
The objective is to verify parity under real conditions, cut over incrementally, and roll back quickly if anything looks off.
Why Export and Import Look Clean and Still Fail
Export and import sounds straightforward. Copy the data, switch systems, and move on. The hidden assumption is that the world holds still while you copy. In a live control plane, it doesn’t.
While a snapshot is running, engineers adjust alert thresholds, update routing trees, enable and disable rules to manage noise, rotate notification endpoints and credentials, and ship changes. The data you copied becomes stale immediately. If you treat that snapshot as truth, you’re baking drift into the migration from day one.
That leaves you with an ugly choice.
Option one is to freeze writes and break operations.
Option two is to allow writes and accept drift.
In observability, drift isn’t harmless. It changes routing and timing. You often discover it only when an alert fails to fire, or routes to the wrong team, and by then, you aren’t managing a migration. You’re managing an incident.
Even when export and import work technically, it creates a cliff moment where you must declare the new system authoritative. If you’re wrong, rollback isn’t a routing change. It’s exporting again, importing again, and explaining why the project is moving backward.
Why Naive Dual Write is Usually the Wrong Default
Dual write can look like the obvious answer. Write to both systems, then cut over once you’re confident. The problem is that it pulls hard-distributed systems problems into the most fragile phase of the migration.
Here are the failure modes that show up repeatedly in real systems.
Retries produce partial outcomes. One side commits while the other times out.
Ordering diverges. Two systems can observe different sequences of updates for the same resource.
Schemas that look similar can carry semantic differences that create slow, silent drift.
One store accepts a write the other quietly rejects, and the rejection isn’t always loud.
You query the same resource in both systems and get different answers about what’s true.
If you’ve spent time in distributed systems, none of this is surprising. The problem is when it surfaces during migration, when you’ve got the least bandwidth to chase subtle cross-system correctness issues.
Why Control Planes Amplify Ordering Bugs
Control planes aren’t just storage. They’re state interpretation systems. Behavior depends on causality, not only on the final snapshot of data.
Consider a simple sequence.
Ownership changes
Routing targets are updated
An alert is temporarily disabled to control noise
If those updates land in a different order across two systems, you haven’t just created an inconsistency. You’ve changed operational behavior. This kind of bug stays hidden until the worst moment.
Dual write can be made safe, but it requires strict idempotency, deterministic conflict resolution, and serious operational tooling. During a live migration, that complexity is a tax you pay when you can least afford it.
Dual Read Keeps Writes Stable and Makes Reads Reversible
Dual read flips the risk profile. Instead of complicating writes, you keep the write path stable and make the read path flexible.
A dual read service layer can read from the old store, the new store, or both, based on a routing policy you control. One capability unlocks three properties that make migrations survivable.
Progressive cutover. You can route reads by tenant, region, or resource type, aligned with team ownership boundaries.
Fast rollback. If something looks wrong, rollback is a routing change. It takes minutes, not hours. No data recovery process.
Measurable parity. Shadow reads let you compare old and new stores in real time and quantify drift before it becomes an outage.
Add circuit breakers, timeouts, and clear telemetry, and the migration stops being a gamble. You can move traffic deliberately and reverse quickly when signals degrade.
The Sync Engine That Actually Holds Up
Dual read doesn’t eliminate synchronization. It makes synchronization safer to operate. You still need an engine that continuously reconciles the old store with the new store, and it must survive production reality. Four properties matter in practice.
Bounded work Synchronization must not starve the rest of the platform. In practice, that means explicit limits on batch size, concurrency, and backfill rate. If your sync job can spike database load or saturate a queue, it will eventually collide with incident response.
Resumable execution. Partial failures will happen, and the system needs to resume from a known good checkpoint. You want progress markers that survive restarts, plus a clear definition of what is safe to replay. A sync engine that restarts from scratch under load isn’t resilient; it’s a recurring outage generator.
Idempotent operations. Whether an operation replays once or twenty times, it must land in the same end state. This is what makes retries safe. It’s also what keeps you from creating duplicates, resurrecting deleted objects, or gradually corrupting referential integrity through repeated apply.
Deterministic conflict resolution. If conflicts resolve differently depending on timing or order, you don’t converge. You oscillate. Determinism means two operators can look at the same conflict and reach the same decision every time, and the system applies that decision consistently.
At scale, synchronization typically runs in two phases.
Bulk backfill to close the initial gap
Steady state delta sync to keep pace with ongoing writes
Safety matters more than speed. The critical question is whether the system can replay operations safely and return to a consistent state after failures, retries, and delivery that arrives out of order.
Conflict Resolution is Not an Edge Case
Live migrations are messy. Concurrent writes, normalization differences, backfill overlap, and partial retries are normal. If you treat conflicts as rare, you end up with silent divergence, and you won’t find out until it hurts.
Strong architectures treat conflict resolution as a core design concern, with explicit rules established before production cutover.
A practical approach includes phase-aware semantics.
Before cutover, the old store is authoritative.
After cutover, the new store is authoritative.
It also requires strict handling for tenancy critical attributes and cautious merge rules for routing, authorization, and escalation-related data.
Operationally, four habits make this work.
Define resolution policies upfront and document what wins.
Automate conflict detection and alerting so humans review only genuinely ambiguous cases.
Keep a conflict log and review it regularly. Patterns usually point to systematic conversion issues.
Test conflict scenarios in staging with realistic concurrency.
Done well, this prevents more than data loss. It builds confidence that both stores tell the same operational story.
What Parity Actually Means in a Control Plane
A simple match score isn’t useful unless you define what matters. In a control plane, parity must focus on differences that change behavior. Early in a migration, you can often tolerate cosmetic differences. You can’t tolerate differences that change paging, escalation, or access control.
In practice, parity checks are gates. You start with looser gates while you’re learning the shape of drift, then tighten them as you expand traffic. If a difference can change who gets paged, treat it like a stop sign, not a note for later.
Parity checks worth investing in are tied directly to operational outcomes.
Routing targets, alert thresholds, and enabled or disabled states
Ownership and escalation mappings
Tenancy boundaries and authorization behavior
Referential integrity, including missing parent objects and broken links
Deletion semantics, including tombstones and retention rules
Early in a migration, minor cosmetic differences can be acceptable. Differences that change who gets paged, or whether anyone gets paged, are not acceptable.
A Cutover Sequence That Keeps Migrations Boring
The best migrations are uneventful. They’re a series of small decisions backed by observable signals, not a dramatic switch flip.
Shadow parity phase: The old store remains primary. Sample reads from the new store, canonicalize results, compare behavior, and record drift. Users are unaffected.
Fallback phase: The new store becomes the fallback when the old store times out or errors. This provides a real operational stress test before the new store carries primary traffic.
Progressive primary switch: Pick a small slice, such as one tenant or a low-risk region, and move primary reads for that slice. Watch latency, error rates, and parity signals closely.
Expansion phase: Expand coverage as metrics stay stable. Tighten parity requirements as confidence increases. Each expansion should be a deliberate decision, not a calendar item.
Decommission: Retire the old store only after sustained parity and clear operational confidence.
The key mindset is simple. Cutover is not a single event. It is a sequence of reversible decisions informed by telemetry.
The Real Win: Easy Rollback Changes Team Behavior
When rollback is difficult, teams either rush and hope or delay indefinitely because the risk feels too high. Both approaches are costly.
When rollback is a routing change that takes minutes, behavior shifts. Teams enforce stricter correctness gates because uptime isn’t riding on one irreversible switch. Drift gets caught early because it’s measured continuously. Cutover aligns with operational ownership instead of being imposed on everyone at once.
Dual read cutovers don’t just make migrations safer. They make migrations repeatable and predictable. And in platform engineering, boring is the goal.
Key Takeaways
Control plane migrations fail for different reasons than data plane migrations. Drift changes behavior, not just data.
Dual write during migration often creates silent divergence through retries, ordering differences, and semantic mismatches.
Dual read keeps the write path stable, makes rollback simple, and lets you verify parity before you bet uptime on it.
Modern DevOps toolchains, often beyond full organizational control, automate and streamline the software development life cycle. They enhance collaboration, reduce context-switching and leverage observability to deliver better software faster. As applications now span complex, distributed environments — including microservices, multiple clouds, SaaS APIs and evolving AI agents — pinpointing the source of failures becomes increasingly challenging when disruptions occur.
The core challenge DevOps teams face today is no longer simply detecting whether something is broken; it is now about understanding why it broke and what to do next — and the quicker, the better. The industry shift now underway is from reactive troubleshooting to proactive, prescriptive resolution. Monitoring must evolve from signaling that a problem exists to guiding teams toward the fastest, most effective path to restore normal operations.
A New Kind of Outage
In the past, failures were often straightforward. A server crashed; a deployment introduced a faulty configuration or a database became overloaded. Engineers could trace the issue and apply a fix. However, the systems we operate today behave differently. Failures are increasingly distributed, triggered by services that live outside the organization’s infrastructure. They can be subtle, i.e., rooted not in total outages, but in small latencies that cascade under load — and as systems become more interdependent, identifying the precise source of an issue becomes far more difficult.
For example, in a workflow that depends on an AI-driven service, a request may travel through multiple remote endpoints before a response is returned. If even one of those dependencies slows down or returns unexpected results, the entire experience degrades. The user sees failure, but the root cause may be several layers removed. DevOps teams might detect an increase in error rates, latency, timeouts or customer support complaints, yet still struggle to determine where the problem lies — in their own code, their cloud provider, an external API or somewhere along the network path.
Where Traditional Monitoring Falls Short
Monitoring systems continue to play a critical role in alerting teams to performance changes. The problem is not alerting; it is interpretation. Traditional application performance monitoring (APM) excels at conveying what is happening, but not why. It can reveal that a page load slowed down, or that error rates spiked, or that customers are experiencing degraded service. What it cannot always do is indicate where the fault lies within a chain of dependencies that spans multiple services and continuously evolving systems and technologies.
The result is familiar to most DevOps teams: Lengthy triage sessions, extended war rooms and time-consuming manual investigations. Engineers comb through logs, traces, dashboards and reports, often working through elimination rather than direct diagnosis. Mean time to identify (MTTI) the issue increases, and mean time to resolve (MTTR) the issue stretches even longer. Customers feel the impact in real-time, even if the root cause lies far downstream.
AI Introduces Power (and Fragility)
The rise of AI and LLM-powered automation has introduced both breakthrough potential and new fragility. Traditional software is deterministic; it behaves as programmed. AI systems, in contrast, can change behavior as models are updated or as new data influences inference. These changes are often introduced silently by third-party providers, which means that application performance can shift without any direct change being made by the DevOps team.
This is especially evident in systems powered by agentic AI — autonomous agents that retrieve information, make decisions and interact with other systems. These agents often depend on a growing web of third-party services. When one service becomes slow or unavailable, the entire agent workflow can stall. Additionally, as AI systems often mask underlying complexity, diagnosing these failures requires visibility into dependencies that are not always obvious.
Moving From Reactive to Prescriptive Response
To meet the demands of this environment, DevOps teams need tools and practices that help them move faster from detection to resolution. This means improving context, not just alerts. It also means being able to trace issues across internal systems and external dependencies and leveraging AI not just to automate tasks, but to support decision-making during incidents.
Currently, several monitoring companies are leveraging AI, but in limited ways. Most are adding chatbots that can summarize dashboards or generate reports that look impressive in PowerPoint presentations. But that’s not solving the core problem.
The better companies are using AI to compare current behavior against historical patterns, identifying anomalies that may not be immediately visible. They use AI to correlate telemetry across multiple services to pinpoint where failures are most likely originating.
But the very best companies are taking this further with AI that eases the chaos during incidents; AI that analyzes your monitoring setup and makes intelligent recommendations about coverage gaps; AI that suggests additional tests, and even pre-configures them for you — AI that recommends next steps based on prior successful resolutions, turning incident response from a process of trial and error into one guided by evidence and experience.
This is how AI solves the fundamental problem that keeps SREs, DevOps engineers and IT ops teams awake at night: The gap between detection and resolution. The space between I see the problem and I know what to do about it.
The goal is not to eliminate human judgment — it is to amplify it. Teams still choose the strategy, validate the fix and restore normalcy — but they do so with better information, faster.
Distributed systems and AI-powered applications are not becoming simpler. Dependencies will continue to multiply and outages will continue to occur. The defining difference will be the speed and confidence with which teams can respond. In short, the question is shifting from what now to what’s next. Moreover, the teams prepared to make that shift will define the next generation of operational resilience.
In the past few years, systems have become more complex than ever. Microservices, Kubernetes, cloud environments and distributed application programming interfaces (APIs) have changed how we build and manage software. However, this complexity has also made it harder to find the root cause when things go wrong.
That’s where observability and artificial intelligence (AI) come together to change the game — helping us move from reactive monitoring to predictive root cause analysis (RCA).
From Observability to Prediction
Traditional observability is all about answering three questions:
What’s happening in the system?
Why did it happen?
How can we fix it?
Today’s tools — logs, metrics and traces — do a great job of showing what’s happening. However, when dozens of services are talking to each other, even the best dashboards can feel like looking for a needle in a haystack.
That’s why engineers spend hours or even days doing RCA — collecting logs, comparing metrics and following traces to find where the problem started.
AI is now changing this story. Instead of waiting for something to break, AI helps us predict issues before they impact users.
How AI Helps With Root Cause Analysis
AI in observability works by learning normal behavior of your system and spotting when something looks unusual. It doesn’t just alert you — it tries to understand the pattern behind the problem.
Here’s how it typically works:
Collect Data: Logs, metrics and traces are gathered from all parts of the system.
Learn Normal Behavior: Machine learning (ML) models analyze this data and understand what ‘healthy’ looks like.
Detect Anomalies: When something unusual happens, AI detects it faster than manual alerts.
Correlate Signals: The system connects related events to find what might have caused the issue.
Suggest Root Cause: AI highlights the most likely reason for the failure and may even suggest fixes.
Imagine your API latency increases. Instead of just showing red alerts, the observability platform might tell you:
‘Latency spike likely caused by slow database queries in service X — starting at 10:23 a.m.’
That’s predictive RCA in action.
Traditional RCA vs. Predictive RCA (AI-Driven)
Aspect
Traditional RCA
Predictive RCA (AI-Driven)
Approach
Reactive — investigate after failure
Proactive — detect and predict before failure
Speed
Slow — manual log and metric analysis
Fast — real-time insights using ML models
Data Handling
Human-driven correlation
Automated correlation across logs, metrics and traces
Alerting
Threshold-based, often noisy
Context-aware, reduces false alerts
Scalability
Hard to scale with microservices
Designed for large, distributed systems
Human Involvement
High — engineers must dig for clues
Low — AI surfaces probable causes directly
Outcome
Fix after impact
Prevent or auto-remediate before impact
This table makes it clear: Predictive RCA doesn’t replace humans — it empowers them to act faster with better context.
Challenges Ahead
While AI brings huge benefits, it’s not perfect. Some key challenges include:
Data Quality: Bad or missing data may lead to incorrect predictions.
False Positives: AI might flag normal behavior as issues.
Explainability: Teams still need to understand why AI made a certain prediction.
That’s why many experts believe in a ‘human + AI’ approach. AI does the heavy data crunching, while engineers use their domain knowledge to validate and act on insights.
The Road to Self-Healing Systems
Predictive RCA is just the beginning. The next stage is self-healing systems, where observability platforms not only detect and predict issues but also automatically fix them.
For example:
Restarting a failed service
Scaling up resources when demand spikes
Rolling back a faulty deployment before users notice
Soon, AI won’t just tell us what went wrong — it will tell us what to do next or even do it for us.
Final Thoughts
Observability used to be about visibility — now it’s about intelligence. As AI continues to evolve, we’re transitioning from dashboards that describe problems to systems that predict and prevent them.
The future of observability isn’t just about finding root causes faster — it’s about making sure they never happen again.
Soon, the smartest systems won’t just react to issues — they will predict, prevent and heal themselves.
For many coders, the idea of distributed systems development might seem like a dark art. They think of high-performance computing wizards crafting mysterious spells with job schedulers to work magic with hundreds of connected nodes in massively-parallel systems. Or large, distributed content delivery networks that balance resources to deliver content across the world no matter how high the demand. The folks doing those jobs have often worked in specialist distributed computing environments for years. But the complex problems they face at a massive scale aren’t so different from those that you deal with every day. In short, you’re a distributed developer too. It’s just that no one bothered to tell you.
Everything is Distributed
The world is a lot more distributed today than it used to be. Back in the day, code would talk directly to one mainframe or microcomputer, generally on the same LAN. Your PDP-11 and your batch order processing software had an intimate, exclusive kind of relationship.
Computing environments began to change with the advent of open computing in the late eighties. It ushered in the era of multiple computers working together other over a network. The concept continued to evolve beyond PCs. Now phones, servers and smart kettles all expect an easy way to talk to each other.
Today, distributed operation is so ubiquitous that people barely see it happening. When you type anything into a browser, it fires off dozens of queries to different backend systems owned by different organizations. These systems generally run on different infrastructure.
Even a small application that makes a database query is a distributed system. The application often talks to a system—or even multiple systems—halfway around the world.
Many apps these days are compositions of multiple API calls to different online services. When a developer gathers weather data from a service provider and feeds it into a cloud-based online geographical information system that decides how much to irrigate crops in Iowa tomorrow, what is that if not a distributed query?
Software is becoming more distributed over time. As software became more complex, developers have long wanted to carve it up into smaller pieces. This began in earnest with the software component movement around CORBA in the mid-to-late nineties, gravitating to XML web services in the early noughts and then to RESTful APIs. All of these involved forms of remote procedure call to increasingly granular services.
While your distributed applications might not be as complex as those facing HPC and hyperscale services, they still come with some challenges that increase as systems become more granular. What happens if a remote API takes longer than expected to deliver a result because it is overloaded, meaning that API queries to different systems arrive in a different sequence than expected? How should you react if a network or system outage or an expired access key prevents an API from responding at all?
Problems like these arise unexpectedly, meaning that you have to get more creative with code from the outset. Good developers do more than master object-relational mapping and send queries. They program defensively to ensure that their programs retry steps when a failure somewhere else delays that job or stops it altogether.
There are different design patterns to handle most common distributed programming scenarios today, even though you might not think of them as such. For example, developers have been building load-balancing code into their applications for decades, distributing requests between different servers to maintain performance.
Anyone who has tried to maintain transaction integrity when writing to multiple databases will also have dabbled in two-phase commit design, which ensures that all of the databases get updated (or that none of them do).
The more experienced you get, the more complex and esoteric these patterns become. At some point, many developers move beyond basic CRUD models in their applications to handle more complexity when reading and writing data. For example, they might read data from a website in one format but collapse multiple records into one when writing data to the website’s back-end store. That means writing code that creates different types of read and write queries. That pattern, called command and query responsibility segregation (CQRS), is also a form of distributed programming.
Then, there’s the Saga pattern, which people use to maintain application state across long-running processes comprising multiple transactions. This pattern relies on an event bus along with messaging and queuing code to handle services with unpredictable delays in response times. It ensures that the overall process transaction finishes in an acceptable state or rolls back.
As you venture into more sophisticated patterns that hold transactions together while maintaining performance, you’ll find your code becoming more complex and, therefore, application development becoming more difficult. Such is the life of a distributed developer, even if you didn’t realize that you were one until now.
Gaining this knowledge and experience as a developer is important, but once learned the hard way, you don’t have to keep doing it manually. There are technology solutions that manage some of these complex distributed patterns for you. As the volume of plumbing code to support distributed design patterns increases, so does the need to offload it. In an industry increasingly built on abstraction, it’s time to turn distributed systems management into a separate service and unburden your code.
And—if you’re looking for an example platform specifically designed to address this issue—check out Temporal, a completely open-source (MIT license) microservice orchestration platform which enables developers to build scalable applications without sacrificing productivity or reliability.
Whether it’s the office, the classroom, the store or the bank, a great many spheres of human life have moved online. Each day, we grow more and more used to consuming applications on demand that instantly connect us with the rest of the world. Our expanding appetite for the internet depends on an ongoing trend toward distributed systems, with most companies today spreading their information technology across hybrid and multi-cloud architectures that consist of small, independent and loosely coupled containers. Companies are scaling their applications by deploying more clusters of containers across an ever-broadening surface. There are four key lessons businesses can observe from the proliferation of distributed systems.
Don’t Underestimate the Benefits of Distributed Systems
There’s a reason why the trend toward distributed systems persists; they are chock full of benefits.
Distributed systems are incredibly scalable, as they allow companies to scale applications by adding, replacing or removing nodes as necessary. New applications can be rolled out quickly and in multiple iterations, accelerating time-to-market. Codebases are smaller and easier to change. Distributed systems optimize developer efficiency and team collaboration because numerous teams can work on different parts of a system simultaneously.
Distributed systems are highly resilient. When (and only when) built effectively, distributed systems will continue working even if a significant part shuts down, as traffic can be rerouted to functioning servers—just look at the resilience of the internet. Fewer single points of failure create more stability and fault tolerance.
With so many benefits, it’s not difficult to see how distributed systems have enabled our connected, digital economy. Netflix is a prime example of how they have supported the growth of SaaS-based applications. Thanks to a complicated distributed system, anyone on Earth can access a massive library of movies, tv shows, and documentaries to stream whatever they want on demand, paying a monthly subscription.
While Netflix is considered to be in the entertainment industry, we think of it as a technology company first and foremost because the success of its business model is based on the efficacy of its distributed system. Netflix’s platform is scalable, resilient, performant, efficient and therefore profitable because it is distributed.
Know That Distributed Systems are Complex and Challenging
Distributed systems are innately complex and therefore difficult to build and operate. They consist of many asynchronous components that must work in tandem with one another. As the system scales, it becomes even more challenging to understand, let alone operate.
These challenges are hardly unique to Netflix. Any company that manages a distributed system at scale suffers because of the convoluted nature of distributed systems. Other internet giants like Spotify, YouTube and Twitter also grapple with compounding complexity. If you’re overwhelmed by the labyrinth that is your distributed system, take comfort in knowing that these challenges are (to an extent) unavoidable.
Turn Networking to Your Advantage
A distributed system is a set of computers, containers or any other nodes that connect to a network to communicate with other entities by passing messages. What truly allows distributed systems to be ‘distributed’ and to function is the network. This makes the network both a liability and an opportunity. Yes, even the best systems will fail at some point on various levels. There is no such thing as complete control in distributed systems, but that doesn’t mean you can’t turn networking to your advantage.
Developers can efficiently manage their distributed system by prioritizing networking and employing the right tools across an architecture. Netflix has built exceptionally well-connected systems with servers deployed in thousands of locations worldwide that connect to the control plane in the cloud. If a server fails, traffic can easily be rerouted elsewhere. The distributed system is scalable, resilient, performant and efficient. Robust networking solutions give the team at Netflix visibility into and control over the application.
Networking truly is the key to bolstering the advantages and overcoming the challenges of distributed systems. Invest resources in your network and it will take care of your distributed system.
Aim for Simplicity
Turn networking to your advantage by taking care when building a networking stack. The two most predictable pitfalls to avoid here are not choosing a stack that addresses your unique challenges and differentiators and selecting tools that are overly complicated for your use case.
It’s all too common for companies building and managing distributed systems to adopt popular or default tools that come with their orchestration platform without giving their suitability much thought. It is never a good idea to copy-paste a networking stack seen elsewhere, assuming that what has worked for others will also work for you. It may, but do not blindly select the stacks everyone is talking about without giving them much thought.
Many organizations followed the herd by adopting Istio, only to later discover that the service mesh is far too complicated and resource-intensive for their platform. While Istio’s numerous features are helpful for some, they quickly become burdensome. The trending tools are not necessarily the best for your use case.
Some companies go the opposite route and build their own custom solutions instead of following the herd. Just as Netflix built Zookeeper for service discovery and Spotify built Backstage for API management cataloging, global companies with expansive distributed systems can benefit from the control offered by a custom tool. This option is expensive, time-consuming and not viable for non-tech or smaller companies.
You’re (probably) not Netflix, so don’t try to be. You may think you have the same problems, but you don’t. Yes, distributed systems are always complex and full of challenges. But the complexity of distributed systems compounds at scale to such an extent that it becomes incomparable. Internet giants like Netflix and Spotify require the advanced features of complicated tools or custom solutions built in-house (and they have the means to invest in these resources). If your platform is a fraction of the size of Netflix’s, your distributed system relies on a far simpler network.
It’s extremely easy to choose solutions that are overly engineered for your use case, but it’s best to opt for simpler alternatives. Start with simple solutions, gain control of your networking stack and then advance to another level of complexity later on. Gaining control of your network will allow you to tap into the benefits of distributed systems while avoiding some of the challenges. Turn networking to your advantage and make your infrastructure scalable by choosing simplicity over complexity.
Whether migrating from a monolithic architecture or building a distributed system from ground zero, there are many benefits to leverage from a microservices architecture–faster software deployment cycles, enhanced scalability, improved isolation of risk. However, distributed systems are complex and as such, accompanied by many challenges, particularly when it comes to the scalability of metrics.
The Importance of Observability in Distributed Systems
In order to realize the benefits of a distributed system, a strategy of data-driven optimization is essential. The paradox here is visibility simplifies complexity, while the complexity of visibility is dramatically increased in distributed systems.
As New Relic points out in their eBook Cloud Native Is the New Normal, traditional monitoring approaches need to be adapted to meet the demands of decentralized architectures, including:
Overcoming the loss of end-to-end monitoring capabilities by learning how to connect the dots across a complex network of environments.
Developing tactics to quickly drill down into issues across interconnected parts to reduce mean time to resolution (MTTR).
Introducing proactive reliability strategies that focus on measurement first, followed by improvement.
Finding the Right Balance Between Monitoring Nothing and Monitoring Everything
Most application engineers leverage some sort of metrics tooling such as Grafana, Prometheus or Graphite to track vitals for their system health. While monitoring is essential, there is such a thing as bloat when it comes to metrics ingestion; and this overload translates to scalability challenges that affect performance, cost and reliability.
So, how do you know what to monitor? In order to derive the most valuable insights from your system, you must first understand and identify the components, and more specifically which attributes within those components drive key metrics. Knowing this requires an intimate familiarity with how all the pieces work together in your distributed system.
Google’s Site Reliability Engineering book provides a framework for determining what to monitor in distributed systems. They break signals down into four different categories:
Latency: Understand resource requirements and identify bottlenecks.
Traffic: Monitor system response to varying levels of load; most useful when paired with other indicators.
Errors: Track frequency and nature of interruptions in request/response flow.
Saturation: Surface capacity-related issues.
Even when you’ve identified the most important metrics to track, scalability is still a challenge that must be addressed, particularly when dealing with applications that handle massive volumes of data across a complex application architecture.
M3: Uber’s Open Source Large-Scale Metrics Platform
A solution directly addressing the challenges of distributed metrics at scale is M3, an open source project developed by Uber’s engineering team.
Before using M3, the Uber team sent metrics to a Graphite stack with storage handled using the Whisper file format in a shared Carbon cluster. From there, they created Grafana dashboards to visualize the data and used Nagios for alerting based on Graphite threshold checks via source-controlled scripts. As the volume of metrics continued to expand, manual processes to scale the solution became unsustainable and repeat failures resulting in permanent data loss led them to build a more fault-tolerant, scalable system.
M3 is focused on providing a reliable storage solution for large-scale metrics over long retention periods, solving inherent single node-associated limitations with Prometheus by offering a multi-tenant store for Prometheus metrics. A key component of this solution is M3DB, the scalable storage backend for M3.
M3DB, built in Go, is a distributed time series store and reverse index with configurable out-of-order writes. By enabling the storage of real-time metrics over long periods of retention, M3DB can replace a Prometheus cluster. In developing the M3DB solution, the Uber team essentially deconstructed Prometheus into microservices to enable the scaling of each component individually. By taking this approach, M3DB is able to offer better reliability, efficiency and performance.
At a minimum, M3DB can be leveraged as remote storage for Prometheus (or used to replace any other remote storage solution). There are, however, other features the solution offers, including an external aggregation engine and a separate query engine.
The M3DB project is still under development with a growing community of contributors.
To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon NA, November 18-21 in San Diego.
