devopsy.kinsta.cloud

Tag: resilience engineering

Observability, SRE and Uptime in Telehealth Platforms: A DevOps Playbook
Virtual care went from nice to have to must have during the COVID-19 pandemic and while in-person visits are starting to pick up again, telemedicine is here to stay. Its growth will continue: health-tech companies are predicting the telemedicine market will be $143.49 billion by 2025 (it will be $167.74 billion in 2025 and $584.99 billion by 2033 at 16.9% growth rate).

Telehealth platforms handle sensitive health information, manage appointments, stream live video, link to patient EMRs and of course must be fully HIPAA compliant. Anything that goes wrong and we mean anything can harm patient care, erode trust and bring on a whole bunch of regulatory headaches. For DevOps teams who build and run these platforms, having a system that’s rock solid and always available is no longer a nice to have; it’s a patient safety feature.

This guide is all about where observability, SRE and uptime management come together in telehealth. Based on research, real case studies and hard-won experience of actually managing DevOps teams in healthcare, it outlines a bunch of practical steps you can take to get your production platform ready for production while meeting all the compliance and user experience requirements.

Why Telehealth Reliability Matters

Telehealth users expect the same uptime as traditional clinical systems. According to a 2024 survey of healthcare organizations, 93% of patients expect digital health services to be available 24/7 and downtime incidents cost an average of $7,900 per minute. Hospitals and clinics operate on thin margins and outages have a ripple effect: staff have to go back to paper processes, appointment backlogs grow and patient safety declines. The 2024 CrowdStrike-related Microsoft outage cost the healthcare industry about $1.94 billion with individual organizations losing about $64.6 million.

Regulators are paying attention. North Memorial Health, for example, paid $1.55 million after a HIPAA violation related to inadequate safeguards and lack of business-associate agreements. Downtime during compliance incidents damages not only finances but also patient trust. As Mitesh Rao, former chief patient safety officer at Stanford Health Care, said, telehealth outages “affect every aspect of patient care”. DevOps teams need to engineer telehealth platforms that deliver continuous availability while meeting security and regulatory requirements.

Beyond Monitoring: Building Observability

What Is Observability?

Monitoring checks if specific metrics are within thresholds. Observability goes further; it provides a holistic view of system health by correlating metrics, logs, traces and events. Bri Morgan of Splunk says healthcare observability is “the path to achieving resiliency across mission-critical services”. Observability tools let IT teams see into the system’s internal state based on its external behavior and gather insights into clinical and operational workflows. John Wilson of SolarWinds adds that observability “helps identify security threats and potential breaches” and helps with HIPAA compliance.

Observability is different from monitoring in its predictive and diagnostic capabilities. It includes distributed tracing, a forensic technique that visualizes the flow of requests across microservices to find bottlenecks and anomalies. Predictive analytics can forecast potential issues so organizations can respond proactively. Patrick Lin, general manager of observability at Splunk, says observability integrates with DevSecOps so teams can catch security issues before they hit production without slowing down the development cycle.

Telehealth‑Specific Observability

The unique challenges that come with telehealth platforms. Video streams, asynchronous messaging & all the rest – remote monitoring devices, EHRs, billing systems – observability needs to cover everything from network to application, & that includes video performance & device health too. Take Codebridge’s advice for a moment – setting service level objectives (SLOs) right from the get-go is a good place to start: 99.95% uptime, video join times of 5 seconds or less, & call drop rates under 2%. Those SLOs translate patient expectations into real targets – something to aim for.

Observability isn’t just about catching errors, though that’s part of it. You need to see the whole picture: all those user journeys & interactions that are happening. For example, if patients are bailing out on authentication all the time, metrics are just the first place to look. You need to dig deeper to find out if it’s the identity provider, network, or the timeouts that are the problem. You should be able to see interactions with all those peripheral devices – digital stethoscopes & blood pressure monitors, that sort of thing. And remote site observability means your telehealth services are running smoothly, patient data & device health are getting checked, & you’re keeping an eye on medical equipment too. DevOps teams want to be able to get that device data, so they can pick up on data quality issues before it’s too late for clinicians.

As the world moves on, more & more of these observability platforms are including AIOps – AI-driven analytics that can spot anomalies & say “here’s how to fix it”. That AI observability stuff can collect & analyze all the data – events, metrics, logs & traces, to name a few – & spot system issues before they become problems, keep downtime to a minimum & make sure services like EHRs, telehealth & imaging are always available. Those AI models can even spot the really subtle stuff, like a slow memory leak or the odd bit of network congestion that you might otherwise miss.

Implementing Observability:
- Instrument all your bits and bobs – use open standards like OpenTelemetry to add monitoring to your code, infrastructure, and networks. Get a handle on metrics (latency, throughput, error rates) and logs (the lowdown on what’s happening, in context) – and don’t forget about those cross-service workflows, captured via distributed tracing. For telehealth, pay attention to video streaming (looking at WebRTC metrics), call quality (packet loss and jitter can be a real problem), and device connectivity.
- Gather all your data in one place and get some visibility across the board. Use a unified observability platform to collect telemetry data and spot problems as they appear, across different parts of the system. And make sure distributed tracing ties user actions back to the processes that are actually working on them – that way you can stop teams from bouncing between different dashboards to try and figure things out.
- Set up your alerts and anomaly detection to save you time and stress. Use machine learning to spot when things are behaving strangely, and get real-time alerts when there’s an issue. And make it easy to fix things fast with some run-book automation to help sort out problems as soon as they come up.
- Put data privacy at the top of your list – observability has to comply with HIPAA and GDPR. Make sure any sensitive data stays encrypted while it’s in transit and when it’s stored, limit who can access it with role-based controls, and draw up business associate contracts. And don’t forget that HIPAA needs end-to-end encryption, audit trails, and frameworks like NIST and zero-trust architecture. Your data pipelines have to fit the bill, too.
Site Reliability Engineering: Turning Metrics into Reliability

SRE is the application of software engineering to operations. Google made it popular by treating reliability as a product feature and quantifying it through service level indicators (SLIs), SLOs and error budgets. Healthcare can benefit a lot from SRE.

A 2024 research paper says Google’s SRE team maintains 99.99% uptime for critical services and the Cleveland Clinic reduced critical incidents by 40% and MTTR by 60% after adopting SRE. A HIMSS survey found 70% of healthcare IT professionals believe SRE practices improve system reliability and performance.

SRE Practices for Telehealth

Design for graceful degradation, so if the video doesn’t work, for example, the system can fall back to audio or chat. Provide asynchronous options like secure messaging or offline notes so care can continue even when live sessions drop.

Do chaos engineering: Simulate node failures, network partitioning and API timeouts to see how the system behaves. In telehealth scenarios, testing should include streaming services, device connectivity and EHR integration. Use the results to enhance failover strategies and client-side resiliency.

Track patient-centric SLIs: CPU and memory metrics are not enough. Monitor metrics like join time, call drop rate, video quality (jitter, packet loss), authentication latency, prescription fulfillment success, and device connectivity. These SLIs are patient-facing. When you put multiple microservices together, use distributed tracing to correlate patient-level transactions across authentication, scheduling, video streaming, EHR integration and billing.

Automate remediation: Combine observability data with automated runbooks to reduce MTTD and MTTR. According to IRJET, advanced monitoring allowed one hospital to detect and resolve 80% of incidents before they impacted end users; 75% of organizations using advanced monitoring reported improved availability and reduced downtime.

Prioritize patient data integrity. There’s no use being reliable if the data is wrong. SRE practices should include thorough validation and error handling. The Cleveland Clinic reduced errors in data entry by 80% and increased the accuracy of records to 95% by implementing full data validation.

DevOps Playbook for Telehealth

Now that we have the theory, let’s turn it into a playbook for DevOps teams to build telehealth platforms.

1. Know the Domain

Telehealth is regulated and patient-centric. Know HIPAA, HITECH and regional regulations. Map data flows and ensure PHI is secure. During planning, involve compliance specialists and clinicians to understand functional requirements (e.g. e-prescriptions and remote monitoring) and non-functional requirements (e.g. uptime, latency and security). As an example of how domain expertise drives success, ScienceSoft telemedicine solutions enable audiovisual patient-doctor communication, patient monitoring and remote care delivery while reducing healthcare costs and increasing care accessibility. This shows that telehealth software must balance technical complexity with clinical usability.

2. Define Reliability Objectives and Architecture

Set SLOs for each user journey: session setup time, video quality, messaging latency, device connectivity and data synchronization. Document error budgets and attach to business objectives. Build architecture diagrams showing redundancy, failover paths, encryption layers and integration points. Decide to use commercial video SDKs or build a custom WebRTC infrastructure. As an example, Codebridge recommends 99.95% uptime, with call drop rates less than 2%; adjust to your clinical use cases.

3. Instrument and Automate

Instrument from day one: OpenTelemetry in microservices; quality metrics in video streaming components and log for EHR integration. Infrastructure as Code (Terraform, Ansible) and CI/CD pipelines for consistent environments and repeatable deployments. SAST and DAST in your pipeline, performance tests. Canary deployments and feature flags to control the blast radius of new features. Strong CI/CD pipeline reduces manual errors: Northwell Health’s 95% error reduction is proof.

4. Operate with SRE Principles

Create on-call rotations and incident response playbooks. Use run-books to automate standard remediation tasks – such as scale pods and restart services. Run game days and chaos experiments to validate failover. Track incident metrics: MTTD, MTTR; have blameless postmortems. SRE is not about heroics; it’s about creating systems that self-heal and freeing engineers to focus on automation instead of firefighting.

5. Continuously Improve Based on Observability Data

Observability should be the driving force that guides where your development and ops teams focus their efforts. Take a close look at those system traces to figure out where you’re really getting bottlenecked – then use that info to tweak code, snag those pesky database queries & network routing issues. And let’s not forget using predictive analytics to stay one step ahead, so you can spot impending hardware doom before it becomes a disaster.

Throw those observability dashboards up on the screen for your clinical stakeholders to see, so they can get a handle on how the platform is doing and maybe even make a strong case for some much-needed investments. In the words of LogicMonitor, you can proactively sniff out issues and keep EHR, telehealth & imaging systems always accessible when you use a hybrid observability platform powered by AI.

6. Prioritize Compliance and Trust

Reliability and compliance go hand in hand – so don’t forget it. Encrypt that data, set up proper access controls and keep track of all those audit trails. Plus, make sure you’re in compliance with the major security frameworks like NIST CSF & zero-trust network segmentation – that’s a given. Then toss in some regular risk assessments & penetration tests for good measure. Document all those data flows, have all the necessary business associate agreements in place with your vendors and make darn tootin’ sure the customer feels safe knowing their data is protected and their care won’t be put in jeopardy from some avoidable security screw-up.

Conclusion

Observability and SRE are key to uptime and reliability in telehealth platforms. By doing proactive monitoring, automated response and DevOps, teams can make every virtual visit seamless, secure and always available when patients need it most.
December 18, 2025
Designing for Failure: 4 Resilience Practices That Make Outages Boring
Last winter, my city Richmond VA suffered water distribution outages for days after a blizzard. Not because of one big failure, but because backup pumps failed, sensors misread, alerts got buried, and then another pump died during recovery. The whole city ended up under a boil‑water advisory. Sound familiar? Replace “water pumps” with “microservices” and you’ve got every cascading outage I’ve debugged in over 15 years.

The timeline mapped perfectly to Dr. Richard Cook’s observations on complex systems: failures are multi‑factor, systems constantly run in degraded mode (not everything is perfect all the time), and single‑cause incident findings often mislead. Modern outages aren’t “one bad deploy.” They’re three or four small issues that line up at exactly the wrong moment. It could be timing or load and a dependency wobble, an alert nobody saw because they were fixing something else.

Here’s what I’ve learned: if you design systems expecting a single root cause to explain everything, you’ll ship brittle architectures and have the same post‑incident arguments every time a major incident happens. The lesson isn’t “never fail”. It’s to shape your systems so small failures can’t cascade, and to practice recovery when things are calm, not during the next crisis.

The Wildfire Prevention Model

Once I watched one bad regex take down three services in a cascade and I wondered if we had circuit breakers in place. Now I think about resilience like wildfire management. You don’t build a fortress, instead you cut firebreaks, clear underbrush, and run controlled burns so when sparks fly (and they will), they hit dirt instead of any other plantations.

In production, firebreaks are
- SLO gates (service level objectives—your promise to users; if you blow the budget, stop the rollout)
- Concurrency caps (don’t let one thing use all the threads/connections)
- Circuit breakers (fail fast if an upstream is sick, rather than queuing forever)
- Per‑tenant bulkheads (separate pools so one noisy neighbor can’t sink the ship)
In addition to that, prescribed burns are like short game days that can test and surface dangerous assumptions before an actual system event. We’re not preventing the incident, but are preparing that the incident is a non-event when that incident truly strikes.

Below are four practices that consistently reduce blast radius and speed up learning.

1) Design for Degraded Mode From Day One

When Richmond’s water system wobbled, they kept pressure flowing for fire safety while fixing the root cause. That’s the model: define “good enough for now” before you need it.

For every critical user journey, write down your minimum viable service. What can you shed (nice‑to‑haves)? What can you delay (batch jobs)? What can you simplify (a lighter model or cached view)? Make degraded mode visible in both dashboards and the product (users handle “we’re slower than usual” better than a mysterious timeout or a site down).

Netflix is a great example, and has written publicly about designing for degraded modes: if the personalized recommendations system is down or slow, they fall back to simpler options (e.g., showing popular titles instead of personalized picks) so the page still loads and streaming continues. That’s “good enough for now” in action—protect the core journey while a dependency recovers.

Try this week: Write a one‑page Degraded Operation Plan for one critical user journey; add the toggles and the dashboard banner and rehearse it for 20 minutes.

2) Keep Problems Contained

Big outages happen when small problems line up perfectly. Your job is to stop the lineup from becoming a cascade. Think of it like Richmond’s water valves—when one section failed, they isolated it instead of draining the whole city.
- Bulkhead isolation: separate thread pools/queues by tenant and dependency.
- Backpressure: slow callers down so retry requests don’t make it worse.
- Circuit breakers: trip early on toxic inputs; fail fast and recover quickly.
The aim isn’t zero faults; it’s faults that stay local. That’s why valves and overflow paths helped Richmond keep water moving while crews fixed the fault, instead of letting one failure drain the whole city.

Netflix’s Hystrix provides the capability to isolate dependencies with per‑service thread pools and circuit breakers. When a backend slows or fails, only that pool is saturated and other features stay healthy, and the fallback responses keep the upstream usable. That’s bulkheads + backpressure in practice.

Try this week: Add per‑dependency concurrency caps to one critical service. Start at ~80% of what you think you need. Publish the limits so other teams plan around reality, not hope. Watch how many near‑misses you catch in week one.

3) Ship in Thin Slices With Automatic Rollbacks

I have observed this several times over the years, a team is making a “trivial” caching change and that results in a 2 hour outage. While trivial the change had the potential to cause a large outage, and that’s why I’m religious about progressive delivery. No more cliff‑edge releases, just small, observable experiments.
- Canary to 1-5% of traffic.
- Watch the SLOs (error rate, latency, business outcomes).
- Promote only if the numbers hold.
- Auto‑rollback when error budget thresholds are crossed.
- Treat infrastructure toggles like feature flags (yes, that cache TTL needs a guard).
Google Chrome ships changes as staged rollouts with guardrails and kill-switches. If crash or performance rates cross a threshold in the first few percent, promotion is halted and the release is flipped off. Most users never notice; the canary metrics caught it and impact stays tiny.

Try this week: Add a promotion guard/check that compares canary SLOs to a budget threshold and flips a rollback flag automatically and rollback the release.

4) Learn From Near‑Misses, Not Just Disasters

Most teams only do postmortems after user impact. That’s like studying only plane crashes but ignoring when pilots report weird engine noises. Real learning happens in near‑misses, those times your error budget threshold was hit for 12 minutes then self‑heals.

On a cadence, review your near‑misses. Keep it lightweight: what triggered it, what was the first signal, where did people get confused, and what’s the smallest change that prevents a repeat? No blame, no heroes, just learning. The City of Richmond’s after‑action review showed most of the pain was preventable, this is exactly what near‑miss loops catch while the sun’s shining.

Google’s SRE teams document “non‑outage postmortems” (near‑misses) when SLOs burn briefly and self‑heal. Capturing the trigger, first signal, and a smallest fix (often retry/backoff tuning or better dashboard correlation) turns almost‑incidents into permanent improvements, this makes the next hiccup invisible to users.

Try this week: Create a Near‑Miss template: 300 words max, one diagram, one smallest fix. Review them during the next operational excellence stand‑up. Make it lighter than a postmortem so people actually file them

Make Observability Actually Useful

None of this works if your logs, metrics, and traces don’t line up. Emit small, machine‑readable events at every key decision: build started, deploy finished, feature flag toggled, circuit breaker tripped, degraded mode activated. Use the same correlation ID everywhere.

One screen should show SLOs next to deploy events next to business metrics. When you roll back, log it as an event you can query later. This turns war rooms into timelines:

“deploy #4821 finished 3:14:22 → error rate up 3:16:45 → rollback 3:18:00.”

Anti‑Patterns That Kill Resilience (and What To Do Instead)
- If your “safe” path is slower than cowboy mode, people route around it. Make the paved road the fastest road by bundling CI, SLOs, flags, and rollback into one template.
- If checks always pass or always fail, they become noise. Tier controls by risk.
- If heroics become the plan, resilience turns into a sport. Ship safety in defaults, not documentation, nobody reads the wiki at 3 a.m.
How You’ll Know It’s Working

Within one quarter, you should measure:
- Fewer customer‑impacting incidents (count them, not the drama)
- Faster recovery when things do break (actual MTTR)
- Calmer on‑call rotations (ask the humans)
- Boring postmortems (“the circuit breaker caught it”)
Expect steadier user experience under stress, fewer rollbacks, faster restoration and calmer on‑calls. Not because incidents vanish, but because faults combine less catastrophically and recovery paths are paved.

If you don’t see improvement within a quarter, revisit your degraded plans, isolation limits, promotion guards, and visibility. Are the firebreaks in the right places? Can everyone see them, and export them? Did anyone run the game day, or is it still on the backlog?

Start This Week

Pick one critical journey. Publish its Degraded Operation Plan. Add a visible degraded‑mode indicator. Schedule a 20‑minute controlled burn next sprint—on purpose—before reality does it for you.
December 10, 2025
The Breakneck Future of Codegen: Why AI SWE Must Be Matched with AI SRE
AI-driven code generation is transforming how software is built — accelerating development and expanding what engineers can create. But as speed and scale increase, so do complexity and fragility. Systems grow denser, and small configuration errors can cascade into major outages. Early research links AI-generated code to higher duplication and vulnerability rates, underscoring the need for new approaches to reliability.

Engineers are already maintaining systems they don’t fully understand — and AI is only amplifying this trend. AI site reliability engineers (AI SREs) will be needed to reason across massive observability data and use parallel search architectures that can test causal hypotheses simultaneously rather than iteratively. This shift toward distributed, concurrent reasoning will be essential to keep modern software resilient as AI reshapes how it’s built.

The Promise of AI Codegen

AI code generation marks one of the most significant shifts in modern engineering — unlocking speed, creativity, and access in ways the industry hasn’t seen before. Engineers can scaffold microservices, generate Kubernetes manifests, and deploy applications in minutes — tasks that once took days or weeks.

With models like GitHub Copilot and Claude producing production-ready code, teams are shipping more features, exploring new areas of the stack, and compressing time-to-market cycles dramatically.

The Double-Edged Nature of Velocity

As development accelerates, complexity multiplies and human context erodes. Each generated service adds configuration files, dependencies, and telemetry — compounding system density and interdependence.

Engineers are also expected to ship faster than ever, leaving less time to understand what they’re deploying. AI codegen blurs traditional boundaries: software developers modify infrastructure, and infrastructure engineers tweak application logic. This flexibility increases throughput but introduces comprehension debt — neither side holds full context across the stack. Over time, systems become opaque. When incidents occur, the question is no longer just “what broke?” but “why was it built this way?”

Keeping Systems Online was Hard Enough Already

The 2024 CrowdStrike outage showed how fragile large-scale systems can be. A single misconfigured update cascaded globally within minutes, grounding airlines and halting commerce. That same pattern has played out before — like in 2021, when a BGP misconfiguration at Meta removed Facebook, WhatsApp, and Instagram from the internet for six hours.

Those incidents occurred before AI began generating production infrastructure at scale. As AI-driven automation expands, similar failures could become more frequent and harder to trace. A single misgenerated network policy, Helm chart, or service mesh rule can ripple across dozens of services. These configurations often appear correct but hide subtle flaws — mis-specified resource limits, deprecated API versions, or over-permissive roles.

Meanwhile, observability systems are ingesting unprecedented volumes of logs, metrics, and traces, stretching even the most seasoned SRE teams to their limits.

Data Points to Growing Fragility

Research already shows that the tradeoff between speed and reliability is emerging. For example:
- GitClear (2024): Duplicate code blocks increased fourfold as AI adoption rose (with some analyses reporting higher), while refactored or reused code declined.
- IEEE-ISTAS (Shukla et al., 2025): Reported a 38% rise in critical vulnerabilities after just five rounds of AI-assisted refinement.
- Veracode (2025): Nearly half of AI-generated samples failed basic security checks in pre-production testing.
The pattern is clear: AI codegen accelerates development but also magnifies technical debt. Code may compile and deploy — but its long-term reliability, observability, and resilience are far from guaranteed.

When Human Context Isn’t Enough

During incidents, engineers often face failures in code they didn’t write and infrastructure they didn’t configure. The traditional SRE model assumes deep system understanding — an assumption that breaks when AI-generated components interact in unpredictable ways. Manual debugging, driven by iterative queries and dashboards, cannot keep up with modern failure modes.

The challenge isn’t that today’s site reliability engineers (SREs) aren’t doing their jobs — it’s that the job itself is about to outgrow what any human team can manage unaided.

The Rise of the AI SRE

To fully realize the promise of AI codegen, reliability must evolve in parallel. The next step is the AI SRE — systems capable of reasoning across massive data volumes to restore lost context.

An AI SRE can parse logs, metrics, traces, and deployment histories to reconstruct system behavior in real time. It can correlate anomalies, identify causal chains, and surface hypotheses in minutes that would take human teams hours to form.

If AI is going to write and deploy code, it must also help monitor and stabilize it. The rise of the AI software engineer demands an equally capable counterpart on the reliability side — one that can distinguish correlation from causation and manage complexity at enterprise scale.

Why Parallelism Matters

Not all AI SREs reason the same way. Some work iteratively — forming one hypothesis at a time and refining it as new data arrives. That approach can handle localized issues but breaks down in large, interdependent systems.

Parallel search scales better. It allows multiple investigative threads — or agents — to reason concurrently, each exploring a different subsystem or hypothesis before merging results into a shared context. In practice, this looks like distributed inference: several reasoning paths run in parallel, exchanging intermediate signals to converge on the most plausible causal chain.

By reasoning in parallel, AI SREs can handle the scale and heterogeneity of modern production systems while engineers stay focused on prevention and long-term resilience.

The Road Ahead

AI codegen is accelerating software development at an unprecedented pace — but speed without reliability breeds fragility. To sustain this momentum, innovation in AI for reliability must advance alongside AI for creation. AI may be building the future of software, but it will take equally intelligent systems — working alongside today’s engineers — to keep that future reliable.

KubeCon + CloudNativeCon North America 2025 is taking place in Atlanta, Georgia, from November 10 to 13. Register now.
October 23, 2025
What Is Resilience Engineering?
Admitting things will go wrong isn’t easy for anyone or any team. But modern engineering practices have moved beyond this fear, giving birth to a new practice in DevOps and site reliability engineering (SRE) known as resilience engineering.

Resilience engineering today isn’t thought of as a function. However, just as DevOps was a description of culture before it was a role and site reliability was an extension of operations before it was a focus, I wouldn’t be surprised if resilience engineering became a function in the new future. The first question most will ask however is, “Isn’t this just SRE?” The purpose of the term is to change the focus from simply reacting to incidents to developing long-term response strategies for them.

Because the expectation in these environments is that things will break, resilience is the responsibility of existing DevOps and cloud operations teams. When applications and services do break, a “fly by the seat of your pants” response strategy will not work.

A Focus on Frameworks

Resilience engineering, while rooted in engineering practices, is largely focused on building strategies and a framework for their execution. This leaves the process of building resilience into a largely unestablished system in part because each system is unique. And, how you respond to issues in that system will likely be unique, even if the management plane that reports issues is not.

The job of resilience engineering is to:

Establish procedures, habits and decision trees

When things break, fight is the only option. Operators and on-call engineers need to address issues in a systematic and repeatable way and do their best to remove emotion and fear from the equation. This not only helps triage and resolve issues, but it also makes sure the activity associated with the issue leads to meaningful insights in post mortems and future collaboration. Part of that is establishing habits and decision-making processes for those who are on-call. The processes help prioritize what to focus on, help catch details because details are critical.

Be data-driven

Resilience engineering must rely on data. This is another place where traditional SRE practices grow with a focus on resilience. In typical SRE environments, the focus is on the now, using real-time dashboards of the current state. When you think about resilience you are thinking about the past’s impact on the future, not the now. The only way to do that is to make sure the data supports it; thus, part of resilience engineering is making sure the data is there. Instead of data silos that most organizations have across their delivery chain. Resilience engineering should ensure that telemetry across the entire delivery chain is captured, correlated and shared. The reason this is critical is that what happens early in the delivery chain directly impacts incidents. That activity can be the source of answers, it can be the triggers for rollback, or it can be the clarity needed to prevent similar issues in the future. Without continuity between each stage of the delivery chain, it’s easy to miss correlated events that can lead to more systemic problems.

Engineering out of reproducible incidents

For most, the best part of resilience engineering is taking what is learned from previous incidents and finding ways to automate future resolution. Learning from data and having consistency in habit leads to the ability to create runbooks and automate remediation for known issues. Often incident response audit trails can read like playbooks for addressing issues of a particular type. When the resolution is not directly related to code and the potential of issues to surface again in the future is inevitable, being able to build intelligence to address it saves waking someone up at midnight and much shorter impact on customers.

The Resilience Stack

Organizations looking to embrace resilience engineering need to have a toolkit built for it. It’s pretty straightforward but worth highlighting. The resilience stack will include:
1. Observability and/or monitoring tool.
2. Incident response tool.
3. On-call strategy documentation.
4. Post mortem process and documentation.
5. Documented processes including recommend response steps.
6. Path to automation.
For those with a relatively mature and automated environment, the next step is chaos engineering—embracing chaos as a way to get ahead of incidents before they happen in the wild.

Get your markdown skills to maximum level. There is a lot of documenting that needs to happen with comprehensive resilience engineering. But these documents should not be shelf-ware—they should be living and ultimately lead to the implementation of automation or feedback to development.

As for monitoring, observability and incident response tools, it’s not enough to simply implement them. Knowing how data will be collected, consumed and actualized is also necessary. So is establishing an on-call strategy with purpose, not just because having everyone on-call is the “cool thing to do.”

It may seem obvious to say that organizations want system resiliency. What is not obvious is how to execute it. When organizations see the gaps (and are often embarrassed by them), they understand that resilience is a focus for either current functions or new ones in the future. That is why it’s worthwhile to talk about resilience engineering and what makes it effective.
August 21, 2020