Application programming interfaces (APIs) are the backbone of modern applications. However, failures — whether from third-party outages, network issues or rate limits — can cause major disruptions. While traditional testing falls short in preparing teams for these real-world issues, something more is needed to build a truly resilient system.
Most companies often make common engineering mistakes when adopting technology, generation after generation. Jacob Caddy, senior technical product manager at Ambassador, stresses the great need for controlled failure simulations to build resilience and ensure services can handle downtime, slow responses and breaking changes. “It is about intentionally introducing faults into a system to see how it responds,” he reasons. “Finding weaknesses before they cause real problems in production.” Therefore, it is critically important for teams to move beyond reactive measures and proactively find gaps, errors and vulnerabilities through deliberate experimentation.
For many, traditional API testing focuses on whether an API call succeeds or fails. However, Matthew Schillerstrom, director of product management at Harness, a software delivery platform, points out, “APIs rarely fail cleanly — and that is the problem. Most teams assume API failures are binary — either the API is up and running or completely down. But in reality, failures are rarely that simple. They often manifest as partial failures — slow degradation, unpredictable error handling or cascading failures across multiple services.” Relying solely on pass/fail tests doesn’t account for the nuanced and often unpredictable ways APIs can fail in production.
Mudit Singh, VP of product and growth at LambdaTest, a cross-browser testing platform, agrees. “Traditional testing mainly checks if everything is working as expected under ideal or predictable conditions. But that is rarely how things play out in the real world,” he articulates. “API issues in systems downstream are the leading cause of flaky tests. For instance, consider a stable login test case that suddenly fails due to a transient 404 or 503 error from a downstream service. In such scenarios, teams might erroneously focus on identifying what changes in their own code broke the tests, only to later discover that the root cause lies outside their control, maybe with an API response from a system downstream.” This not only wastes valuable debugging time but also raises doubts about the overall robustness of the continuous integration and continuous deployment (CI/CD) pipeline as downstream API issues can lead to misleading test results, highlighting the need for testing beyond internal code.
Caddy, drawing on his technical product management experience at Ambassador — an API development company providing solutions to accelerate development, expedite testing and optimize the delivery of API resources — shares the following, “Traditional testing usually means unit and integration tests, which focus solely on functionality but do not account for infrastructure failures. Load and performance tests, on the other hand, simulate traffic but usually follow predictable patterns because they are designed by developers who already understand how the system should work. The problem? Real-world failures do not follow controlled test cases.” This reinforces the idea that conventional testing often fails to replicate the unpredictable and infrastructure-related causes of real-world API failures.
Breaking Things to Build More Resilient APIs
As the limitations of traditional testing become apparent, a more proactive and realistic approach is needed — one that builds confidence in your systems by intentionally causing controlled failures.
Chaos engineering helps do exactly that. It originated in 2010 when Netflix created Chaos Monkey, a tool designed to randomly switch off production software instances to test the resilience of their cloud services, after the company suffered a three-day outage.
Chaos engineering involves intentionally introducing faults or failures into a system to observe its response, to identify and address weaknesses before they cause real-world outages or security incidents. “By embracing controlled chaos, teams can prevent surprises, improve reliability and deliver a better user experience,” says Priyanka Tembey, co-founder & CTO at Operant AI, a runtime AI application defense platform that protects live cloud and AI applications. “This means that instead of waiting for failures to occur unexpectedly in production, teams can proactively simulate them in a controlled environment to understand how their systems behave and identify areas for improvement,” she adds.
For consolidating development efforts, chaos engineering is crucial. Caddy explains, “Chaos engineering is essential because it shifts the focus from merely testing happy-path conditions to actively preparing for failures. In the real world, issues arise frequently — servers crash, network delays happen and external services can fail.” He elaborates, “By intentionally introducing these failures during the development cycle, teams can proactively identify weaknesses in system architecture, error handling and recovery mechanisms before they impact end users.” Chaos engineering enhances resilience by guiding developers in designing systems that anticipate failures and respond gracefully, rather than dealing with issues reactively when they occur in production.
It is also important to recognize the practical limitations when it comes to system resilience at scale. Siri Varma Vegiraju, technical leader at Microsoft Azure overseeing Azure security, emphasizes an important point: “Netflix popularized chaos engineering with tools like Chaos Monkey, which deliberately introduces faults into production to test system resilience. However, every organization cannot afford a team and infrastructure like Netflix. And this is where providers like Ambassador Blackbird, with its advanced mocking capabilities, bridge the gap.” The principles of chaos engineering are powerful, but the availability of specialized tools and platforms makes it accessible to organizations of all sizes to actively probe for weaknesses and build more resilient and reliable applications.
Making Failure Testing Part of Your Workflow
Integrating API failure testing into the CI/CD pipeline is one of the best and most highly recommended engineering practices for building resilient services. This means deliberately testing for failures in isolated environments and as part of the regular development process.
For simulating real-world failure scenarios, tools play an extremely useful role. Caddy highlights how Blackbird’s Chaos Mode simulates real-world failure scenarios directly within the API testing process, “It enables teams to introduce error responses and latency into mock API endpoints, allowing them to test how the application behaves under unexpected failure conditions often seen in the real world. By facilitating controlled simulations of timeouts, incorrect payloads and other failure modes, Blackbird enhances the testing process, ensuring that systems can gracefully handle errors before they occur in production. This capability helps teams prepare for chaotic situations, aligning with the core principles of chaos engineering.”
Vegiraju points out the broader need for such capabilities, “With advanced mocking strategies, I can use Blackbird to inject failures, rate limits and latency spikes into API responses during development, testing and — if required — in the canary and production environment. And this means as a developer, I can now test how the application behaves when an external API returns 500 errors or times out. Check if the service degrades gracefully when responses slow down unexpectedly. Ensure API clients handle 429 Too Many Requests errors correctly instead of breaking under load.” The more granular control developers gain over simulating specific failure modes, the easier it becomes for targeted testing of different resilience patterns.
Beyond specific tools, API Gateways also offer built-in capabilities to streamline some of these testing efforts. Schillerstrom notes, “API Gateway platforms with built-in mocking and traffic shaping capabilities allow teams to test failure scenarios proactively before they occur in production. Teams can validate whether their APIs can handle real-world disruptions without degrading the user experience by simulating rate limits, latency spikes and service outages.” He gives examples: Before launching a new Mobile Alerts API, teams could use an API Gateway to simulate
- Rate limits by throttling API requests to see if the system handles 429 Too Many Requests responses gracefully.
- High latency by injecting artificial response delays, ensuring services function properly under slower-than-expected network conditions.
- Service unavailability by returning failure responses, allowing teams to validate retry mechanisms and fallback strategies.
This highlights the convenience of leveraging API Gateway features for simulating common failure scenarios without needing separate specialized tools.
“Using BlackBirdʼs chaos testing mode, one can generate millions of requests and simulate the spiky traffic,” Vegiraju mentioned. “Similarly, for latency testing, WireMock allows simulating artificial delays to assess API behavior under slow responses. Combined with Mockaroo, these tools help generate diverse test scenarios, ensuring the API maintains functionality and falls back to a stable state during intermittent or unexpected failures.” The specialized tools can provide comprehensive coverage for various types of API failure simulations, from high traffic to latency issues.
By integrating these simulation techniques into the CI/CD pipeline, teams can catch potential issues early and build more resilient APIs, ultimately leading to more stable and reliable applications in production.
Taking Failure Testing to the Next Level
Basic API failure testing, such as timeouts and retries, is a good start. Teams can go further to ensure true resilience. Schillerstrom shares some advanced practices using Harness Chaos Engineering: “Beyond basic API failure testing, such as timeouts, retries and mock failures, Harness Chaos Engineering enables targeted, automated API failure injection, helping teams validate resilience in ways that traditional testing cannot.”
He gives an example of injecting failures at the Kubernetes pod level: “Look for tools that can test and simulate real-world API failures in a controlled environment. Instead of relying on mock responses, teams can test actual failure behavior on live deployments, including how services handle unexpected 503 Service Unavailable responses, whether retries and fallback mechanisms work under 429 Too Many Requests scenarios and how APIs degrade when internal services fail — without needing to modify API Gateway rules.” The ability to simulate failures in live environments is invaluable in understanding true service behavior under stress, going beyond the constraints of simple mocking.
Another advanced practice involves automation. Schillerstrom explains, “One challenge in API resilience testing is knowing which APIs and services to target. Harness automates this with built-in service discovery and dependency mapping, ensuring failure tests align with system dependencies. Harness automatically identifies all running services and their dependencies, so teams do not need to define test targets manually. And teams can use this discovery data to create automated, recurring chaos tests for critical API paths. Harness also tracks test results over time, providing a resilience score showing how API behavior improves (or worsens) with each deployment.” This level of automation, particularly in identifying test targets and continuously assessing resilience, offers significant advantages by providing measurable insights into system health over time.
Vegiraju also offers key best practices: “First, begin simulating small failures in non-production environments. Gradually, move to canary and production environments as confidence in the system’s resilience grows. Understand different API failure points like network failures, latencies and dependencies unavailable, and produce an initial set of tests. Use generative AI to build additional test suites. Use tools like Blackbird and Chaos Monkey to run these tests and create hypotheses — such as failure impacts and recovery times. Finally, as you start fixing things, run the tests periodically to make sure you are progressing and making the infrastructure resilient.” This is a good starting point for teams embarking on API failure testing, emphasizing a phased approach, a focus on understanding potential failure points and the importance of continuous validation.
These advanced techniques and best practices help teams move beyond basic testing and truly prepare for the unpredictable nature of real-world API failures, leading to more reliable and robust applications.
A Shift in Mindset: Embracing Failure for Improvement
Adopting chaos engineering is not just about tools and processes. It can also lead to a significant shift in how development teams view failure. Instead of fearing outages, teams start to see failures as opportunities to learn and improve.
Caddy has seen this impact firsthand: “Chaos engineering has created a culture that views failure as an opportunity for improvement rather than something to fear. It has shifted our mindset from reactive to proactive. We no longer wait for failures in production; instead, we actively test and learn from potential failure points. This mindset is essential for building more reliable systems, as teams now recognize that while failure is inevitable, how the system responds is what matters most.” At Ambassador, he shares, “Chaos engineering has also fostered collaboration across teams, requiring a comprehensive approach to designing resilient systems. By embracing the concept of ‘failing fast and learning from it’, our team has become more agile and confident in our ability to deliver dependable software.” It promotes a forward-thinking approach to system reliability, where learning from simulated incidents and team collaboration becomes central to building resilient systems.
“We at LambdaTest learned this the hard way. We understand the challenges development and testing teams face in ensuring API reliability, and the significant investment it took to build our scalable infrastructure,” Mudit shares. “But this proactive approach, where teams intentionally seek out weaknesses, fosters a culture of continuous improvement. It encourages collaboration and shared responsibility for system resilience. Instead of reacting to incidents, teams are better prepared to handle them, leading to increased confidence and faster recovery times.” Intentionally seeking out vulnerabilities not only strengthens the system’s defenses but also cultivates a mindset of continuous improvement and shared accountability within the team.
This shift in mindset, from fearing failure to embracing it as a learning opportunity, is a key benefit of adopting chaos engineering practices. It empowers teams to build more robust and dependable systems.
Building a Future of Reliable APIs
Modern applications rely heavily on APIs. Hence, ensuring these APIs are resilient is critical for a good user experience. By intentionally injecting failures into testing workflows, teams can proactively identify weaknesses and build more robust systems.
Comprehensive tools that offer end-to-end capabilities — such as Blackbird or a combination of specialized tools like API Gateways (Kong, Tyk) and mocking tools (WireMock, Mockaroo, Blackbird) — provide the capabilities to simulate a wide range of failure scenarios within the CI/CD pipeline. This allows teams to thoroughly validate their resilience mechanisms and ensure their APIs can withstand unexpected disruptions.
Caddy sums it up well: “API resiliency is important for better customer experience, and the tooling is the easiest way to enhance it.”
Ultimately, embracing controlled chaos is not about causing problems — it is about preventing them. By proactively testing for failures and fostering a culture that values resilience, teams can deliver more reliable services and a better experience for their users.
