devopsy.kinsta.cloud

Tag: DevOps Testing

Why API Test Generation Is a Judgment Problem, Not a Code Generation Problem

When we started using large language models for API test generation at KushoAI, the results were impressive on the surface. Tests appeared in seconds. Coverage breadth went up. The team was excited.

Then we looked more carefully at what was being produced.

On one endpoint, the model generated 26 tests where a senior QA engineer would have written 9. On another, it completely missed the boundary cases that mattered most. On a third, it generated tests that checked whether a response arrived but said nothing meaningful about what the response contained. The tests were valid. The test suite was not useful.

What we had was a fast generator with no taste, and in testing, taste is almost everything.

This article is about what we learned while building toward something better: How we diagnosed what was actually wrong, why common fixes such as better prompting fall short, and what the architecture looks like when you treat test generation as a judgment problem rather than a code generation problem.

The Six Ways Fast Test Generation Fails

After running our pipeline across thousands of real APIs, six failure modes consistently emerged. Understanding them precisely is what pointed us toward the right solution.

Over-generation is the most visible problem. The model produces 20 to 30 test cases where 8 would be sufficient. The excess cases are often plausible but redundant, and they make suites harder to review and trust. A QA engineer looking at a 30-test suite for a single endpoint will struggle to identify what is actually being covered versus what is noise.

Under-generation is the opposite and arguably more dangerous. The model skips obvious scenarios — required fields left out, invalid enum values, boundary conditions, malformed formats, missing authentication cases and expected error responses. These omissions are hard to see precisely because they are absent.

Redundancy is subtle. Three generated tests labeled ‘invalid value’, ‘incorrect value’ and ‘unsupported value’ often map to a single check. The model treats them as distinct because the phrasing differs. A human reviewer sees immediately that they are the same assertion stated in three ways.

Inconsistency is what makes a test suite hard to maintain. Two structurally similar endpoints receive vastly different treatment on each run. One gets thorough positive, negative and boundary coverage. The other gets five tests, mostly a happy path. There is no predictable standard.

Weak assertions are perhaps the subtlest failure. A generated test checks if a response arrives and matches a broad schema shape. What it does not check is whether the transaction ID is absent when a payment is rejected, whether the error code is machine-readable rather than a human string or whether the rejected state is reflected consistently across nested fields. The test passes. The bug ships.

Cross-field relationship failures are the hardest to generate and the most expensive to miss in production. Individual fields are tested in isolation: Is the amount missing, is the currency wrong, is the payment method invalid. What gets skipped is the interaction: What happens when the amount is valid, the currency is valid, the payment method is valid and the combination creates an invalid business state. These failures do not trigger a 500 error. They return a 200 with corrupt data somewhere three levels deep in a nested response.

All six of these are failures of judgment, not knowledge. The model knows what an HTTP endpoint is. It knows what schema validation means. What it lacks is the calibration to decide what deserves to be tested, at what depth and with what assertions.

Why Prompting Helps but Plateaus

The natural first response to these problems is better prompting. We did a lot of it: Structured prompts, multistep chains, context injection, explicit output schemas, few-shot examples and detailed instructions about redundancy, coverage breadth and assertion specificity.

Prompting helped. It improved structure and reduced the most obvious redundancy. Few-shot examples produced the largest single improvement, pulling outputs closer to what a reviewer would keep.

That result was informative. If a few examples in the prompt improve behavior, the missing ingredient is broader exposure to examples of preferred QA decisions. The model already has the general capability. It needs calibration toward specific judgment patterns.

But prompting plateaus. It mostly improves field-level exhaustiveness: More missing-field tests, more wrong-type tests and more boundary tests. These are useful, but they remain independent mutations. Prompting does not reliably produce reasoning about field relationships, business states and workflow behavior. The hardest failure mode — cross-field relationship testing — is the one that prompting improves least.

This is exactly what we see in our public benchmark, APIEval-20. Schema-level bugs are easier to generate tests for. Complex bugs that only surface when multiple valid fields combine into an invalid state are systematically harder. Prompting narrows the gap but does not close it.

Separating Judgment From Mechanics

The architectural shift that actually helped was treating test generation as two distinct problems requiring different models.

The first problem is judgment: Which scenarios should exist, which edge cases matter, what should be asserted, how much is appropriate and what is redundant. This is a calibration problem. It requires exposure to many examples of reviewed QA decisions across many API types.

The second problem is mechanics: Constructing valid payloads, generating executable tests, writing framework-specific output and handling authentication context. This is a code generation problem. General-purpose frontier models already handle it well.

We fine-tune narrow models for the judgment layer. Frontier models handle the mechanics layer. The boundary between them is the most consequential design decision in the architecture.

The test intent model takes API metadata and a target field, operation or group of fields and then outputs a structured list of scenarios in plain language. It is responsible for coverage breadth, relevance, redundancy reduction and cross-field scenario selection. For a payments endpoint, it should not only produce tests for a missing amount or a wrong currency type but also identify cases where amount, currency, payment method, refund status and idempotency key interact to create invalid business states.

The assertion intent model takes an expected response schema and an observed response and outputs validation statements in plain language. The difference between a weak assertion and a useful one is specificity. A weak assertion says the response should be valid. A useful set says the status should be 400, the error response should include a machine-readable error code, no transaction ID should be issued and the rejected state should be reflected consistently in the response body.

Both models output intent, not executable code. The frontier model receives that intent and turns it into payloads and tests. This keeps the fine-tuned surface small and easy to evaluate. When the frontier model changes, the judgment layer remains undisturbed.

What Gets Fine-Tuned and Why

A reasonable question is whether retrieval-augmented generation (RAG) solves this instead of fine-tuning. The distinction matters.

RAG supplies context the model does not have — internal documentation, product behavior, customer-specific policies, historical incidents and organization-specific testing standards. It is the right tool when the system needs information.

Fine-tuning calibrates behavior the model already has but applies inconsistently: Coverage calibration, scenario prioritization, assertion quality, redundancy reduction and consistency across similar APIs. These are behavior-shaping problems, not retrieval problems. No amount of supplied context teaches a model which tests a QA engineer would keep. A mature system uses both as they are solving different problems.

What we fine-tune on is reviewed QA decisions, not raw API examples. For each example, the pipeline generates tests, a reviewer then removes redundant cases, adds missing scenarios, rewrites vague assertions, normalizes phrasing and stores the corrected output as the training target paired with its input context. The correction signals include accepted tests, rejected tests, redundant cases removed, missing edge cases added, weak assertions rewritten, flaky tests flagged after execution and cross-field cases added by a reviewer.

Each of these is an explicit judgment about coverage, prioritization or assertion quality. The dataset is not a collection of API examples. It is a collection of QA decisions.

Reviewer consistency is architecture, not preprocessing. Fine-tuning amplifies whatever patterns the data contains. If reviewers handle similar APIs differently, the model learns the inconsistency. Correction guidelines, accepted and rejected examples and review standards have to be documented and applied uniformly. The goal is a clean distribution of preferred decisions, not the largest possible pile of examples.

The Full Pipeline

The judgment and mechanics layers sit inside a seven-stage pipeline: Context ingestion; behavior modeling; coverage planning; test and assertion intent; payload and code generation; execution validation; feedback and maintenance.

To make this concrete, take a payments endpoint that accepts amount, currency, payment method, refund status and idempotency key. Context ingestion pulls in the schema, sample payloads, authentication requirements and any existing tests. Behavior modeling identifies the field relationships and the constraints between them. Coverage planning maps not just the individual field tests but the cross-field scenarios — what happens when all five fields are individually valid but combine into an invalid business state. The judgment models then define the specific test intents and assertion intents for that scenario in plain language. The frontier model takes those intents and produces the executable tests. Execution validates them against the live API. Any test that passes cleanly, fails unexpectedly or behaves inconsistently across runs feeds back into the correction workflow.

Since the layers are separate, a weakness in coverage planning can be diagnosed and improved without touching payload generation. A model provider can be swapped in the generation layer without disturbing the judgment layer. The layers that differ most across systems are coverage planning and intent, where judgment lives, and the feedback loop, where corrections accumulate. These are the layers worth investing in.

Execution Closes the Loop

Generated tests are only useful if they run, and if their results can be trusted.

Tests run against live APIs. Failures are isolated rather than discarded. The system identifies which part of a suite failed and repairs that part rather than regenerating everything. A suite that passes cleanly in one run and breaks in the next is scored as unstable rather than treated as correct. For high-risk changes, a human approval step gates deployment.

Execution feeds back into corrections. Accepted tests, rejected tests, edited tests and flaky tests all become signals that refine the judgment layer. This is what turns generated tests from a one-time output into a system that improves. The execution that validates a suite today produces the corrections that improve generation tomorrow.

Evaluation: What Usefulness Actually Measures

A generated test can be valid and still find nothing. Evaluation has to measure usefulness, not just validity.

The qualitative dimensions we track are coverage across scenario types, precision against redundant or low-value tests, calibration of test volume relative to API complexity, consistency across similar endpoints and across reruns, and assertion relevance. The execution-based metrics are bug detection rate against live implementations with known bugs, execution success rate, run-to-run variance, false-positive rate, flaky-test rate and the reviewer correction effort a suite requires.

The most informative evaluation method is blind reviewer comparison: Engineers judge outputs without knowing which system produced them. This surfaces quality differences that aggregate metrics obscure.

APIEval-20 is a public anchor for this. It is intentionally narrow: Each system receives a JSON schema and one valid sample payload, then generates tests executed against live APIs with planted functional bugs across 20 scenarios and 7 domains. The benchmark isolates a specific capability — whether generated tests can move beyond simple schema mutations and expose failures that depend on field relationships or operation semantics. That is where the separation between systems appears.

What This Points Toward

Most real failures do not live inside one isolated endpoint. They appear across state transitions, permissions, create-read-update-delete sequences, payment and refund flows, user onboarding and downstream side effects.

The clearest direction for adaptive test coverage is reasoning at the workflow level. The system already generates tests across related operations. The deeper work is reasoning about a workflow as a coherent flow, understanding how one operation’s state constrains the next, rather than testing each endpoint independently. Public benchmarks scope out these multistep and stateful scenarios today, which marks them as the next capability to build and measure.

Key Takeaways

API test generation is a calibration problem before it is a code generation problem. Models already know what APIs are. What they lack is the judgment to decide what is worth testing, at what depth and with what assertions.

The six failure modes in fast test generation (over-generation, under-generation, redundancy, inconsistency, weak assertions and cross-field relationship failures) are all judgment failures, not knowledge failures. Fixing them requires changing what the model is optimized for, not just what information it receives.

Fine-tuning and RAG solve different problems. RAG supplies information the model lacks. Fine-tuning calibrates behavior the model applies inconsistently. A mature system uses both; however, conflating them leads to the wrong architectural choices.

Separating the judgment layer from the mechanics layer makes the system easier to debug, improve and govern. Fine-tuned models decide what to test and what to assert. Frontier models construct payloads and generate executable tests. The boundary between them is the most important design decision in the architecture.

Execution feedback turns test generation from a one-time output into an improving system. Accepted tests, rejected tests, edited tests and flaky tests become the corrections that make the next generation sharper. The training signal is not API examples. It is reviewed QA decisions.

August 21, 2026
How Test Management Tools Give Engineering Teams the Visibility They Need to Ship With Confidence

Ask any engineering team whether they have test coverage, and most will say yes. Ask them whether they trust what that coverage is actually telling them before a deployment, and the conversation gets more complicated.

The gap between having tests and having visibility into what those tests mean is where most deployment anxiety lives. Teams with extensive automated test suites still pause before major releases. Teams with high coverage percentages still experience production incidents from regressions that should have been caught. The tests exist. The confidence does not always follow.

Test management tools are supposed to close that gap. In practice, how much they close it depends almost entirely on how they are used — and most teams are using them for the wrong thing.

The Difference Between Tracking Tests and Understanding Coverage

The most common use of test management tools is organizing and tracking test cases. Which tests exist, which ones ran, which ones passed, which ones failed. This is useful as far as it goes. It gives teams a record of testing activity. It does not give them visibility into what that activity actually means for the risk profile of a specific deployment.

Coverage visibility and coverage tracking are different things. Tracking tells you what happened. Visibility tells you what it means.

A team with genuine coverage visibility can look at a proposed deployment and answer specific questions. Which test cases cover the code areas that changed in this release. Whether those test cases reflect current system behavior or were written against behavior that has since changed. Whether there are gaps in coverage for the services most likely to be affected by this specific change. Whether the test cases that passed have been passing consistently or intermittently.

A team with only coverage tracking can tell you that tests ran and passed. That information is less useful than it sounds when the tests that passed may not be the ones relevant to what just changed.

The shift from tracking to visibility is what test management tools make possible when they are connected to the right information sources and used to answer deployment questions rather than just produce activity reports.

What Visibility Actually Requires

Visibility into test coverage requires three things that most teams treat as separate concerns but that test management tools can bring together.

The first is traceability between test cases and the code they cover. When a developer changes a specific service or module, the relevant question is not whether the overall test suite has adequate coverage. It is whether the test cases covering that specific area are current, comprehensive and passing consistently. Without explicit traceability between test cases and code areas, this question cannot be answered from a test management tool. With it, the answer is available in seconds.

The second is currency of test case behavior. A test case that was accurate six months ago is not necessarily accurate today. In systems where services deploy independently on their own schedules, test cases covering service interactions can drift from current behavior without anyone noticing. The test still runs. It still produces a result. However, the behavior it is validating may no longer reflect how the service actually behaves.

This is where the source of test cases matters more than most teams realize. Test cases written against API documentation or developer assumptions about service behavior start drifting immediately. Test cases sourced from recorded real interactions stay current because they reflect what services actually do rather than what someone expected them to do.

Keploy addresses this specifically as an open-source API testing tool — it captures real HTTP traffic from running services and generates test cases from those actual interactions. When those generated test cases feed into a test management workflow alongside manually authored ones, the coverage picture that the tool surfaces reflects current service behavior rather than historical assumptions. The visibility the test management tool provides is only as accurate as the test cases it is organizing, and test cases grounded in real traffic carry significantly more accuracy than test cases grounded in documentation.

The third requirement is change-aware reporting rather than aggregate reporting. Most test management tool dashboards show aggregate statistics — overall coverage percentage, total test count, pass rate across the full suite. These numbers are useful for understanding long-term trends. They are not useful for making a deployment decision about a specific change.

Change-aware reporting asks different questions. Given the code areas modified in this release, which existing test cases cover them. Of those test cases, which have been passing consistently, and which have been flaky. Are there coverage gaps in the affected areas that should be addressed before this change ships. This kind of reporting requires test management tools to understand the relationship between code changes and test coverage — not just to aggregate test results.

How This Changes Deployment Decisions

When test management tools provide genuine visibility rather than just tracking, the quality of deployment decisions changes in a specific way.

Before visibility, deployment decisions tend to be binary. The tests passed, so we are deploying. The tests failed, so we are not. There is no graduation between these states. Either the pipeline is green, or it is red.

With visibility, deployment decisions have more information. The tests passed. The coverage for the changed areas is current. The test cases covering the highest-risk integration points have been passing consistently. There are no known coverage gaps for the services affected by this release. That is a substantively different deployment decision than the one based only on a green pipeline.

The converse is also more informative. The tests passed, but the coverage for the authentication service — which was modified in this release — has not been updated since the last API contract change three months ago. That is a signal worth investigating before deploying, regardless of what the pipeline says.

Neither of these deployment decisions is available without visibility. They require test management tooling that connects test results to coverage information, traces coverage to specific code areas and surfaces gaps and currency issues rather than just pass/fail aggregates.

What Teams That Ship Confidently Actually Do

The engineering teams that consistently ship with genuine confidence rather than hope tend to share a few practices that distinguish how they use test management tools from how teams that struggle to use them.

They review coverage as part of code review rather than as a separate QA activity. Before a significant change merges, someone checks whether the existing test cases adequately cover the affected area and flags gaps that should be addressed before the change ships. This moves coverage from a post-deployment concern to a pre-merge one.

They treat coverage gaps as information rather than as failures. When test management tools surface an area of the system with incomplete or outdated coverage, the response is not to block the release automatically, but to understand the risk. Sometimes the gap is in a low-risk area, and the deployment proceeds with awareness. Sometimes the gap is in a critical path, and it gets addressed first. The difference between these two outcomes requires visibility that a pure pass/fail pipeline cannot provide.

They maintain test case currency as an ongoing practice rather than a periodic cleanup. When a service changes its API contract, the test cases covering that service get updated alongside the change rather than eventually. This is easier to sustain when test management tools make coverage currency visible — stale test cases are flagged before they accumulate into a systemic problem.

Conclusion

The gap between having tests and having genuine coverage visibility is not a tooling gap in the sense of needing better test runners or more sophisticated CI platforms. It is a visibility gap — a gap in what test management tools are being used to surface and how that information connects to deployment decisions.

Teams that close this gap do not necessarily have more tests than teams that do not. They have a clearer picture of what their tests are telling them. They understand which areas of the system are protected, where the gaps are and whether the test cases they are relying on still reflect current system behavior.

That picture is what test management tools exist to provide. The teams that use them to build it are the ones that ship with confidence — not because nothing ever breaks, but because they understand their actual exposure before they deploy rather than discovering it afterward.

August 12, 2026
Why Automation Fails Without the Right QA Mindset
In the DevOps world, with its faster release cycles and continuous delivery, quality assurance (QA) cannot be sidelined until the end of the process. Companies that actively use automation achieve up to 60% faster release cycles compared to those relying solely on manual testing. However, automation alone is insufficient. Without the right QA mindset, moving faster can easily backfire if quality is not built into every stage. QA teams must evolve to become quality enablers throughout the software delivery cycle. Rather than joining at the tail end of the lifecycle, they must work alongside product owners to understand functional requirements, collaborate with architects and developers to grasp solution design and ensure visibility into end-to-end solution delivery.

Getting involved more deeply from day one means QA can help build quality from the outset through deployment. In response to this shift, QA professionals need to step up by:
- Designing and managing automated tests in continuous integration/continuous delivery (CI/CD) pipelines
- Focusing on risk-based testing
- Collaborating with teams to ensure quality is integrated into every step of the delivery process
In a fully automated pipeline, QA teams add the most value when they go beyond just testing and contribute as collaborative partners, strategic advisors and critical thinkers throughout the entire development process.

The Importance of the Right QA Mindset

Automation is key to fast, continuous delivery, but its real impact depends on the mindset driving it. In fact, 63% of organizations deploy code without thoughtful testing in pursuit of faster delivery timelines. The so-called developmental mindset focuses solely on speed, code coverage and scripting mechanics — often resulting in a check-the-box mentality, where tests verify expected outcomes but overlook real-world usage, potential risks and user experience.

The QA mindset, by contrast, offers a broader, more critical perspective. It is rooted in exploration, skepticism and curiosity. As QA professionals, we don’t just ask whether the software works — we ask how it might fail, where it could break and whether it fulfills the user’s ultimate needs. This mindset is essential for prioritizing automated tests based on business or user risk, rather than simply following functional decomposition.

With this strategic mindset, we design automated tests that anticipate edge cases, test assumptions and reflect actual user workflows. This leads to earlier detection of defects, fewer production incidents and, ultimately, a smoother user experience. In the long run, it also improves team productivity and reduces rework and reactive firefighting by catching issues in advance.

Automation alone doesn’t guarantee quality — it is only as effective as the tests it is scripted to run. If the requirements are misunderstood, automated tests may pass while critical issues remain undetected. I have seen failures where teams relied solely on automation without involving proper QA practices, leading to tests that validated incorrect behavior. Automation frequently fails to detect new or unexpected issues introduced by system upgrades. It often misses critical problems such as faulty data mapping, incomplete user interface (UI) testing and gaps in test coverage due to outdated scripts.

Lack of adaptability is another common obstacle that I’ve repeatedly seen undermine automation testing efforts. When UI elements are tightly coupled, even minor changes can disrupt test cases. With the right QA mindset, this challenge is anticipated — promoting modular, maintainable automation strategies capable of adapting to frequent UI and logic changes.

Automation lacks the critical analysis required to validate business logic and perform true end-to-end testing. From my experience, the human QA mindset proved essential during the testing of a mortgage loan calculation system. While automation handled standard calculations and data validation, it could not assess whether the logic aligned with real-world lending rules. Here is where human insight was crucial: Automated tests confirmed data processing and standard calculations, but couldn’t verify whether the outputs complied with evolving lending regulations and complex business requirements. QA ensures that test scenarios are aligned with business goals and remain adaptable to change.

As QA professionals, we often serve as a bridge between developers and business teams — translating business requirements into actionable testing scenarios and ensuring the team’s efforts focus on what truly matters.

Why Human QA Thinking is Irreplaceable

Automated scripts can only validate what they have been explicitly designed to check. That is why human-driven testing remains essential, even in highly automated pipelines. Human-driven testing is especially valuable for UI and user experience (UX) validation, exploratory testing and identifying edge cases that automation may miss.

For example, in a human capital management payroll system upgrade project with zero downtime, we applied a hybrid testing strategy that combined automated and manual efforts. Automated regression scripts ensured core functionality remained stable, while human-driven tests were used to validate new scenarios and feature upgrades. This ensured proper integration with other modules and subsystems. The result was a fast, stable CI/CD pipeline that still allowed space for creativity.

Human insight remains a critical part of both DevOps and QA. Without a mindset grounded in strategy, risk awareness and a deep understanding of how real users interact with software, automation can mislead teams. Quality teams don’t just bring tools — they bring perspective. By asking the hard questions, they ensure testing efforts stay aligned with what truly matters to both the business and the end user.

Key Tips to Strengthen Your QA Mindset

In conclusion, here are the key QA mindset principles for successful test automation:
- Integrate QA from the start. Involve QA early in the process — don’t wait until the end.
- Adopt a risk-based, user-focused approach. Prioritize testing based on business impact and user risk. Go beyond validating expected outcomes — explore how the software might fail in real-world scenarios.
- Design maintainable and adaptable test automation. Avoid brittle scripts that break with every minor change.
- Blend automation with manual testing. Let automation handle data and calculations, and human testers cover new features, UI/UX and edge cases through exploratory testing.
October 2, 2025
Accelerating DevOps Pipelines With AI-Native Testing

Achieving speed in DevOps and moving fast with CI/CD pipelines means getting builds done quicker, deploying more often and making changes faster to keep up with user needs.

However, keeping that pace while also testing thoroughly and ensuring high quality & security is exceedingly hard, as the older ways of testing — involving manual work or automation that breaks easily — slow things down, acting like a bottleneck that stops the whole pipeline from running smoothly. This also often leaves teams wondering how to actually speed up their DevOps without cutting corners on important testing.

Artificial intelligence (AI)-native testing changes this significantly. According to Shahid Ali Khan, Principal Engineer at LambdaTest, a cross-browser testing platform, “It’s not just adding some AI bits onto existing testing tools; it’s about using generative AI and predictive AI together to completely change how we think about, build and run tests through the whole DevOps process.” Using AI in testing lets teams automate things and add intelligence in ways they couldn’t before. Enabling continuous testing that actually matches the speed of continuous delivery is clearly the way forward.

It’s more than just a small step forward; it’s like hitting the reset button on how we approach ‘software quality at speed’. Michael Kwok, Ph.D., VP at IBM Watsonx Code Assistant and Canada Lab Director at IBM, sees AI-native testing “revolutionizing software development,” promising to bring speed and quality together, not as things that work against each other, but as built-in parts of the DevOps pipeline.

I spoke to industry leaders to explore how AI-native testing is making a real impact. Drawing insights from these conversations, we’ll look at the main AI features driving acceleration, their tangible impact on DevOps metrics and developer productivity and what to consider when bringing AI-native testing into your own process for building fast, secure software in the future.

Putting AI to Work in Your Test Pipeline

Getting AI into testing isn’t just one small task; it involves building several smart features that work together to speed up the whole testing part of the pipeline, getting software out faster with fewer hassles.

Intelligent test creation is a big one, because building good test cases takes time and a lot of thought to cover everything comprehensively. Generative AI changes this process fundamentally. Gary Arora, Chief Architect of Cloud and AI Solutions at Deloitte Consulting, stated using a platform powered by LLMs helps break down bigger project goals into user stories, and then GenAI takes over to “generate comprehensive test scenarios — positive, negative and edge cases — which are then converted into executable test scripts and integrated directly into our SDLC.” He called it “intelligent, end-to-end test creation at scale.”

Jon Matthews, VP, Engineering at Functionize, also talked about generative AI doing this autonomously, saying it helps get “full coverage achievable in days versus months,” which really removes technical hurdles in building tests quickly using things they call AI agents.

Predictive AI, on the other hand, helps teams decide which tests matter most in a given situation. After all the tests are created, or even if you have tons of older ones, you don’t always have to run every single test every time, especially if you need feedback fast within the pipeline. Khan of LambdaTest talked about predictive test selection, looking at things such as code coverage and how often tests have run before to “optimize the most relevant test suites.” He sees it helping “to identify out of all features which are the most important ones that I should run first in my pipeline based on a priority.”

Arora affirms this, saying predictive AI adds foresight by “prioritizing tests based on code impact and historical failures to help ensure we test what truly matters.” This focus means that teams can be confident about the most critical tests much faster, speeding up CI/CD pipeline runtimes, which currently can feel really unpredictable.

And when tests do fail, figuring out why is a real time sink for developers. However, AI helps here too with rapid root cause analysis. Karan Ratra, Senior Engineering Leader at Walmart, explains that when there are failures, generative and predictive AI can help “segregate the errors based on issues rather than just printing them in the logs, and help us categorize them.” He sees this saving up to 80% of developers’ time in finding the actual root cause.

Matthews stated that AI-driven root cause analysis analyzes defects “to uncover their origins,” generating reports automatically within “seconds” instead of the “15–30-minute manual fix, often across hundreds of tests per release” it might take otherwise. Arora also mentioned predictive AI analyzing logs and tracing errors straight back to the code change, saying this “turns hours of debugging into minutes of insight,” and when problems pop up, the system can even suggest the root cause based on past issues it has learned from.

AI also helps with auto-healing and test adaptation, keeping tests useful over time, especially with frequent code changes — as tests written today might not work exactly right tomorrow if parts of the application are updated or refactored. Matthews suggests that incorporating auto-healing capabilities lets tests “adapt dynamically to changes in the application without requiring manual updates.”

This combination of AI powers — intelligently creating tests, picking the right ones to run first, figuring out quickly why something broke and even helping tests fix or adapt themselves — is what makes AI-native testing capable of pushing things faster through the DevOps pipeline.

But Does AI-Native Testing Actually Move the Needle?

Understanding what AI-native testing can do in terms of capabilities is one thing but seeing how it actually changes the numbers and helps people doing the work is where the real story is told. Putting these AI capabilities into practice has a clear impact on the metrics that matter most in DevOps and on how much work developers and QA can actually get done daily.

Considering key DevOps metrics, such as how often teams can deploy or how quickly they can fix things that break, AI-native testing shows some serious promise and is already delivering results. Experts are anticipating and observing substantial improvements across the board. Deployment frequency could increase by 40% to 60%, lead time for changes could improve by 40% to 50% and mean time to recovery (MTTR) could decrease by 40% to 60%. Arora of Deloitte Consulting is seeing this happen already in early projects, where implementations “are already showing meaningful reductions in MTTR and notable drops in change failure rates,” meaning that there are “fewer surprises in production and faster recoveries when they do happen.”

Matthews stated that AI-native testing speeds up the whole software development cycle by “removing the primary bottleneck in QA and release processes,” which directly “reduces failure rates, shortens remediation time and thereby lowers mean time to recovery (MTTR) & change failure rates.” These significant benefits result from capabilities such as predictive selection, which allows teams to deploy faster by not having to wait for every single test to run before deploying, especially if the most important ones pass quickly.

This leads right into how AI-native testing makes developers and QA teams much more productive in their daily work. Not having to spend so much time on the boring or frustrating parts of testing and debugging means that they can channel their energy into more valuable work — building features and innovating. Matthews explained, saying it “reduces the time developers spend creating and fixing tests,” and because they get feedback faster within the pipeline, they “can work more efficiently overall.” He also added that with testing cycles being shortened, “more time can be devoted to coding and innovation.”

Khan also talked about how the developers at LambdaTest are now spending “less time in writing boilerplate codes or debugging fail builds” and getting “instant feedback on the code that they’re writing,” which then means “more frequent deployments in production with more confidence and less delay.” He sees QA teams building “high-value validations” and catching “oversighted or overlooked scenarios” they might have missed before, ultimately creating “stronger, firmer validation frameworks” that demonstrably improve customer experience.

Bringing AI Testing Into Your Team

Seeing the potential and the early results of AI-native testing in DevOps is exciting; however, actually introducing it into the way teams work every day involves more than just picking the right tools or platforms. For organizations looking to accelerate their DevOps processes using AI, thinking carefully about how to adopt it and how well it suits developers and QA is absolutely key to success.

Kwok of IBM sees this big picture, stating clearly that AI-native testing “will revolutionize software development.” It’s not just about making existing steps faster; it’s about reshaping the entire process and how teams operate together. This kind of fundamental change needs careful thought about how AI integrates into the existing culture and workflows without disruption.

And when it comes to bringing AI frameworks into teams effectively, Khan mentioned some really important points in our interview based on his engineering purview at LambdaTest. He talked about the importance of “how we start adopting this and how we start onboarding the teams into adopting AI frameworks” — and specifically — “how do we use that for cross functional collaborations” effectively. He also stressed that people in the team shouldn’t feel like “they are getting oversighted, or they are getting ignored because of this technology. It’s not replacing anybody.” So, making sure that everyone understands clearly that AI is a tool designed to augment and help, not replace jobs, is particularly important for trust and adoption.

Khan strongly believes this needs to go “hand in hand with transparency, ethics and enabling personnel in the teams” so they understand the technology and its benefits. Making this shift smoothly can’t happen with just a top-down mandate; it requires deep collaboration across developers, QAs, DevOps and even product managers and leadership, as pointed out earlier by Khan. It’s about building a “very symphonic framework or environment” that fosters cooperation between all the different groups involved in the software life cycle, and getting that adoption and onboarding right, built on clear ethics and thoughtful enablement, is what will really make AI-native testing a success in the long run, helping everyone from the developers to the senior leadership understand its strategic benefits.

AI-Native Testing as the Future of DevOps Acceleration

So, looking at how much ground AI-native testing covers — from making smart test cases and picking the most important ones to run, to figuring out why things broke in seconds and even helping tests fix themselves or adapt — it’s clear that AI-native testing isn’t just another tool added to the stack. It’s fundamentally the next big step in actually accelerating DevOps pipelines sustainably.

By bringing generative AI and predictive AI directly into testing workflows, teams can finally move at the speed DevOps promises without having to worry about letting crucial issues slip through quality gates. It’s definitely about getting software delivered faster for sure, but it is also about making sure that the software is reliable and that the developers building it aren’t getting burned out by frustrating manual work.

Making this shift means factoring in the platforms designed from scratch to handle this kind of intelligent, automated quality and security throughout the entire CI/CD pipeline. For making secure, high-velocity software delivery a consistent reality, platforms that bring together different security and testing capabilities and make them work smoothly within the existing DevOps ecosystem are becoming absolutely essential.

June 11, 2025
A New DevOps Test Environment Platform is Needed
Test environment management is bottlenecking DevOps transformations and is driving the need for a new test environment management platform solution. On-demand test environments in which developers and testers do not have to wait for a shared environment are sorely needed.

The evidence is clear. In my book Engineering DevOps I indicated that finding and setting up test environments is a primary source of bottlenecks in DevOps value streams.

In my February 2021 webinar with Plutora, I presented Best Practices for Developing Test Management Environments. I indicated that orchestrating and managing test environments is often a primary source of delays and waste within value streams.

The State of Testing Report 2020 by PractiTest indicates that DevOps practices increased 57%; from 26% in 2017 to 41% in 2020. The report revealed that 46% of testing has shifted to using CI/CD processes, 55% are actively involved in shifting testing left and 36% are testing in production. This is clear evidence that DevOps testing is on the rise.

Wolfgang Platz’s December 2019 article Why Software Testing Remains a Bottleneck indicates that testing is responsible for more delays than any other part of the development process – accounting for 52% of the bottlenecks in the development process. And 53% of respondents to the World Quality Report 2020-21 indicated that test environments are not available when needed, 67% lack the capability to book and manage test environments and 71% lack visibility of available test environments.

An article by Velocity echoes the problems with pre-production environments. “With the rise of microservices and cloud-native applications, environments are becoming more complex. Often you need to wait or compete to get access to develop on a specific microservice, or can spend time waiting to deploy to a shared staging environment only to realize something already changed, and you’re back to square one.”

The Continuous Testing Report – 2020 by Capgemini defines test environment management (TEM) as managing infrastructures, test data and applications to support testing on demand. These environments are one of the greatest impediments to continuous testing. In this report, 36% of respondents said they are spending more than half their time building and managing test environments; 44% of respondents indicate they spend weeks of their time searching, managing and generating test data.

Test environments should be regarded as functions to be spun up or destroyed at will, with virtualization, infrastructure-as-code (IaC) practices, service virtualization, containerization and improved test data management.

Given all the above evidence, it is clear that a new approach to DevOps test environment management is needed.

The following are examples of benefits that DevOps test environment management platforms need to offer:
- Compatible with current tools:
  - Automatically map and configure application infrastructure to develop and test multiple services in an environment that runs just like it would in production. This needs to be fully agnostic to cloud and tech stacks, such as Kubernetes and other environments.
  - The platform needs to work on applications in parallel without disrupting others, with full access to data and third-party dependencies.
- Automatic configuration and maintenance:
  - The platform needs to automatically configure and update any changes within the production infrastructure’s mapping in real-time so ephemeral environments will always be synced with what is in production. Instead of having to spend time making sure any changes are configured, the platform will ensure the right environment that replicates production, without having to wait.
- Data and third-party dependencies:
  - The platform needs to load the relevant data and third-party resources that are vital parts of the environment to test, work and debug against, so test environments will, as closely as possible, mimic working in production.
- Team collaboration:
  - The platform needs the capability to share ephemeral test environments with other stakeholders so they get visibility and can collaborate for faster feedback cycles. This is also important for collaborating for debugging.
What This Means

DevOps testing is on the rise, and test environments are a major bottleneck for DevOps testing. A new DevOps test environment management platform is needed to reduce the bottlenecks. On-demand test environments in which developers and testers do not have to wait for a shared test environment are needed. Such a solution would improve agility and reduce wasteful orchestration bottlenecks in test, improve quality and risk management through comprehensive and automated testing and enable continuous improvement by providing more intelligent insights and feedback.
August 9, 2021
EP 8: Testing Is Not a Monolith

With DevOps comes implementation of CI/CD, automation and testing. Is this sufficient to achieve “shift left testing”? Or does testing need to shift further left? Do we need new or additional tests for acceptance (ATDD) or behavioral testing (BDD)?

In this episode of DevOps Unbound, we are joined by Christine Fisher of NAIC, Leandro Melendez (aka Señor Performo) of Qualitest, Ken Pugh of Ken Pugh Inc., and Mitch Ashley of Accelerated Strategies Group to explore the state of testing in DevOps and CI/CD pipelines, and where testing can evolve to keep pace with software development and delivery.

The video is immediately below, followed by the transcript of the conversation. Enjoy!

Transcript

Alan Shimel: Hey, everyone. Thanks for joining us on another edition of DevOps Unbound. We have a really great DevOps Unbound topic to explore today, and that is that testing is not a monolith. A lot of people, especially people who aren’t familiar with testing, just say testing—QA, testing—and we think of, oh, it’s a test. And we know there’s different kinds of tests, but do we really know all the different kinds of tests that need to be done, as we’re developing and deploying and managing and monitoring software applications and so forth? I don’t think so.

I think, really, we could cater this show to those of you out there who do know, but I think today, we want to talk a little bit to those who don’t know as well and let’s try to get you educated on what some of the various aspects of testing are that you may not be familiar with. They’re important, nevertheless, and how they influence what you see and the apps you use and what we may see going forward.

I’ve got a great panel to introduce you to today. I’m gonna allow them each to introduce themselves. Let me first start off with Leandro Melendez, I can’t say it with that Spanish roll of the tongue that you can, Leandro, I apologize. And Leandro is known in the industry as well as a name—I’m gonna let, I don’t wanna steal your thunder. Leandro, introduce yourself.

Leandro Melendez: Thank you very much, and don’t worry, you made a great pronunciation of my name, way better than what I’m used to, so that was pretty good.

Shimel: Okay.

Melendez: Yeah, repeating, I’m Leandro Melendez, a performance test manager from Qualitest Group, and as well, I’m known on the internets as Señor Performo, as you all said, Performo with strong R’s.

And well, we provide several services to—my specialty is performance testing, integrating it right now with full continuous integration DevOps and agile methodologies. But as well—and you mentioned it very well earlier—it starts to permeate in other areas of testing when you start to do it holistically and at a large scale, and thinking on all of the steps on the SDLC. So, we will talk a little bit more about it.

The last thing about me, as I said, Señor Performo, on the internets, I have a YouTube channel, I try to help educate people on especially performance testing, but anything that concerns testing and methodologies, agile, DevOps, and all that, I’m very happy to help provide information as well as hear whoever gives me a chance to speak and share the knowledge, I’m very happy to.

Shimel: We’re honored to have you here, Leandro, thank you. Next, we’re gonna introduce Ken Pugh. Ken is—well, Ken, you introduce yourself. Go ahead, I’m sorry.

Ken Pugh: That’s okay. Hi, I’m Ken Pugh. I’ve been in the programming software development field for about two-fifths of a century, and in agile for about a fifth of a century, and doing everything from gathering requirements to final testing.

I’ve got a book out called “Lean-Agile Acceptance Test-Driven Development: Better Software Through Collaboration,” and I’ve been teaching, training, and emphasizing acceptance test-driven development-slash-behavior driven development for the past 15 years.

Shimel: Fantastic. And for our fraction challenged friends in the audience, a fifth of a century is about 20 years, so that’ll give you a little idea.

Next up, we have Christine Fisher. Christine, welcome, introduce yourself, please.

Christine Fisher: Thank you. My name is Christine Fisher, I manage a team of BAs, QAs, and UX at the National Association of Insurance Commissioners. My team and I have been on a journey with behavior-driven development over the last two and a half years, and as we get ready to talk about testing here, my big secret is, I’ve actually never been a tester, I just have managed some. But with our jump into DevOps, I felt very strongly about advocating for what our role should be and how the work should change to keep up with the other changes that were happening with DevOps, which helped lead us to BDD.

Shimel: Fantastic. Pre-COVID times, we’d say, “But I did stay in a Holiday Inn Express last night,” but we can’t even say that any more.

Last but not least is my co-host and anchor on our show, Mitchell Ashley. Mitchell, why don’t you—quick introduction?

Mitch Ashley: I don’t know what fractions I’ve worked, but I do know that I used that joke last night and my family had never heard it, so—so much for the Holiday Inn Express joke. [Laughter]

But, no, I’m an old software person, started in software, and have done lots of things around both cloud as well as, I’ve also been on the vendor side, so network security and SaaS services. So, testing people are some of the most unique and, in my world, beloved people, because those are the folks I really look to to say, “How are we doing? Where are we?” And I think even more so in the DevOps time, so I’m excited to explore this with you all. So, thanks for joining us.

Shimel: Fantastic. So, first of all, let’s assume that our whole audience aren’t test experts, they’re more DevOps generalists, there’s a good smattering of cybersecurity folks, cloud native, infrastructure, ops, and we’re throwing out some ADD—behavior-driven, BDD—you know that they may not be familiar with.

So, why don’t we do a quick kinda definition? And we’re gonna ask the non-tester, if you will, Christine, if you wouldn’t mind leading off with what are these references we’ve made to these types of testing?

Fisher: Sure. So, when we’re talking about behavior-driven development, we’re talking about understanding the behavior of the end user and understanding what the workflows are, why they are trying to accomplish that task, and working on testing, understanding the requirements, testing those requirements in that way, not just writing a requirement, developing a requirement, testing a requirement without any true understanding of what the person who’s going to use the software on the end does.

And when I talk about not being a tester, more on the functional side with requirements as a BA, things like that, that was always the interesting part to me. And then sometimes, we would test something and it would be tested correctly, but we still missed the mark. And so, having the whole team understand what the end user wants to do, putting it in the common language, that’s really important for behavior-driven development.

Shimel: Agreed, agreed. Ken, do you want to add anything?

Pugh: Yeah. So, one of the big things in behavior-driven development or acceptance test driven development is having the triad, and Christine sort of made a reference to it. The customer, developer, and tester get together prior to implementation, create the scenarios, the scenarios become the tests, and the developers are given those tests to run against their code, and those tests are the agreed upon, shared understanding of how the system should work.

Shimel: Got it. Señor Performo—is that correct, yeah? Sí?

Melendez: Very well said, yeah. [Laughter]

Shimel: Señor Performo, how does acceptance-driven, behavior-driven development figure into your performance testing kind of regimen, if you will?

Melendez: Well, as I mentioned earlier, these type of concepts permeate in all the testing practices, if I could say, and very well as it was said here, I like to call it the three amigos, that you put them together and even before you start rolling anything, drafting anything, you get the requirements well together with what you are going to be testing and what has to be marked as passed before you even—or a developer—can mark a task as completed. And I might have said it wrong, a developer, because in general, you’re supposed to be a team, and everyone is responsible for each element.

And this comes together with performance metrics, where, in general, most of the people, when they think about performance, their mind goes right away to load tests, when I have to put hundreds of users to slam a system, where, if you see it from the beginning as well as some of these requirements when you generate them that, I don’t know, I’ll give an example, that the text field has to just allow a certain given of characters or be this color, rounded—little definitions like that, it should have included performance metrics as well inside of it. So that, when you are generating any piece of software or functionality, it’s measured there and then.

I have to say it’s an issue that I found often in the old waterfall days. I would arrive to do a load test after six months of development and start to automate some things that I’m like—this, hey, everybody, this is visibly slow. Just, I’m a single guy. I don’t need to load test it, why no one caught this before?

And those are the most general problems, in terms—I think, as well, in functionality, some things that come from even unit tests, low level tiers where you could catch many of those bugs and performance SLIs that are not met and that’s how it integrates a lot with performance. I always preach that you need to start to think about it as early as possible. And as Christine mentioned, getting the people together, what are the goals, what is the user gonna require, and what are the tasks? Because not everything should respond super fast and not everything should or can have a good response. I mean, we all wish that a—I don’t know, monthly billing period can be closed in three seconds, but that’s just for that response time from a click.

So, we need to filter that out depending on what is the user needing, willing to endure, to call it in a way, and put it together, and this permeates as well. It’s some sort of functionality like—yeah, I need this to respond in this period of time for me to be comfortable using it, for it to be useful, functional.

Shimel: Comments, thoughts from the team?

Pugh: So, as you were mentioning that, I don’t know if you’re familiar with healthcare.gov.

Melendez: [Laughter] Oh, yeah.

Pugh: I think they should’ve had a few performance tests done well before they released it. It seems like—I don’t know, that it was like, maybe the day before, they went, “I guess we’re gonna have a lot of people on this.” [Laughter]

Melendez: Yeah, and I think, and to paint and explain what I was saying, with a single element that doesn’t perform well, that doesn’t work well, even from the initial tier, they should have created these requirements, saying, I don’t know, “When I log in and enroll, healthcare starts the process when I click this. It should respond this fast, it should not do, I don’t know, this many full table scans, connections to the database,” things that you can go way, way low level in the IT tier, and others and it’s just like, “Yeah, the user should not wait more than five seconds on that click”—a single user, one person.

I am almost sure that most of the problems that they had was that they didn’t start to tune that, and they allowed the developers to check in all that awesome code, to call it euphemistically. And that’s when you start to scale up problematic things is when everything derails. And yeah, healthcare.gov, aside from not doing good load testing, they had multiple issues at the single piece level, and that’s where it is, push it left. Start to think at a unit level, single item requirements, when you have the three amigos together talking about these things, take everything into account and don’t let your little Post-its to be marked as done and move to the next area of your board without covering some of those basic requirements, which most people doesn’t think about or include them. Or whenever, and this is a little biddity or tiddity, when the developer is working on it, should have in mind all those requirements—functional, performance, security.

And that’s where I say, all this permeates to all the testing areas and other metrics when other areas of testing are doing their tests—manual, functional, automated, even security—my goal is to start to gather performance metrics so that everything happens together and we don’t have to wait until the end for big load tests to figure out that healthcare.gov will have what happened, sadly. [Laughter]

Pugh: So actually [Cross talk]—no, so, actually, you were talking about something that I’ve, in fact, I’ve done a few blogs on this, but it’s been around for a while is the testing matrix. I.e., that shows, since we’re talking about testing and it’s not just about functionality, it describes all the different sorts of tests, from the functionality tests which are on one side of the matrix, to the quality tests which are on the other side, including performance, security, usability and so forth. And that you’re testing—I mean, ATDD and BDD tend to concentrate on the functionality, because we’re gonna be talking with the customers, that’s what you want.

But you ought to have all those other tests, the performance tests as you were mentioning, already established ahead of time—100,000 users, how fast should it be, and so forth. Oh, and in fact, just the usability you were talking about—how long should it be after I click that I get to the next page? And those things could be established ahead of time, and if a developer goes up and it’s not meeting it, he knows to fix it. That’s, we don’t have to—

Melendez: And I would say here, not only established, but worked on from day one. Because as you mentioned, even some usability, security—those are to be checked when everything is assembled, and there are many techniques that we can start to embrace, starting out with automations at different tiers.

As I said, I’m a big proponent or supporter of—yes, have unit testing, automated everywhere, have the coverage according to the pyramid. Don’t focus so much on manual, front end, or after the fact, leaving everything for the second Friday of the sprint when everyone wants to leave and here, tester, there you go—start our meeting. That’s where everyone has to start to work together and pushing it left.

Since day one, start thinking about performance, security, functionals. And even some have told me, “Yeah, but there are things that, as you said, usability, that cannot be done until the end.” Well, you can start creating mocks, you can start figuring how things should work and start to work them. And, again, I think the mindset of keeping testing phased, waterfall ways—we need to start to work everything on par and get it rolling, yeah, I totally agree.

Fisher: I’d like to speak to that point a little bit with DevOps and with the testing. We’ve been talking about a team and we’ve been talking about getting people involved, and what I was seeing in my organization when we first started talking about DevOps is, everybody is an engineer. We can all do these jobs, things like that, right? And jobs might change, there’s room for everybody, but what you’re doing might change a little bit. But that can be really scary to hear, and especially if you are a manual tester who really doesn’t have an inclination or desire to learn automation—and that’s okay. Like, that’s a different conversation, but I am always a proponent of manual testers.

So, one of the reasons that really pushed me toward that is hearing that conversation, seeing people on my team being told that their job might change, but we don’t know how, you’re an engineer, you can do what everybody else on this team does, but that’s not true. Not everybody has those skills.

And so, making sure that the team is involved, and when you can have these conversations, BDD, all of these things that we’re talking about now, right, are getting the testers involved in the conversation earlier, and that makes them part of the team. And we have taken our BAs, we’ve taken manual QA, we’ve taken our automation QA—we get them all involved in that conversation. We make sure the developers know where they are in the process, we make sure the developers are adding to the conversation and part of that process.

And it was really important to me in that DevOps environment to get out in front of that and help define that role when somebody is saying, “We don’t”—you know, again, your role might change, but we’re not quite sure how—I wanted to help my team define that and put us in a good position so we weren’t just sitting there to have, God forbid, a developer come down a few years later and tell us, “No, you do it like this.” We wanted to put ourselves into that position and having those conversations, I think, is such an important part of that and changes some of that dialogue a little bit about we can all do the job. We can’t, but we all have important jobs to do to make it work.

Ashley: You know, Christine, I think about just an analogy—robotics for surgery. I still want that doctor in there guiding what’s happening, right? We don’t need any bad things happening. [Laughter] Well, same with software. You know, some of the worst folks at testing software are developers, right? You’re just too close to it, even if you’re writing the automated tests.

To your point, and I think several of you have made this, is, it’s an opportunity for us to (a) collaborate much more closely together rather than testing to be the thing right before documentation at the very, very, very end of the release, right, which is what it used to be. But also, what do we automate? It doesn’t help to automate the wrong tests—let’s get the right things in there. And where is the user in, the customer in performance, in functionality, in speed?

By the way, one of the things that kind of baffled some development teams is when I would say, “Speed is quality.” They’re like, “What do you mean, speed is quality?” I’m like—well, if Netflix takes three seconds to start your video, okay, that’s good. If it’s pausing for three seconds every five minutes, that’s bad, right?

So, speed makes a difference in terms of the perception of the experience. So, it is really kind of a harmonious effort where we have to bring all of that together.

Melendez: And now that you mention it, I’m gonna play a little bit devil’s advocate towards and against at the same time manual. Some forums that I participate in here in Mexico talk about that a lot, like—hey, too much automation. Are we manual testers gonna be out of a job and the machines will replace us, and all those paranoid ways of speaking.

And I generally explain to them with analogies—I like, a lot, to use analogies. In New York City, before there were any automobiles, cars, there were lots of horses, and horses produce a lot of stuff that was dropped on the streets, and there was a job needed for that, to pick up some of the horse. And I’m sorry if this is an analogy of manual to what the horse produces, but the point that I’m trying to do is that once cars became available, that job decreased considerably. But we still have places where there’s people that work with the horses, maintain them, brush them, and do all those tasks which, as many cars as we may have, we won’t ever have a horse polishing machine or anything like that. There are things that, truly, a human being is the only person that can do it really well—manual testers are very good for those tasks.

I think, on the other hand, there are too many things that are tried to be done with manual testing. I have seen in many organizations that there are tasks and tests that could be pushed left, that could be started to be done before automated programmatically and leave whatever is truly necessary and where the manual testers are going to be the rock stars and the only ways to do it allow them to.

So, I think—yes, I would suggest most manual testers to start to pick up a little bit on technical and developing, yadda yadda. I think in the modern days and the upcoming times, everyone should know some sort of development, that should be in schools. And we’re moving into a technological world at a pace that whoever doesn’t pick it up might be a bit handicapped. But, on the other hand, the skills of a good manual testers, when you have that clinical eye to say, “Hey, I see the error” or you don’t even have to touch the application; let’s say, most probably, it accepts fields that it shouldn’t in that field. And most of the times they are right, and you need that input.

So, on both sides, they won’t be replaced, but they should get used to using the tools as they are. Technology is not here to replace them but to empower them, and they should be able to use it, I think.

Fisher: Skills can be—excuse me. Skills can be enhanced, absolutely, and I agree with what you say. Nobody wants another meeting on their schedule, but on my team, we have added what we call a test strategy meeting and we get the BA and the manual and the automation QAs in the room and we talk about each of our stories, and should we automate it, what are we looking for manually? And they took that further than when we first started, and they came up with a scoring guide to say, you know, what is our level of effort with this, what’s the repeatability for the user?

Those types of things, and they’ve realized that those conversations are so important and you’re exactly right—the manual testers aren’t doing the same job, but now that we’re able to really give them the time to pull those skills out, the critical thinking and what you were mentioning, you know, looking at this field, we are getting so much more quality from our tests, from both sides, and it’s really been a fantastic change.

Melendez: Yeah, that’s great, and optimizing and, in a way, calling it where you get the best bang for your buck, where to use those efforts, how to use them, prioritizing. And even some things that—hey, this is not a big impact if we get a defect here, we won’t even pay attention to it manually, whoever made it, or some others that this is critical if we get a problem that has to be done manually with two pairs of human eyes, capable eyes, to identify it, or we can let the machine do it.

I mean, there are balancing on testing, and I think falling into the cognition mental fallacy of, it’s called man with a hammer—someone learns how to use a hammer well and will see nails everywhere and try to hammer them. There are multiple other tools that we can use when it’s needed, when it’s more optimal, and I think it’s, as you say, prioritize, distribute, and make the best effort with each one of them.

Pugh: So, just getting into that, one of the things in the matrix and one thing I always suggest is that exploratory testing. You take your manual testers and turn them into exploratory testers—finding those things that the functionality tests are not gonna cover.

I always give an example, for those who don’t know exploratory testing, of—you’re on the web, you’re ordering an item. You press the submit button and the screen comes up and says, “Do not hit the back button. Do not refresh this page.” And the exploratory tester goes, “Yeah, back, forward, back, forward, refresh, refresh—maybe I’ll get it for free. I don’t know!”

Ashley: [Laughter]

Pugh: There’s no way of telling. [Laughter]

Shimel: We live for those moments—we get it.

Pugh: [Laughter]

Ashley: Yeah, don’t make any buttons red, they’ll be pressed, for sure.

Shimel: Right, yeah. But let me kinda take our conversation a little bit of a way, here. You know, we’re 25, 30 minutes in. We haven’t spoken a lot about DevOps, which is interesting, right? Especially in light of our original conversation off camera.

You know, when we talk about behavioral-driven testing and ADD testing and performance testing, these testing disciplines all existed before DevOps, right? Ken, you’re doing them for a fifth of a century or more, right?

Pugh: [Laughter]

Shimel: That certainly predates what we call DevOps. And I was here when I started devopsy.kinsta.cloud back in 2013-’14. There were a lot of testers who were running around like, you know, Henny Penny, “The sky is falling,” that they were gonna be out of a job, they were gonna be obsolete overnight. And, of course, that hasn’t happened.

Christine, you said that DevOps was kind of a driving factor in why you took the baton, here, why you took this position, why you feel it’s important. I’d like to bring it back, our conversation, a little bit to DevOps. How has DevOps changed BDD, ADD, right? Automation? Absolutely, right, but automation is not necessarily DevOps, right? We automated. We’ve mentioned a little shift left, you know, pushing it further, not pushing it further on the developers’ shoulders, because as, Ken, you said, we don’t meet the new boss same as the old boss, right? We don’t need that.

But how can we be more efficient, better at testing? Is it just a matter of automation, or is there something else there? Christine, I’m gonna ask you first, because it’s something that I think is near and dear to your mission.

Fisher: Sure. And I think for me, really, it is the conversations, how long has QA sat outside of the development organization, right? I mean, we can bring them in if we talk about agile and scrum and things like that—even, again, before DevOps, we can have a tester embedded in that team, but we still have lots of organizations that don’t do that.

Realizing that quality, not testing, that quality is the responsibility of the entire team, and that everybody needs to contribute to that conversation, we talk about breaking down the silos and everything with DevOps and we’re all in this together, and that includes testing. And for me, that’s been the biggest part. We can talk about BDD or whatever we want to talk about, however we want to test something. The most important part to me is that the testers are at the table and that they’re part of that conversation.

Shimel: And I think that’s inherently part of the DevOps cultural kinda mantra is that everyone should have a seat at the table. Ken?

Pugh: So, one of the things, getting back to the DevOps, is the fact that every time you have a test failure after something has been actually developed and put into the pipeline, it means you’ve gotta loop back. It means that you are slowing down the flow of business value through that pipeline.

So, the reason we want to create all these tests first is so that we pass them, and when it goes into deployment or staging, wherever you put it, that you don’t get a test failure. You don’t have to triage everything. And so, that’s why creating all of the tests first that the developers can be running while they’re developing their code or in their development environment is so important.

Now, I’d just like to add one more little phrase that I always use or talk about is—requirements should have a test for it, right? Pretty straightforward. How do you know whether the requirement is actually being met? But every test is a requirement. If you cannot ship a system with a failing test, then that test is a requirement the system must meet.

One of the things that developers always complain about is not having enough requirements, unclear requirements. The test that you create prior to implementation—BDD, ATDD or however you get ‘em—are the requirements. And now the developers have the test that they can go against those requirements, make it past, and then the flow through the pipeline doesn’t stop. It doesn’t, you don’t have a blockage because of failure later on.

Shimel: Fair. Comments on that?

Melendez: I think catching those errors, defining them, and being a bit more synergistic, to call it, in a way, it’s crucial. You very well mentioned all the testing practices, areas, and disciplines that exist come from waterfall and even before.

But I think a shift that was forced by DevOps and the agile methodology, continuous releases forced us to think in a different way. And I don’t know if it was the chicken or the egg where we started to see that we needed to throw down these silos and as well that the communication channels and capabilities to observe what was happening. Because as a tester, 15 years ago, I remember, you would just receive something finished, you had no idea, and somehow, you had some peculiarly written requirements that you had to write tests for and try to figure out, but everything was already packed up, backed up, ready to ship. And you were just trying to stop, like, a meteorite that was about to hit, and you were like—I don’t even have control. I detect something, I send it before a production release, and I have this box that I detected. I’m sending them somewhere, probably the developers were even gone.

All those situations that we experienced before on any testing area that now that everything is happening continuously, fast, we have feedback, we have—we don’t own only the development side, the testing side, the operations side, but we own all the loop, we’re involved, and hopefully, we have communication with each one of those areas. And I would say not communication that the lines that divided us are blurry now. We are just multi-capped T-shaped people that can jump into, “Hey, now, this is a deal with development on Ops, I understand that this is happening, we have logs. We have different release methodologies.”

I think DevOps and the openness that is happening as well as, from my perspective performance that I have some tools now that allow me to see into the code, to monitor it, telemetry—all these fun things that we can work with have allowed us to evolve and to integrate as, it was very well said, from clear requirements, clearly knowing what is needed. If we detect something wherever, not only before releasing to production, but now it’s in Ops, there’s a bug that was not caught and we catch it right away, we roll back or we release, quickly, a fix—those things were forced, given those things. Or I don’t know if forced—again, I don’t know if it’s a chicken and an egg situation, what came first. If now we have more visibility and capabilities, we start to release more often and communicate development and operations.

But all this synergy has to happen and is needed for us to truly embrace DevOps, have quality in it, and keep releasing often with quality.

Ashley: Alan, I have a question I’d like to pose to the team about DevOps. We don’t do DevOps just for DevOps’ sake, right, there’s usually some drivers behind it. And when you think about the conditions we’re in today, economic conditions, changing markets, rebuilding our supply chains, contactless services—there’s a great example—in days and a few weeks, businesses had to respond. Point being, one of the things businesses are looking for from their technology, from their software, is agility—the ability respond quickly, experiment, go after opportunity, defensive, whatever it might be.

From a testing standpoint, how do you think that’s changed how you respond and how you do the work that you do, knowing that there are situations you have to react fast, and how has that changed how you work? Any thoughts on that?

Melendez: Well—sorry, go on. I have played too much today. [Laughter]

Pugh: Well, let me just give an example of a current situation. There is one place that I was consulting with—their manual testing effort took 5,000 hours to release a product. You’re not gonna be very agile if it takes you 5,000 hours, that’s 50 testers working over a month, and I don’t even know what they did, if they found a defect, whether they restarted again.

So, the whole concept in being agile is, you need, as we talked about all the automation around everything, because if I’m gonna make a little change to something, I wanna make sure I haven’t broken anything else. So, I need the automation that matches exactly what the requirements are so that, then, I can run it through the pipeline. If I don’t have the automation, I have to pause for three days for manual testing, I’m not gonna get things out pretty quickly.

Fisher: And to build onto that point with BDD, you can see so clearly where you have a failure in your test. When I joined my current organization, they had just really started with automation, and there wasn’t a lot of direction there—which is fine, that is not any kind of a criticism. But those tests got so complicated and only the people who were writing the tests knew what was happening in them and where those failures were and troubleshooting a failure could take just as long as troubleshooting the code.

So, taking something like BDD, we’re gonna build on the agility, we wanna make it open to the team, the manual tester can find out where that failure was, a developer can find out where that failure was, and then we can very easily test to see—test the test to find out if it was a problem with the test or if it’s a problem in the code and then send it to the appropriate place to get it addressed. I think that that piece has just been a huge improvement for what’s been happening.

Melendez: And I want to add, as well, from what you mentioned, what’s changed from the old waterfall days to how we are working now with the new trends and everything—I think the first problem that I personally experienced, and I fought against this agile B stuff that started to happen. “Why don’t you allow me to still do automated, big-ass load tests every, before every release? How am I going to be able to keep up with this if I have constant releases every two weeks?”

And here, I think the biggest thing is a change of mindset, because many of us—and again, I said this myself, I was indoctrinated into how the waterfall days and silo-based release cycles happened. And I cannot even call it cycles, it was just like the big bang release and that was it and you’re out. And you used to think that way. You had to do big cycles of automation that you could execute that took hundreds and hundreds of hours, people involved. And now you need to think about, how can I tackle this now that it’s supposed to happen so often?

But a big change is that it’s not so massive any more. We are not doing a whole full release that we need to do multiple regressions here and there and everywhere to make sure. Now we can just say, “Hey, this sprint, I’m just gonna release this tiny, little, little feature that we can just quickly test. Something breaks, something happens, we are sure that our team, ourselves included, can switch direction, detect where is the issue, and implement the fix. And probably tomorrow, if we are super efficient, super quick, and doing things differently, we can recover from it.

As you mentioned earlier, when there’s tons of tests that you need to execute and there’s an error that you detect, it’s there. There’s a bug, but we want to release every week, every two sprints—sometimes even modern mindsets can be, “Well, let it go to production. It’s gonna pester us for a day or two, but we know that we can quickly apply a fix, release it. Let it pass. It will cause a tiny bit of problems, and even change the course, change the direction if you need it, like, hey, that’s a feature we don’t need, we can take it off.”

So, all that mindset changing on how you approach testing with this has to change. It’s very different, and I work with lots of customers where I jump in and I’m like, “Yeah, why are you still doing these things that are a decade, two decades old where all this DevOps, continuous, agile is already happening and you’re not even trying to do agile, but you have this big bang release, huge pieces of functionality that you want to push, that’s not gonna help you. And you need to be paying attention constantly and enable that you can recover. Your testing needs to be aimed at enabling that tiny functionality tests everywhere and just relevant to what you are doing.” It’s a complete switch of mind of what was testing before and what it is today.

Ashley: Fascinating.

Shimel: Nice. Go ahead, Mitchell, I’m sorry.

Ashley: Just that idea of being able to release software quickly, right, instead of 6 months or 12 months—that’s a mind boggling shift, right there. Sorry, Alan, you were taking—

Shimel: No, no, I’m actually saying I think we’re about out of time. I was gonna ask us to wrap up—I was gonna wrap up, but you know what, Mitchell, I gave you the last word, that’s great.

Hey, guys, as I said earlier on, the time does go really quick. We’re out of time for this episode. I’d love to have you all back, though, and maybe we can continue this conversation. You know, I find it fascinating as someone who doesn’t come from the testing world, right, that testing in its infinite variety has been so profoundly impacted by DevOps and agile and these technologies. But rather than being left behind, it’s been accelerated, right? Through automation and speed and, quite frankly, more focus, in many ways, DevOps has been one of the best things that’s ever happened to testing, right?

And that’s the thought I want to leave you with out there or DevOps Unbound. This is Alan Shimel, we’ll see you on our next show. We have a great roundtable coming up, by the way. Check DevOpsUnbound.com for all the latest. Thanks, everyone. Thank you to our panel. Mitchell, as always, thank you. Have a great day.

Pugh: Bye.

Fisher: Thank you.

Melendez: Gracias.

March 30, 2021
How Shift-Left Testing Improves DevOps Efficiency and Software Quality

Shift-left testing is the key to deliver quality software at speed. Developers shift testing left to shorten long test cycles, prevent defects and mitigate risk earlier in the development cycle, rather than in later stages, to avoid rework and delays.

In this interview, IBM Z DevOps Testing Lead Nathan Cassata talks about shift-left testing, the benefits of shifting mainframe application testing to the left and how IBM Z DevOps testing solutions help accelerate enterprise cloud-native development.

Nathan is also one of the distinguished speakers presenting at Open Mainframe Virtual Summit. During his session, he will explain how to shift left testing for your traditional z/OS applications.

To learn more about how testing is integrated into a developer pipeline and how to create real unit tests, attend Nathan’s session at Open Mainframe Virtual Summit on Feb. 11. To check out the full agenda or to register for free, visit the Open Mainframe website.

February 8, 2021
Integrating Security in the Development Process With DevSecOps

Occasionally, it’s worthwhile to reflect on how we develop and deliver software during times of rapid change and significant disruption. In those moments of reflection, we learn from the exciting trends and changes that are taking place. Recently I had the opportunity to commiserate with two technical thought leaders at Perforce: Stuart Foster, product manager, and Steve Howard, static code analysis evangelist.

The disruption caused by COVID-19 goes much deeper than just the rapid shift to work-from-home. Development teams are being asked to keep up developer velocity, while the organization as a whole accelerates digital transformation projects, shifting priorities and emphasizing investments needed to successfully navigate difficult and uncertain economic and market conditions.

In our conversation, we note how investments already made in shifting to the cloud, remote and cloud-based development, automated testing and pipeline automation and security tools are helping organizations today. We also explore how current conditions influence product roadmaps, causing software teams to take a holistic look at their development processes and tools, and organizations to rethink how to take their software delivery capability to the next level.

Stuart and Steve share their software best practices on collaboration, automation, security, standardization and more.

Explore our conversation in more depth by viewing the video and transcript below.

Transcript

Mitch Ashley: Well, I’m happy to be joined by Stuart Foster, who’s product manager with Perforce, and Steve Howard, static code analysis evangelist, and I think a lot of other things, too—does a lot with Perforce. So, welcome, gentlemen—good to be talking to you today.

Stuart Foster: Thanks, Mitch.

Steve Howard: Hello.

Ashley: You know, we’re talking about—you know, Perforce does a lot of things, and I think we’re just kind of focusing on what maybe some of the current conditions, current best practices are. There’s a lot happening right now in everybody’s world—a lot of change, a lot of disruption.

But before we get to that, could I have you both introduce yourselves? Maybe Stuart, would you start first?

Foster: Yeah. Hi, everybody, I’m Stuart Foster, I’m Perforce’s product manager for our static code analysis products.

Ashley: Great—Steve?

Howard: Yeah, and I’m—well, I have, as you say, many jobs and roles within the organization, but primarily focusing on the evangelism around using static code analysis for both of our products, Klocwork and QAC, here at Perforce, yeah.

Ashley: If you’re working with developers, you gotta be doing evangelism, right?

Howard: Yeah. [Laughter]

Ashley: They expect you to give them support, and be talking to them, in a different kind of way. Tell us—would one of you tell us a little bit about Perforce and what all you all do?

Howard: Yeah, sure—well, I’ll take that. So, Perforce is an organization, obviously as, you know, originally comes out of the version control software. But really, it’s about developer tools and how we can help make the developer’s life easier in the day to day tasks. So, whether that be trying to prove compliance to standards or just trying to get the stuff out the door in the first place—anything that we can do to help make that process smooth and reliable is kind of where we are. And, you know, these days, that has a bit of a—a buzzword, if you will, around the DevOps tooling ideas, but essentially, it’s developer tools and making their life more comfortable.

Ashley: That’s great, because you all fit into a number of places in the sort of tool chain, pipeline, workflow process across the development life cycle.

Howard: That’s right, yeah.

Ashley: So, that’s awesome. Well, let’s get into this. I’m really curious about what you’re finding in market, and one of the things we like to do when I’m talking with experts like yourself is, let me just try to share some of the things that we’ve experienced within the past, but also kinda recently that it’s happening, because we’re all dealing with economic change, disruption, COVID, work from home, all of those kind of things, but you know, we’ve done some research at my company and there’s a lot—of course, a great, great emphasis on acceleration of digital strategies and, of course, that puts a lot of pressure and emphasis and people are betting money on software being the answer to helping them implement their strategies.

Are you seeing the same kind of thing, that acceleration of digital transformation and software—elevation of software projects?

Foster: Yeah. We’re seeing that across the board, even in our own company here. You know, a lot of us have transitioned to work from home, and then you have to start thinking about the considerations for that, right?

And what we’re finding ourselves and working with our customers is, a lot of people are shifting more technological tools, right, to be able to better produce workflows and code reviews, how to—working from home, security, right? How secure are your remote development work stations? How is security being built into your software and your workflow? So, tools are being used across the board for that. Just being able to rely on a piece of software versus in-office workflow has become more and more important. And amongst all that, then there’s the goal to, you know, keep development velocity up, which could be a challenge when working from home, right? So, we’re seeing a lot more DevOps automation of, you know , a lot of workflow process and things like that.

Ashley: Yeah, there’s no more popping over the edge of the cube or to the table next to you, right? Somebody sitting there at the café, in the office, or down the street. It’s a fully digital world, right? You’re not sitting with people very often.

Foster: Yeah, we’re relying on Slack, we’re relying on Slack calls, Zoom videos, everything to just stay in touch and work and collaborate with one another, yeah.

Ashley: Steve—yeah.

Howard: But I think that’s where—yeah, so, I was gonna say, I think that’s where those centralized processes that we’ve kind of all been working towards, luckily, for the last few years, are really gonna pay off. Because how do we not have that? If you imagine how things were maybe five years ago, it would’ve been a much bigger transition to get to where we are now, whereas it was almost like most of our organization was kind of ready to go. We were already there, just—you could call foresightful, but really, it was a little bit of chance, I think. Yeah, those processes have held things together remarkably well, and we’re very lucky for it.

Ashley: You know, nobody would wish for something like COVID or a pandemic on any of us, but if you look back, you know, I think back when we were virtualizing things and developers were starting to be able to just create a fully self-contained environment to develop on their laptop—I saw people working on planes, right? First time I saw somebody writing code on a plane—that’s awesome.

But so many things, and then of course, adding the cloud and all the services and software and tool chains and pipelines that we can create now, it—I mean, I think we’re lucky, if you were gonna have something like this happen, now is the time where we can actually continue to be productive, and put more emphasis or pressure on ourselves to deliver software. Do you agree?

Howard: Yeah.

Foster: Yeah, absolutely.

Howard: Yeah—sorry, Stuart.

Foster: [Laughter] We agreed at the same time, that was great.

Ashley: [Laughter]

Foster: But you’re right, I think, like, some of it—you know, it’s a little bit accidental, but as you mentioned, setting up the development workflow to be able to do it on a laptop, people are shifting their hardware, they’re removing their hardware and going to the cloud, right? So, it’s kind of been like an expansion and a reduction, all—just naturally, but also, you know, you think about DevOps and there’s also, you know, cost savings to be had by not having to have this footprints, not having to worry about these things.

So, this could be, in some ways, pretty revolutionary or changing for development costs for a lot of companies.

Ashley: Steve, you were gonna jump in there—anything to add?

Howard: Yeah, no, exactly the same. I think the fact is that we—you know, a lot of that infrastructure is being built up and the idea of being able to run our development pipelines kind of in a cloud environment or whatever and having that all ready to go just meant that we didn’t have to pick up ginormous work stations. I remember back in my days of developing in aviation software, we used to literally have, you know, Unix boxes stacked up on the side somewhere that we had to go and log in on each one and run our testing for each single one. It’s just unthinkable. It would never have been possible.

Ashley: That would be insane today. You would never. I mean—

Howard: Yeah.

Ashley: – if you have a choice, you would never do that, right? [Laughter]

Howard: Well, exactly, and the very fact that, you know, 10 years ago, we still had to do that, how could we have transitioned? It wouldn’t have been possible. And so, it really is testament to how far we’ve really come in 10, 15 years to see where we are with our testing processes, the fact that we can do, you know, 99 percent of it, if not all of it can be done completely with no connection to the physical space.

Ashley: Mm-hmm.

Howard: It’s all—and that can even mean, you know, in the cloud completely, so it’s not even within our organization. It’s securely being done off site somewhere in the ether. [Laughter]

Ashley: Yeah.

Howard: And that process is still completely manageable, it’s something we can all see as developers, everyone can get to see where we currently are with that process, you know, what the latest test run gave us across the whole—

Ashley: I’m really curious—yeah, I’m really curious what your experience in market is. Do you see more people who are kinda doubling down and getting even more serious about their DevOps implementation and taking it to the next level, or do you see more people coming to you that maybe are relatively new to DevOps and are looking for tooling and looking for ways of helping them, you know, not just get started and learn all the problems themselves, experience them themselves, but looking from the experience of a software provider like yourself, or is it a little bit of both?

Howard: I’d say a little bit of both. I mean, obviously initially, there was such a lot to do already and everyone kind of had no time to really think—understandably, but probably within, you know, since the end of the summer, maybe, things have started to come back and people are actually going, “Well, hang on a minute. We’ve got a process, but now, how can we actually make this a lot easier for ourselves?” and looking to improve that.

And I think, you know, for a lot of the organizations where, perhaps, doing things off site would never have even been considered. And I think this was to Stuart’s point—that kind of, that’s been forced upon people and it’s almost gonna change the way we develop forever now as a result of that, because we have now, we’ve done it, we’ve experienced that it’s possible, and it’s safe and it’s secure.

How can we now make that better and make our lives easier by not having that massive footprint that Stuart was mentioning? We’ve actually got something that’s very, very lean, it’s in the cloud, we’ve got machines when we need them, but not when we don’t need them, and how can we really make that process as efficient as possible from the results side, you know, making sure we’re doing what we’re supposed to be doing as quickly as possible, but also actually trying to reduce our footprint on process and power, for example? You know, the development time that’s involved in that, how can we make that as efficient as possible?

Foster: Yeah, and I also think what we’re seeing, too, is, you know, our existing customers, they’re doubling down on their tools, you know, because they need to rely on something, right? And that’s also, what it’s done is also, I think, is changing some of the roadmap items that all products are looking at, right? There’s a whole new—not a whole new paradigm, but you know, there’s a little bit more seriousness being taken in terms of some of these features and functionalities of being able to work remote, or the requirements for more cloud usage, right? We’re talking ease of use, we’re talking about things that are, you know, related to workflows to make working in these conditions—not just in these conditions. Assuming you get back to the office, just more efficient as well.

Ashley: Mm-hmm.

Foster: So, we’re seeing that at the same time, too. And then, you know, from the perspective of new customers coming in, they’re almost expecting these things to already be there, right? Because they’ve run up to these problems as COVID has created and they’re trying to find the solutions that are gonna be kinda turnkey out of the box.

And also, I’ve done, I’ve been talking with various analysts related to our product, but it seems as though the analyst community itself is, they’re seeing the requirement for, you know, ease of use in DevOps becoming more important as well at the same time.

Ashley: Mm-hmm. I think that may be also, in part, the expansion of DevOps into the organization. They may have had a core group of teams that were really taking the challenge of DevOps and implementing it and now, how does the rest of the organization do the same thing, but accelerate their adoption of it?

Foster: Yeah, and I think also, a lot of development groups within a single company—you know, you may have a security team, you may have a development team, you may have a LiveOps team—they’re starting to recognize that there are choke points in the development processes.

Ashley: Mm-hmm.

Foster: And so, how do you unblock that? How do you make it more easy? How do you offload some of that choke point onto a broader group, right? A lot of the idea of DevOps, DevSecOps, whatever you wanna call it, right, is that developers are owning the code that they produce in more broad ways. That’s writing unit tests, that’s making sure they do the QA on it, right? That’s when they’re, you know, they have to implement the security requirements then and there, right? They need to ensure the traceability is done, right? They have to—they’re taking that into their hands. It’s becoming not so much that there’s a DevOps group that magically handles all this infrastructure, it’s becoming part of each developer’s responsibility as well, right?

Ashley: Hmm. Interesting. You know, I know, as you scale things out and you adopt new tools, tools do a lot of things for you, but there’s oftentimes, there’s approaches that you need to adapt into your tools, things like what are our coding standards, what are our testing, what level of testing do we do? How do we break it into chunks? When do we do security testing of our code?

And that’s part of that learning curve to is, how do you adapt a tool set to both what your requirements are, but also, you would like it to kinda take you to the next level. What are some of the things that really can help people kind of propel their expertise in the organization’s ability to deliver software more effectively?

Howard: Yeah, that’s a really good point. So, one of the things, you mentioned coding standards there, but obviously, you know, those can come from multiple sources, they can be something that’s actually being required by a standard, for example, if it’s in a safety critical, security critical world. It could be something that’s actually demanded by the organization itself—they have their own rules or there’s certain things in any particular project that, if you do things in a certain way would make, you know, it would be dangerous or insecure or something. So, they have some controls, as well, of that.

And what we do see is, the more that we can get that information out to our developers, for example, the more value that has as well. So, yes, we can help enforce coding standards, for example, with our Static Analysis tools. But that’s only half the story. Enforcement is one thing; actually getting the developers to understand what the problems are as they’re creating them and to have easy access to the help and the quick feedback of those problems makes it much more powerful, because it gives them, (a) you know, all the resources they need to deal with the problem quickly and to not perhaps find it a challenge to have to deal with, but also, they learn.

And so, the next piece of code they write doesn’t generally have the problem. So, there’s a developer learning element there as well, which we can control centrally, and then spread out to all of these remote workers working—you know, without that team collaboration that we used to have, we’re still able to do the same things. So, that’s become something that’s really focused people’s minds on the capability there in terms of what that means. And as I say, that applies to security standards that are well known or requirements for quality that we have to work to for a security or safety standard—

Ashley: Mm-hmm.

Howard: – but also the company’s own rules.

Foster: Yeah, and I think, you know, outside of just simply the tools, development teams are, you know, taking a step back and re-evaluating or thinking more about what their software development process is, holistically, right? You know, Steve mentioned there some of the tools, best practices, but you know, you also have to think what kind of, you know, software development methodologies might you be using. You know, sometimes it doesn’t matter whether it’s agile, waterfall, whether you’re using Scrum or Kanban, you know, I think it’s giving people a moment to think about what’s the most important kind of Dev process that is gonna fit this team or how you have to work now, how it fits the businesses.

And then implementing processes to ensure the organization, communication, and what systems need to be put into place to help prevent any problems that arise during development and kinda identifying and understanding that. Then you’re picking the tools, you’re building your development workflow, whether it’s in environments, it’s setting up DevOps pipelines to ensure that everything’s followed correctly. It’s the tools, but it’s the process, and that’s changed, too.

Ashley: That’s interesting. You know, we’ve done some research about the influence that developers have had on the organization, too. Even outside of development teams, you’re seeing DevOps and agile kind of principles, even though they aren’t calling it that—daily standup meetings, shorter intervals of deliverables, more kinda asynchronous communication tools and collaboration kind of tools, which really makes it easier, I think, for the development teams, the product teams—you’re a product manager yourself, Stuart.

Foster: [Laughter]

Ashley: And beyond—operations, security—everyone is kind of forced to work a little bit differently, but also can benefit from what each other knows about how to do that more effectively. And we seem like we’re at this sort of serendipitous time of being able to leverage that, which fits right into the tooling approaches that we can take.

Foster: Yeah, and you know, when you look at it, like, it always just ends up communication. Whatever the communication is, whether that’s the workflow, whether that’s in-person, whether that’s meeting, whether that’s documentation, it’s all about communication, and that’s true not for just development, but for business, right? Developers working, you know, with other groups—marketing, sales—everything, it’s all communication. It just keeps business running. And that’s kinda the funny kinda perspective that may not necessarily be outright visible by simply saying, “It’s communication.”

Howard: What’s interesting is, that’s also kinda mirrored in the tooling, because a lot of the work we’ve been doing recently is about how to make sure the tools have all got open interfaces to communicate with the next thing in the pipeline or to feed that data back up to some dashboard that’s gonna be shared with the rest of the team.

So, yeah, it’s kind of an interesting point, but that also applies with the tooling that we’re trying to build around that.

Foster: Mm-hmm.

Ashley: Mm-hmm. A really great point. So, to give you fair warning, I’m gonna ask you your three recommendations on best practices in the current climate, so be thinking about that.

I was thinking about your comment, Steve, about being able to better integrate tools together and then taking the data from the tool sets, the pipeline of the process that we’re going through and the pipeline of work—there’s so much value in that data that sometimes ends up as exhausts or on the cutting room floor, right, to use two different metaphors. But that information could be extremely valuable, putting it into dashboards, feeding it into continuous improvement processes, shortening the window for getting feedback for, “Here’s a security vulnerability or a coding standard” or some kind of an error that you can fix in minutes instead of days and weeks and sometimes months, right? The security audit at the end of the release is much easier, because you’ve already gone through so much to it. Those can all be things that can help us.

So, I gave you a little bit of time there to think about your three best practices. I did my best job of stalling that I could. So, who would like to go first? What would you recommend people consider today?

Foster: Yeah, I’ll go. Just, I want to reinforce that message of, you know, figuring out how you’re going to work as a development team, develop that communication, right? And that’s using your tools.

I think in the world of remote work from home, having kind of a common language—again, back to communication—of how, you know, each developer’s system is set up. So, common environments, they should be using as close to the same setups as possible so that, you know, when one of you is running into an issue, it’s easier to get that information across, right? You’re not gonna have to install something obscure because somebody’s using something obscure. You should all be trying to talk and use the same tools so you’re all speaking the same language. That allows you to have that velocity that you could otherwise lose.

And in this world, too, security hasn’t been more important than ever—security is more important than ever. So, think about securing your Dev processes, developing secure software, it’s gonna be very important. And those are kinda my three.

Ashley: Okay, great. All good ones. Steve?

Howard: Yeah, I’d answer—so, my three would be, first of all, to automate; take the process away from the day to day effort of the development teams and organizations around them. So, yeah, that’s the first one.

Report—so, that’s the communication piece, I guess, but get the information to the right people at the right time and hopefully in the right sort of time window.

And then shift left. So, then, we make it more efficient. Then we look to where we can try to remove the fat in the process and take away the time that was wasted and try and move it to the right position in that process. So, yeah, those three, in that order. [Laughter]

Ashley: Great. Excellent, excellent. I’m curious what you both think will be some of the things that will stay with us, the things that’ll last past a forced work at home, forced work remote. Maybe people are returning to the office, maybe they aren’t. What do you think are some of the things that will stick with us? Because it wasn’t like we were doing this for two weeks and then we’d just go back to the office—this is an extended period, we’re developing new habits, we’re changing how we’re working. Why go back, sometimes, right? What are your thoughts?

Foster: I think you’re gonna see a lot of these workflows stay around. I think that the reliance on the numerous tools to keep velocity up and reduce and automate are just gonna stick around. You know, you don’t want to be wasting time on trying to get something to work. It’s easier to just simply automate it and get it out of your hands so that, you know, at least from a development perspective, you can just focus on what’s important, which is, you know, coding, the feature, getting the product out the door, right?

Those are gonna stick around, and I think the tools, over the course of time, are just gonna keep reinforcing and strengthening that kind of automation pipeline.

Ashley: Good.

Howard: Yeah, no, and I think, from my perspective, just to add the one small piece there is, the cloudification, essentially, of the processes. So, I think, you know, people will stop having the premises to host great, big server racks in, and that’s just all gonna move off site, off premise, outside of the organization, for sure, yeah.

Ashley: Yeah. To kinda add what you both have said, which I think supports your thoughts on this is—you know, as we change to whatever we’re gonna go back to, I don’t think it’s gonna be back to what we were doing. But it’s gonna be a hybrid, right? I can’t imagine everyone shifting back to a different way. It’s—some people are still, maybe even more percentage of our teams are working remotely, never going back into an office, but we may have some that do.

We’ve broadened the reach across the globe of people that we’re working with, the product and vendor, tool vendors that we’re working with. Those things aren’t gonna change, right?

Howard: No. No.

Ashley: We’re gonna stay doing that. If we’ve found something that works better, you’re gonna stick with it, right? So, you’re gonna stick with—you’re gonna stay in the cloud, push more in the cloud, right? And leverage some of the automation, the processes that you talked about, Stuart.

Well, this has been really fascinating. Would love to have you both back, have you guys back and we can talk some more about—I’m gonna delve into static code analysis evangelist, your role, Steve, get into some of that, which I know you all have some of your products do and to security and to some other areas within the software pipeline, but this has been really great. I’ve really enjoyed talking with you both.

Howard: Yeah, it’s been a pleasure.

Foster: Great. Thanks, Mitch.

Howard: Bye bye.

Ashley: You bet. We’ll see you soon.

December 2, 2020
Aim to Be a Top 12% Global Testing Team
We know that you’re thinking, “Really? Another fluff post about the top people in my profession and what I need to do to be just like them?” Listen, we hear you, but this is not some fluff piece with vague ideas of what might (or might not) work.

In case you missed it, on May 27th, the 2019 State of Automated Visual Testing was released. Based on independent research sourced from over 350 testing teams around the world, we learned 12% of you are getting much better results than the other 88%. We’re talking four times more successful as measured by the things you (and your boss) really care about–test coverage, release velocity, application quality, overall R&D teamwork and cold hard cash. This is not our opinion. It’s not subjective. It’s objective data and information. Data and information that came from you and your peers.

In other words, we’re not asking you to take our word for it; we’re asking you to take your word for it.

What Separates the Top 12% from the Other 88%?

Digital Transformation. Two words that have been baked into our world over the past 20 years. To such an extent it may be fair to call it fluff? Fluff or not, IDC forecasts worldwide spending on technologies and services that enable digital transformation will reach $1.97 trillion in 2022, per the (IDC) Worldwide Semiannual Digital Transformation Spending Guide.

That’s a lot of fluff! When your bosses’ boss is spending that kind of cash, it’s always worth paying attention. It could be good for your career. As it turns out, 12% of the world’s testing teams did pay attention, and it’s their approach to managing the challenges of digital transformation that have set them apart.

The Testing World’s Digital Divide – Digital Transformation Quantified.

You can quantify it for yourself with some simple math and see how you compare to other R&D and testing teams around the world. Got that calculator ready? Here we go…

How many applications do you have in production today? (Don’t forget those native mobile apps, they really add up).

How many pages or screens do you have in production on average for each of these applications? (Single page applications can be tricky but give it your best guess).

How many viewport breaks do you support? (The market average is six if you are not sure).

How many human languages do you support? (We’re talking about localization here–English, German, Spanish, Chinese, Japanese–not coding languages in case you’re wondering).

Now, multiply those four numbers together. Congrats! You have just quantified the digital footprint of your business. People can write about digital transformation all they want, but you’ve just transformed all that fluff into something quite real. It’s probably a big number. Over 90,000 for a typical company and over 624,000 for the largest 30% of companies in the world. Here’s the kicker. You and your R&D team are responsible for managing the visual and functional quality of every single one of those pages and screens.

When functional test automation first emerged way back in 1989 with the launch of Mercury Interactive XRunner, it was a much different time. Browsers didn’t even exist. Since then, the browser wars have come and gone, and standards are now in place. Applications have grown far more complex with native mobile, dynamic content, single page applications (SPA) and responsive design now an everyday reality. Digital transformation for any size company is officially past tense. We’re not transforming, we’re transformed. And now you have to deal with it, but how?

What Defines a Top 12% Global Testing Team?

Top 12% teams have overcome the massive challenge posed by digital transformation. Like any technical challenge we have ever faced, it started with someone on the team who felt the pain and set out to solve for it. And when they did, good things happened:
- Test coverage increased by 60%
- Release velocity became 2.8x faster (even though coverage increased 60%)
- Visual and functional quality improved 3x with far fewer escape bugs (even though they release 2.8x times faster)
- R&D teams are 4x more satisfied with their visual and functional quality outcome
All of this despite the 12% of testing teams were managing applications 2.2x larger than the other 88%. In all likelihood, they felt the pain before most of us, and have now led the way for the rest of us. We just need to follow.

Today, successful continuous management of application visual quality creates competitive advantage. Business leaders know this and are paying attention. As a result, testers are in a better position than ever to be heroes respected by their R&D teams for driving huge value for the companies they work for.

— James Lamberti
August 5, 2019
The Human Value in Automation-Driven DevOps Testing
Enterprises are embracing DevOps underpinned by automation to ensure the quality of software across all digital touchpoints. In fact, automation has become the crux of continuous integration and delivery, the vital DevOps testing goals. Without automation, DevOps would become a distant dream, and businesses would struggle to keep pace with seamless delivery and integration. However, DevOps entails more than the implementation of the latest automation tool and demands a more holistic approach. It does so by bringing together processes, people and technologies under a shared vision. As defined by Microsoft, DevOps is the coalition of products, people and processes to facilitate the seamless delivery of value to the end customers.

DevOps is considered to be the latest advancement of the agile movement. It includes a greater number of stakeholders in the process of delivering value by linking the silo-based teams. It may not be the best practice, but it’s a discipline of people-centric management. Continuous delivery is the technical foundation of DevOps automation, which extends to the agile foundation as well. The close collaboration between the operations and development teams is known as DevOps, which is used to automate the delivery pipeline.

Role of Automation in DevOps to Increase Productivity

Since digital transformation has taken over the current business model, more and more companies are embracing the DevOps practice to drive the automated deployment of software. We are shifting toward an organizational culture in which development and testing take place simultaneously, quickly and more often. This helps maintain consistently high ratings on the quality of software with faster turnaround time. The main agenda of DevOps testing is to derive quality outcomes all along the software development life cycle (SDLC) and beyond. So, with the help of DevOps culture, a new code of development, testing and delivery is introduced in the overall SDLC.

The automation in DevOps empowers companies to:
- Enhance the reliability, re-usability and quality of all components.
- Save costs of upgrades and maintenance.
- Improve the rate of success for standardized processes to ensure the rapid delivery of applications.
Automate Everything — Is It Even Possible?

With the help of the DevOps approach, developers automate and accommodate operational activities, from the source code commit to operational productivity. The primary goal of DevOps automation is to remove bottlenecks in the pipeline by implementing automation in a single level of development. However, DevOps is not a fully automated process and is not operational without human intervention. It is the DevOps testing specialists who are responsible for the process to run flawlessly. Governance at every stage of DevOps testing is still indispensable in various processes undergoing automation. Importantly, human administration is still crucial in handling exceptions to complete the tasks of automation.

The Strategic Role of Humans in DevOps to Ensure a Continuous Pipeline

Remove the Boundaries of Roles

The primary feature of DevOps automation culture is about an enhanced collaboration between the operations and development teams. It has led to a cultural transformation at the organization’s level and inside the teams as well, to encourage continuous delivery. Since shared collaboration is an important aspect of the DevOps culture, it strengthens the collaboration within teams. Before the DevOps approach, the perceived lack of interest by the development team regarding maintenance and operations used to affect DevOps testing and business goals. However, this scenario has since changed due to an organizational shift toward automation and DevOps testing.

The gap between the silos of operations and development no longer exists with the arrival of DevOps. It has removed the boundaries of several organizational roles. Teams are required to focus on the shared goals of DevOps to enhance the end user experience, so the team members need to rethink every investment and decision around this shared goal, which requires human intervention. Automation cannot substitute the decision-making process of humans, for the latter has become vital across the organizational ecosystem as, and when, needed.

Omit One-Size-Fits-All Approach in DevOps Testing

Automation cannot be viewed as the eliminator of roles. The organization still requires expert knowledge and skill set inside your team. The focus should be on developing agile and cross-functional teams that are recognized for removing long-established stumbling blocks between the operations and development teams, and for facilitating collaboration. Developing efficient and functional teams would ensure the delivery of superior quality products.

Continuous Monitoring, Testing and Improvement

One of the key agendas of DevOps testing is to carry out a deeper level of monitoring. Since rigorous monitoring is needed at every layer of the testing process, from integration to infrastructure and transactions to components, it has become a crucial part of the road map of the DevOps approach. In this digitally connected world, monitoring and testing are no longer distinguishable. Continuous monitoring by testers offers a positive response to the user experience while eradicating pre- and post-production obstacles.

Conclusion

With the impetus in the growth of DevOps and agile practices in the past few years, testing has taken a new structure and form. It has evolved into a crucial discipline, and it’s not restricted to a phase of the development cycle anymore. It has become an essential part of the operational activities of production. So, is the role of a tester transformed radically from only a checkpoint to a crucial active participant in the DevOps testing? Teams need to cooperate based on the shared vision of enhancing augmented user experience. We need to understand the fact that automation is not an elixir to deliver on DevOps goals, but merely a part of the DevOps strategy.

— Komal Lopez
June 12, 2019