ProjectDiscovery has made available an autonomous security testing platform that leverages an open source artificial intelligence (AI) testing framework to detect and validate vulnerabilities at a lower total cost.
Company CEO Rishi Sharma said version 1.0 of the Neo platform is also available via a cloud service that makes it possible for DevSecOps teams to run tests and conduct investigations using a consumption-based pricing model.
Additionally, Neo 1.0 adds integrations with DevOps tools and platforms such as GitHub, Jira, Confluence, Slack, Linear, application programming interfaces (APIs), webhooks, and the Model Context Protocol (MCP).
Based on an open source project, ProjectDiscovery, in addition to using Neo to detect, validate and route vulnerabilities to the responsible developer, also provides access to Nuclei, an open-source vulnerability scanner driven by customizable YAML templates to detect misconfigurations and exploits. Also included are tools such as Subfinder for discovering subdomains, an httpx tool to probe HTTP, a web crawling tool dubbed Katana, and a port scanning tool known as Naabu.
Collectively these components provide the foundation for an open source ProjectDiscovery ecosystem that already includes more than 100,000 practitioners, making it less costly to discover and remediate vulnerabilities using an autonomous security testing and penetration platform that continuously generates alerts in real time or can be used to run tests at scheduled intervals, said Sharma. Relying on code scanning tools is no longer enough, he added. Instead, there needs to be an entire tool chain in place that not only identifies vulnerabilities but also makes it simpler to identify which vulnerabilities can be reached and actually abused, noted Sharma. Otherwise, DevSecOps teams will soon find themselves overwhelmed by the number of false positives that are generated by AI models, he added.
As artificial intelligence (AI) models advance to the point where discovering vulnerabilities and creating exploits becomes trivial, DevSecOps teams now more than ever need to be able to discover those vulnerabilities before software is deployed in a production environment. The challenge is that, historically, the cost of applying AI to test code has been too high for many organizations to adopt, said Sharma.
In fact, organizations need to realize that application security in the AI era is not just a technical issue but more importantly requires new processes to be defined and embraced that go beyond simply asking an AI model to identify vulnerabilities in source code, he added.
Mitch Ashley, vice president and practice lead for software lifecycle engineering at the Futurum Group, said the truth is there has never been a shortage of vulnerability findings. DevSecOps teams drown in them and burn triage time sorting real threats from noise. Neo validates what it finds, then routes the exploitable issues to the developer who owns the code. The scan-and-fix cycle was built for human code at human speed. AI writes and exploits faster than weekly scans keep up, so testing now has to run where code gets created, he added.
It’s not clear how long it will be before there is a full-blown vulnerability crisis, but it’s apparent DevSecOps teams are running out of time. Technical debt that has been kicked down the proverbial road for years now needs to be rapidly reduced because many more cybercriminals will have access to more advanced models in a matter of weeks. Each DevOps team will then need to determine at what rate to remediate vulnerabilities in existing applications versus opting to replace legacy applications with ones that are, hopefully, more secure.
In the meantime, however, DevSecOps teams, while continuing to hope for the best, would be well-advised to start preparing for the worst-case scenario as soon as possible.
A newly open-sourced Python charting library, called XY, from Reflexoffers an advantage few other charting tools provide: massive scalability.
The library rethinks how to render a set of points on the computer screen, and opens new possibilities for using charts in daily workflows.
Most Python charting libraries today, such as Plotly, Bokeh, or Matplotlib, take a similar approach to creating charts: they serialize every data point and send it to the browser, where it is usually parsed by JavaScript. GPU acceleration can help, though portraying thousands to millions of rows can hog memory, slow interactions and frequently lead to crashes.
The XY library shifts all this heavy data processing to speedy Rust-built libraries, where the rendering job can be matched to the pixel/density surface of the screen. The core then only sends to the screen what the screen can physically display.
The library keeps the chart values in a ColumnStore format. The Rust-compiled libraries compute the level of detail, and transfers the results out as typed binary buffers.
A New Kind of Charting
This approach brings untold benefits, not the least of which is the memory saved by not sending data the browser can’t use.
The time it takes to render a chart stays about the same—around 80 milliseconds—regardless of whether you are plotting 10,000 or 10 million points.
While reducing the number of points, the software preserves the peaks, troughs and overall shape of the data. Even outliers and tail behavior is captured.
Another benefit is that the rendered charts are dynamic. Should the user want to zoom into a region of a chart, XY will recalculate the new area of interest. The user can drill all the way down to individual data points.
Reflex built XY to be a drop-in replacement for Matplotlib workflows. Existing charts in the pyplot format can be rerendered with XY.
The developers admit that XY doesn’t cover some charts rendered by other libraries. This initial release of XY only handles two-dimensional charts. It doesn’t draw polar or 3D charts. Overall, Plotly, Bokeh, and Matplotlib cover more chart types.
A New Horizon for Charts
The ability to quickly render large charts opens a new range of uses previously considered impracticable.
Entire datasets can be rendered without additional sampling. Dashboards can now show live data. The chart.append() method updates an existing chart, meaning dashboards can connect to a live data source without requiring a rebuild.
The library also sets the stage for sharing charts as stand-alone interactive files. A 10-million-point interactive scatter chart exports to 258 KiB of HTML, which is small enough to send as an e-mail attachment. The Plotly equivalent would be 259 MiB, Reflex calculated.
Time series data – Telemetry, market tick data, and log-derived metrics – can also be plotted. The library “renders the full series in one pass with every spike and drop intact, and zoom resolves them without a second query,” the documentation notes.
Installing XY into your Python environment is simple: either `pip install XY` or `uv add XY`, depending on which packager you are using. Import the library in your code and then the numbers can be added through a Python container. NumPy may or may not be used for this task.
Of course, XY can be integrated into Reflex, the company’s open source Python-based Web framework. Any XY chart can be turned into a Reflex component. No JavaScript, iframe, or separate chart service is required.
During a recent UiPath Test Cloud showcase with Tech Field Day, I had a chance to look past the usual “AI can automate testing” message and think about how agents might fit into a real enterprise delivery process. As a Cloud Solutions Architect with a DevOps mindset, I tend to look beyond whether a platform can automate a task and ask what happens once that automation becomes part of an actual operating environment. AI-assisted development is helping teams create and change software faster, but every one of those changes still has to be tested before it can be released with confidence. UiPath described that growing imbalance as a release gap between development and QA. What interested me most was not the idea of putting an agent everywhere, but how agentic reasoning, deterministic automation, and human judgment might work together.
The Release Gap Is More Than a Testing Problem
Software development teams have spent years improving how quickly they build, integrate, and deploy applications. AI is pushing that work even faster, while many testing processes have not evolved at the same pace. During the showcase, UiPath described how development teams are producing changes daily or weekly while QA teams are still dealing with a large amount of manual testing and disconnected automation. The faster development moves, the harder it becomes for testing teams to keep up and confidently say a change is ready to release.
What kept standing out to me was that this is not only a test-execution problem. A lot of the delay lives around the test: reviewing requirements, figuring out what needs to be validated, preparing data, maintaining scripts, investigating failures, creating reports, and waiting for approvals. A team can have an automated regression suite and still rely on a mostly manual process to move from a new requirement to a release decision. At that point, the automated test becomes one fast step inside a much slower system.
This is where my DevOps brain kicked in. One of the questions I asked during the session was whether UiPath could help distinguish between an application defect, a broken test, and an issue with the environment. I pushed to get clarity on that distinction because each one needs a different owner, a different investigation path, and a different response. UiPath explained that its self-healing capabilities can recognize and repair certain automation failures, such as an interface element moving or changing, a blocking overlay, or a timing issue. Failures that fall outside those supported recovery patterns, including potential application defects or broader environment problems, still require further investigation and human judgment.
I also asked whether the platform could help identify which tests were most relevant to a particular application, workflow, or infrastructure change. That is a practical problem I run into in my own work: teams know they need to test something, but the harder question is often, “What exactly should we test to prove this change is safe?” The response from UiPath was that requirements, user stories, existing tests, and supporting documentation can provide context for identifying and refining candidate tests.
Not Agents Everywhere: Finding the Right Boundary
UiPath Test Cloud brings several parts of the testing lifecycle together. Test Manager manages requirements, test cases, executions, results, and traceability, while also serving as an entry point for AI-assisted testing workflows; Studio creates repeatable robotic test automations; Orchestrator schedules and coordinates execution; and Insights provides reporting and analytics. UiPath also emphasized that it can integrate with existing application lifecycle management and CI/CD platforms rather than asking every engineering team to replace the tools they already use.
The distinction between robots and agents was one of the most useful parts of the showcase. In UiPath terminology, a robot follows explicit instructions and produces a more deterministic result, while an agent reasons through the task and adapts as it goes. The presenter explained that robotic execution does not require LLM inference to carry out each predefined test step. Agentic execution may take longer, cost more to run, and produce slightly different behavior from one attempt to the next.
That difference matters more than it may seem at first. A large regression suite usually needs predictable execution, clear assertions, and repeatable outcomes. A UAT scenario, an exploratory workflow, or a frequently changing interface may benefit from an agent that can interpret intent and work through some ambiguity. The UiPath hybrid approach allows teams to keep the stable parts of a test deterministic while inserting ScreenPlay where reasoning or flexibility is actually useful.
The pattern that made the most sense to me was not agents everywhere. Let the agents help where interpretation, investigation, or adaptation adds value. Let deterministic automation handle the work that needs to be fast and repeatable. Then keep people involved where risk, governance, and accountability matter.
That approach also preserves an important role for experienced QA practitioners. The presenters from UiPath did not describe AI as a replacement for the testing mindset. They talked about helping testers move toward test strategy, automation development, agent design, and workflow orchestration. The mechanics may become easier to automate, but someone still must decide whether the system is testing the right things.
From Better Requirements to a Connected Testing Workflow
The UiPath Test Cloud demonstration started in Test Manager before any test was executed. UiPath first evaluated the quality of a business requirement, suggested missing details, generated possible tests, and looked for existing tests that might be redundant or obsolete. The platform can use the requirement along with screenshots, release notes, documentation, existing test cases, and other supporting material as context.
I liked that the workflow started there because weak requirements do not suddenly become strong tests just because AI generated them. If the source material does not explain validation rules, error behavior, session handling, security expectations, or important boundary conditions, the test suite may only automate an incomplete understanding of the application. In the demo, the AI surfaced those gaps for a person to review before the suggested scenarios became actual test assets. The presenter came back several times to the idea of trusting but verifying the output.
Because I regularly help customers evaluate and implement AI platforms, I asked whether an organization could control which models were available inside Test Cloud. UiPath said organizations can govern which approved models are available for supported AI capabilities and can also configure bring-your-own-model options where available. For a regulated organization, that is not just a preference setting. Model access has to align with approved vendors, data-handling rules, and the organization’s broader AI governance policies.
I also asked what the deployment model might look like for a financial institution or another customer that could not accept a standard SaaS implementation. UiPath has its Public Test Cloud, which UiPath operates, and Private Test Cloud, which customers operate through Automation Suite. Requirements such as data residency, isolated environments, or government deployment need to be mapped to the appropriate edition and current availability. The tradeoff is an important one: a self-managed deployment may help address isolation or residency requirements, but the customer also takes on more responsibility for maintenance and platform updates. UiPath also explained that some AI capabilities use platform-based consumption or feature-specific entitlements, while bring-your-own-model arrangements may shift underlying model charges to the customer’s provider agreement.
The larger Test Cloud architecture story came together in UiPath Maestro. UiPath demonstrated a BPMN-style workflow that coordinated environment creation, test-data generation, several testing frameworks, failed-test investigation, proposed fixes, human approval, cleanup, reporting, and notification. Agent Builder supplied specialized agents, while Maestro connected those agents to deterministic automation and people as part of one end-to-end process.
What stood out to me later in the demo was the orchestration layer. Most enterprises already have test runners, pipelines, automation frameworks, and specialized tools. The challenge is not necessarily adding another one; it is coordinating what already exists and preserving context across the full release workflow. From that perspective, Maestro started to look less like another testing feature and more like a possible release-confidence control plane.
The demo was interesting, but the day-two questions were what stayed with me. Can one run be traced across the agents, robots, external frameworks, and human approvals involved? Can teams see which model, prompt, test data, and automation version influenced a result? What happens when an agent times out, an approval stalls, or environment cleanup fails? A workflow can look autonomous on the happy path, but operational maturity shows up in how clearly the platform handles everything that happens outside of it.
Conclusion
My main takeaway from the UiPath Test Cloud showcase is that agentic software testing does not have to mean turning the entire testing process over to AI. The stronger pattern is to let agents assist with interpretation, investigation, and adaptation, while deterministic automation handles repeatable execution and people remain accountable for the decisions that carry real risk. The RPA foundation of UiPath gives it an interesting starting point for that model, while Test Manager, Studio, Agent Builder, and Maestro extend the conversation from individual test scripts into a connected testing lifecycle.
Technology selection, though, is only one part of adopting this kind of platform. When I asked what the customer partnership looked like, UiPath described an implementation approach involving professional services or certified partners, initial integrations and data connections, representative test-case development, and hands-on enablement for customer teams. Low-code tooling may reduce the barrier to building automation, but it does not remove the need for architecture, governance, operating-model decisions, or organizational learning.
I left the showcase thinking less about whether agents will replace traditional testing and more about how deliberately teams will design the boundary between reasoning, automation, and accountability. For teams working through the growing gap between faster software creation and slower release validation, the UiPath Test Cloud showcase offers a useful look at where agents may help—and where determinism and human expertise still belong. If you want to see the demonstrations and hear the full discussion for yourself, you can watch the UiPath Tech Field Day Showcase on the Tech Field Day website.
At 2 a.m., a payment service starts throwing errors. Within minutes, your observability stack fires off 40 alerts: Elevated latency on three services, a spike in 5xx errors, a memory warning on a downstream cache and a handful of dependency timeouts. Somewhere in that noise is the one signal that actually explains what broke. Finding it manually, alert by alert, is how a 5-minute fix turns into a 45-minute outage.
This is the core problem AIOps was built to solve — not more dashboards, not more alerts but correlation. By automatically grouping related signals across metrics, logs and traces, AI-driven correlation collapses an alert storm into a single, prioritized incident with a probable root cause attached.
In this guide, you’ll learn how incident correlation actually works, how to set it up on top of your existing observability stack and how to measure whether it’s actually reducing your mean time to resolution (MTTR). By the end, you’ll have a practical framework for moving your team from manual alert triage to AI-assisted incident response.
Why MTTR Breaks Down Without Correlation
MTTR is typically broken down into four phases: Detection, triage, diagnosis and remediation. Most teams invest heavily in detection — that’s what observability tooling is good at. The bottleneck almost always sits in triage and diagnosis, for a simple reason: One root cause typically produces many alerts, and someone has to manually figure out which alerts belong together before the diagnosis of the actual problem can even begin.
A few patterns show up repeatedly:
Alert storms mask root cause. A single upstream failure can cascade into dozens of downstream alerts, each looking like an independent incident.
Signals live in silos. Metrics, logs and traces often sit in different tools or dashboards, so the engineer on call has to manually stitch together a timeline.
Tribal knowledge bottlenecks diagnosis. The person who ‘just knows’ that a cache-eviction warning usually precedes a checkout failure isn’t always the person on call.
AI-driven correlation directly targets this middle stretch of the incident life cycle — the time between ‘something is wrong’ and ‘here’s what’s actually wrong’ — which is usually where the most time is lost.
Prerequisites
Before you begin, make sure you have:
An existing observability stack producing metrics, logs and traces (StackGen’s ObserveNow or a Prometheus/Grafana/Loki/Jaeger-style setup)
Alerting already configured on key services, even if it’s currently noisy
Admin access to configure integrations and correlation rules
At least one recent incident with alert history you can use to validate correlation output
Estimated time: 30–45 minutes for initial setup, plus 1–2 weeks of tuning based on real incidents.
Step 1: Connect Your Observability Stack
AI-driven correlation is only as good as the data it can see. If your metrics live in one tool, logs in another and traces in a third with no shared identifiers, correlation has nothing to work with.
Start by making sure your telemetry shares common context — service name, environment and ideally a trace ID or request ID that threads through logs and traces.
# Example: Shared labeling convention across metrics, logs and traces
service: checkout-api
environment: production
trace_id: “{{request.trace_id}}”
If you’re using ObserveNow, connecting metrics, logs and traces into a single pane is largely a matter of pointing your existing Prometheus, Loki and Jaeger (or equivalent) instances at it — StackGen integrates with these directly rather than requiring a rip-and-replace.
Expected Result: A unified view where a single service’s metrics, logs and traces are queryable together, ideally with a shared identifier connecting them.
Troubleshooting: If correlation later feels inaccurate, this is almost always the first place to look — inconsistent service-naming or missing trace propagation breaks the AI’s ability to link signals correctly.
Step 2: Configure AI-Driven Correlation Rules
With unified telemetry in place, the next step is telling the correlation engine what ‘related’ should mean for your environment. Most AIOps platforms, including Aiden AI Copilot, combine two approaches:
Topology-Aware Correlation: Using service dependency maps to group alerts from services that call each other
Pattern-Based Correlation: Using historical incident data to learn which alert combinations tend to co-occur
# Example: Correlation policy
correlation_policy:
time_window: 5m
group_by:
– service_dependency
– shared_trace_id
confidence_threshold: 0.75
The time_window matters more than it looks. Set it too short and you’ll miss slower-cascading failures; set it too long and unrelated incidents start getting merged. Five minutes is a reasonable starting point for most microservice architectures — narrow it down if your services are tightly coupled with fast failure propagation, widen it for slower, more decoupled systems.
Expected Result: Incoming alerts are grouped into incidents rather than arriving as isolated tickets.
Troubleshooting: If everything gets grouped into one mega-incident, either your confidence threshold is too low or your time window is too wide. Tighten both and re-test against a known incident.
Step 3: Test Against a Real Incident
Don’t wait for a live outage to find out if your correlation setup works. Pull the alert history from a recent real incident and replay it against your new configuration.
Ask three questions about the output:
Did the correlation engine group the alerts that actually belonged together?
Did it surface a root-cause candidate that matches what you now know actually happened?
Did it leave out anything that should have been included or wrongly include something unrelated?
This is also where Aiden/AIOps automated root cause analysis is worth validating directly. It should be able to point to the earliest anomaly in the correlated group and rank likely causes, not just group alerts together and leave the analysis to you.
Expected Result: The correlated incident matches (or comes close to) the retrospective understanding of the incident, with the root-cause candidate ranked at or near the top.
Step 4: Monitor, Tune and Automate Remediation
Correlation isn’t a one-time setup. It improves while identifying more of your real incident patterns. Over the first few weeks:
Review every correlated incident and confirm or correct the grouping.
Watch for recurring false groupings (two unrelated services that keep getting merged) and adjust topology mappings.
Track false-negative correlations — alerts that should have been grouped but weren’t.
Once correlation accuracy is consistently high, you can layer-in automated remediation for well-understood incident patterns — for example, automatically restarting a service instance when a specific correlated pattern (memory leak signature + latency spike + no recent deploy) is detected with high confidence. This is where Aiden for SRE typically comes in: Using the correlated, root-caused incident as the trigger for a pre-approved remediation action, rather than waiting for a human to read the alert and act.
Expected Result: A shrinking gap between detection and remediation, with routine incident patterns increasingly self-resolving.
Testing Your Setup
Before trusting correlation in production, run it through a structured validation pass:
Replay 3–5 historical incidents of varying severity and complexity.
Compare AI-correlated groupings against what your team determined manually during the actual incident.
Check root-cause ranking accuracy — was the true cause in the top 1–2 candidates?
Confirm the time to first correlated alert is meaningfully faster than your team’s manual triage time.
If correlation accuracy is inconsistent across incident types, it’s usually a sign that your topology mapping is incomplete for certain services, which is why it is worth revisiting Step 1 before adding more correlation rules on top of gaps in the data.
Real-World Results
Teams that move from manual alert triage to AI-driven correlation typically see improvement concentrated in the triage and diagnosis phases specifically, since that’s the part of MTTR correlation is designed to compress:
Alert Volume per Incident: Often reduced from dozens of individual alerts down to a single correlated incident
Time to Root-Cause Identification: Meaningfully faster when root-cause ranking is accurate, since engineers stop manually cross-referencing dashboards
On-Call Burden: Fewer distinct pages per incident, since related alerts are grouped rather than firing independently
Your specific numbers will depend heavily on how mature your topology mapping is and how much historical incident data the correlation engine has to learn from — treat the first few weeks as a tuning period, not the final benchmark.
Next Steps
Now that you have AI-driven correlation running, you can:
AI-assisted development has quietly changed which part of the software lifecycle is hardest. Writing code is no longer the bottleneck. Understanding what the resulting system is actually doing in production is. When agents and assistants can generate whole services in an afternoon, the ability to validate change, investigate incidents and reduce risk becomes the constraint that actually decides how fast a team can ship — and that puts observability engineering squarely at the center of the conversation.
Alan Shimel talks with Liz Fong-Jones, Technical Fellow at Honeycomb, about what that shift looks like from the ground. Fong-Jones draws a sharp line between telemetry and observability that most teams still blur. Telemetry is the raw material: logs, metrics, traces describing what is happening inside a system. Observability is the broader property of being able to combine that data with human knowledge and process to actually understand the system and make better decisions. Her framing is that observability is a living quality of software — like testability or accessibility — and a team is never done with it.
They get into how AI reshapes the observability loop itself. Fong-Jones sees real leverage in agent-assisted workflows that generate instrumentation code, sift through mountains of telemetry and help engineers move through investigations faster. The point is not that AI removes human judgment from the loop. It changes where humans spend their time, pulling them off toil and toward the interpretation and decision-making that only a person can do. The second edition of the O’Reilly book Fong-Jones co-authored reflects that shift — the field has moved past arguing whether observability differs from monitoring and into how it supports AI-driven software delivery.
The bigger arc is that cloud native has become the default substrate for modern software, containers and Kubernetes are now assumed rather than debated, and OpenTelemetry sits underneath the whole picture as the standard everyone builds on. In that world, observability engineering is not a nice-to-have added on top of production. It is the thing that lets teams move faster with AI while keeping the risk of moving faster inside acceptable bounds — and the organizations that treat it as a first-class engineering discipline are the ones that get to keep shipping when everyone else is chasing incidents.
Code creation has never been faster, but the machinery around it has not sped up in step. AI coding tools can now produce more lines in a morning than a team used to ship in a sprint, while testing pipelines, review processes and device coverage still move at the pace they did two years ago. The result is a widening gap between how much software is being written and how much of it anyone can actually verify — and that gap is where production incidents live.
Mike Vizard sits down with Prince Kohli, CEO of Sauce Labs, to work through what that imbalance is doing to release quality. Kohli’s framing is that leaders keep mistaking code velocity for product velocity. Generating an application in an afternoon is not the same thing as shipping one. Reviews still need to happen, tests still need to be authored, device coverage still needs to be real, user journeys still need production-like validation. Skip those layers and the AI advantage evaporates the first time a subtle defect makes it to customers.
Kohli lays out a specific failure pattern engineering leaders should look for: using the same AI model to write the code and verify the code, which he compares to letting a student grade their own homework. Mature teams are moving to independent verification systems that evaluate application intent, author their own tests, run them in the right environments and troubleshoot the failures they surface. Human reviewers also carry a heavier cognitive load in this world, because developers now read a lot more code than they write and often lack the context behind the AI-generated architecture in front of them.
Sauce Labs research puts numbers on the risk. Kohli says 80 percent of organizations have traced a production incident or outage back to AI-generated code, 90 percent reported serious business impact from those incidents, and 66 percent admit they compromised quality or testing standards to hit faster release deadlines. His pitch is not to slow AI down. It is to invest in verification at the same rate as generation, because the teams that treat testing as a second-class citizen in an AI-first pipeline are the ones already writing incident post-mortems they did not need to write.
Conducted by Wakefield Research on behalf of Sauce Labs, a provider of an application testing platform, the survey finds 83% of respondents work for organizations where more than 10% of the code running in production environments has been generated using AI, with 28% now running more than a quarter of the code in their production environments using AI tools.
A full 93% said they also receive reports on that code, but only 38% receive them regularly. A total of 89% also said they disclose to customers when code has been generated using AI, but only 27% said they do so consistently.
A total of 70% said their organizations now spend more than $1 million annually on AI-assisted application development tools and platforms, with 41% spending $5 million or more. A total of 89% described the return on investment (ROI) in AI as either significantly positive (30%) or somewhat positive (59%). More than half (51%) expect fully autonomous testing and software deployment to be possible in either two (27%) or three years (24%), compared to 8% that believe it is achievable now.
More than half (53%) said deploying AI too quickly is the greater risk than falling behind rivals, while conversely, 47% said falling behind is the greater risk. More than 91% also said software quality has either significantly improved (18%) or somewhat improved (73%) thanks to adoption of AI tools.
Overall, only 12% said leadership was greatly concerned about risks that result from deploying AI-generated or AI-tested code in production environments, compared to 58% that are moderately concerned. Only 12% are extremely concerned that safeguards are not effective at preventing AI-generated or AI-tested features that have quality or safety concerns from reaching production, compared to 81% that are very or somewhat concerned.
Additionally, the survey finds a full 84% of respondents lead organizations that have eliminated or significantly reduced roles because of AI adoption, with 53% having cut junior/entry-level developers, while 42% have cut QA testers. Another 34% have reduced the number of technical writers employed, with 24% having cut automation engineers. A total of 42% have also reduced the number of entry-level software engineering positions, compared to 38% that have increased them.
Sauce Labs CEO Prince Kohli said, overall, the survey results suggest that given the number of production incidents traced back to AI code it is apparent that not enough attention is being paid to the need to test code generated using AI tools before it is deployed in production environments. As the volume of code being generated increases, it’s now more a question of how many more incidents will there be before DevOps teams are required to revisit existing workflows, he added. On the plus side, a full 61% said that code is at least sometimes tested using AI tools, but only 6% said they use these tools all the time or often.
Of course, many DevOps teams have not been consistently testing code for years. The survey, for example, finds 53% of respondents admit their organization sometimes (47%) or frequently (6%) has shipped software to production with known or unresolved testing issues in the past year. Two thirds (66%) said their organization has either definitely (21%) or probably (45%) compromised on quality or testing standards to meet release deadlines in the past 12 months. Nearly two thirds (65%) said the approximate financial impact of their most significant software quality incident in the past 12 months exceeded half a million dollars.
Those software defects also resulted in negative media coverage (40%), loss of a major contract/customer (38%), a regulatory inquiry or fine (34%), customer lawsuit/legal action (33%), stock price impact (30%) and an executive departure (27%).
On the plus side, nearly two thirds (64%) report dedicated quality assurance/testing headcount has increased in the past year, with 18% seeing a significant increase.
As always, the proof is in the quality of the proverbial application pudding. While AI has clearly improved productivity, it is also worth remembering that it only takes one or two disgruntled customers to wipe out any of the business benefits that might have been gained.
For years, DevOps had a pretty straightforward mission: help teams ship reliable software faster by getting development and operations folks working together. Tools like automation, continuous integration, continuous delivery, infrastructure as code, and better observability have all been key pieces of the puzzle. But now, things are shifting again. This time, artificial intelligence is changing the whole landscape.
AI isn’t just some fancy code-completion tool anymore. Modern engineering teams are building it into almost every stage of how they deliver software. From writing deployment pipelines, to troubleshooting incidents, and analyzing data, AI is stepping into each corner. People are starting to call this new wave “AI-Native DevOps.” In this world, AI’s not just here for a productivity boost. It’s actively involved in pushing software out the door.
The change might sound subtle, but it’s actually a pretty big deal. Old-school DevOps relied on automation, sure, but it all followed predefined rules. These new AI-powered systems go further. They get the context, spot patterns, offer suggestions, and sometimes even take action without much help from a person. That’s making teams rethink what automation even means.
Take continuous integration, for example. Before, the pipeline just checked if code could build, if the tests passed, and if the quality gates were hit. Now, AI digs deeper. It looks at where things have failed before, flags risky changes, and even tells you which pull requests really need your attention. Instead of treating every code change the same, AI helps teams zoom in on what matters most.
You see the same thing happening with deployments. Teams used to rely on set rules. If the tests pass and everything builds, then maybe someone checks it off and out it goes. AI’s changing that. It sifts through signals like past release trouble, service dependencies, infrastructure health, and previous incidents. It doesn’t take the decision away from humans, but it gives teams more insight before they ship something live.
Then there’s incident management. When stuff breaks, engineers usually lose time chasing logs, poking through dashboards, or searching documentation just trying to find what went wrong. AI can pull data from all those different tools, tie things together, and even hand over a shortlist of likely causes. People still need to check its work, but getting to the root problem? That’s way faster now.
Observability tools are changing too. Nowadays, monitoring systems collect ridiculous amounts of data. Figuring out what actually matters? That’s never been easy, especially as everything gets more spread out. AI helps spot weird patterns, connects events from different services, and filters out the noise. Instead of hitting teams with a flood of pointless alerts, these new tools highlight just the ones that really deserve a response.
Infrastructure management’s seeing a shift as well. Infrastructure as Code is still the go-to for running cloud setups, but keeping all those environments clean and secure is a grind. New AI assistants can flag security gaps, spot inconsistent configs, suggest improvements, and even whip up code templates to match your architecture. Engineers still review everything before it goes live, but the whole process gets easier and less error-prone.
Maybe the biggest change is the rise of AI agents. Unlike old assistants that just answer when called, these agents can run entire workflows within safe limits. Say there’s a performance dip. An operations agent might spot it, pull diagnostics, check for similar past incidents, put together a fix, and ask a human for the green light. It’s a big leap from the kind of automation where you had to program every single step ahead of time.
Even with all this progress, AI-Native DevOps isn’t just about cranking up automation. Trust is still at the heart of everything. Production needs things to work as expected, and teams need real accountability. AI’s suggestions only matter if people understand the “why” behind them. So, clear approval paths, audit trails, and tight permissions aren’t optional. They’re the rule before any AI system makes an operational call.
Data quality matters too. AI’s only as smart as the information it gets. Weak documentation, flaky monitoring, mislabeled incidents, or bad operational data? That just makes the AI less helpful. Teams looking for real value often have to clean up their own processes with better documentation, improved monitoring and tighter discipline before the AI can do any heavy lifting.
Security’s another big deal. These AI tools need deep access: repos, pipelines, cloud environments, monitoring, the lot. If you’re not strict about identity and privilege, all that integration piles on new risks. Security needs to be baked in from the start, not added at the end.
DevOps engineers are feeling this shift firsthand. Instead of grinding away at repeatable tasks, they’re expected to review AI’s work, set the guardrails, keep operational data solid, and make sure automation’s reliable. Their focus is shifting from doing routine chores to supervising smarter systems and making bigger-picture decisions.
Don’t get the wrong idea. AI isn’t replacing DevOps pros. Production systems are messy, business needs change on a dime, and chaos never goes out of style. There’s no substitute for hands-on experience and solid engineering judgment, especially when things hit the fan.
What’s happening is DevOps is just moving up to the next level. Automation’s still huge, but what we call “automation” is evolving, getting smarter and more aware of the bigger picture. Teams that mix AI’s strengths with strong engineering, real guardrails, and sharp human oversight will come out ahead.
In the end, it’s not about how many AI tools you stack up. The real winners are building a kind of partnership, letting smart systems boost their own expertise rather than trying to replace it. As software delivery keeps getting more complex, finding the right mix of artificial intelligence and human common sense might end up defining the best DevOps teams in the years to come.
devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience.
Our goal in these challenging economic times is to make it just that much easier for DevOps professionals to advance their careers.
Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role, it tends to create opportunities for others.
The ten job postings shared this week are selected based on the company looking to hire, the vertical industry segment and naturally, the pay scale being offered.
We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration.
Dice
Booz Allen Hamilton
McLean, VA DevOps Engineer
$77,600 to $176,000
Alarm.com
Centennial, CO DevOps Engineer
$120,000 to $130,000
HarbourVest Partners
Boston, MA DevOps Engineer
$135,000 to $145,000
Auria Solutions USA, Inc
Boulder, CO DevOps Engineer
$140,000 to $160,000
AI coding assistants write code fast. Whether that code can be trusted is a separate question, and it’s becoming a more urgent one. Surveys this year put average developer trust in AI-generated output at just above the midpoint of a five-point scale, and more than half of developers admit they ship AI code without testing it first. Coverage numbers on AI-written code often look fine on paper, but plenty of that coverage turns out to be shallow: tests that check a result isn’t null and call it a day.
Microsoft is trying to close that gap with a new open-source agent built specifically for unit testing. Called code-testing-generator, it lives inside the dotnet-test plugin in the dotnet/skills repository, and it’s designed to answer the questions a bare “generate unit tests” prompt leaves open: which code needs coverage, what framework the project already uses, where new tests belong, and whether the build will actually find them.
That last point matters more than it sounds. A new test project can compile and pass locally and still never run in CI, because nobody wired it into the solution or the test command. It’s a quiet failure mode, and one Microsoft says its agent checks for directly.
How it Works
The agent doesn’t jump straight into writing code. It starts by researching the repository: detecting the language, finding the test framework already in use, and studying existing tests to learn local conventions. From there, it scales its approach to the size of the job. A single method gets a direct read-write-validate pass. A larger request triggers planning, and a request to cover a whole subsystem runs an iterative loop until it hits a coverage goal.
Once it starts writing, the agent works from simple code toward code with more dependencies, mapping behaviors to test files as it goes. It runs the tests as it writes them, and if something doesn’t compile or an assertion is wrong, it fixes it before moving on. It won’t touch production code, and it avoids writing tests that hit real URLs, open ports, or depend on exact timing — the kind of shortcuts that make test suites flaky.
The last step is the one most testing tools skip: checking whether the tests are actually useful. Microsoft’s agent runs a lightweight form of mutation testing, introducing small code changes that should break a good test, then confirming the existing tests catch them. It also checks for weak assertions, verifies every requested scenario has a matching test, and runs the full suite before calling the job done.
What the Numbers Show
Microsoft benchmarked the agent against stock GitHub Copilot using the same underlying model, across 152 tasks pulled from real repositories. The specialized agent completed 92.1% of tasks, against 78.9% for stock Copilot — a 63% cut in failures.
The gap was widest on vague prompts, the kind that leave most of the decision-making to the agent. There, the specialized agent passed 88.8% of tasks against 66.3% for stock Copilot. On prompts tied to a specific code diff, the agent passed all 15 test cases in the benchmark; stock Copilot passed none. When prompts were already detailed, the two setups tied, which tracks: a good prompt does the agent’s research for it.
The gain didn’t come from generating more tests. The specialized agent actually produced slightly fewer tests than stock Copilot, with roughly the same average line and branch coverage, and finished about 5.5% faster on average. The improvement was in reliability — getting a working result on the first try — not sheer volume.
Microsoft also ran the comparison across models, including Claude Opus 4.8, GPT-5.5, and Claude Haiku 4.5, and found the workflow helped all three, with the biggest failure reduction on Opus. It held up across languages too, more than doubling the completion rate on Python tasks and passing every Go task in the benchmark, though stock Copilot still edged it out slightly on PowerShell.
A tougher, independent benchmark called SWE Atlas — which checks whether generated tests actually catch injected bugs — showed a smaller but still positive gap: 36.4% task completion for the specialized agent against 27.3% for stock Copilot.
Mitch Ashley, VP and practice lead for software lifecycle engineering and AI-native software engineering at The Futurum Group, says the real story is in the method, not the model. “Coverage numbers have been lying to teams for years. AI-generated code made the lie bigger,” Ashley said. “Microsoft’s agent runs mutation testing on its own output, breaking code to confirm the tests actually notice.”
Ashley points out that the benchmark’s headline number — a 63% cut in failures — traces back to procedure, not a smarter model. “Verification debt does not get paid down by generating more tests,” he said. “Engineering leaders should require proof that a test fails when the code breaks, before coverage targets go into any AI policy.”
The Bigger Pattern
None of this makes AI-written code trustworthy by default. It points at something narrower and more useful: a workflow that treats test generation as a process with checkpoints, not a single prompt-and-done request. Given how much AI-generated code teams are shipping right now, and how thin the testing behind it often is, that kind of structure is worth paying attention to.
The agent is open source and available now through GitHub Copilot CLI, with support in Visual Studio Code and VS Code Insiders in preview. Microsoft says Visual Studio support is coming. It covers a dozen languages beyond .NET, including Python, Go, Java, and Rust, and Microsoft says it’s exploring whether the same approach could extend to other kinds of testing down the line.
