devopsy.kinsta.cloud

Tag: devops

Ten Great DevOps Job Opportunities

devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted as part of an effort to better serve our audience.

Our goal in these challenging economic times is to make it just that much easier for DevOps professionals to advance their careers.

Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role, it tends to create opportunities for others.

The ten job postings shared this week are selected based on the company looking to hire, the vertical industry segment and naturally, the pay scale being offered.

We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration.

Greenhouse

HERE
New York, NY
Government DevOps Engineer – USA
$145,000 to $185,000

Rocket Lab
Littleton, CO
DevOps Engineer II/Senior
$115,000 to $170,000

Mixpanel
Remote, US
Software Engineer, DevInfra
$157,500 to $213,000

Sphinx Defense
Remote, US
DevOps Engineer
$140,000 to $160,000

VIA
Somerville, MA
Senior DevOps Specialist
$165,000 to $195,000

BlackSky
Herndon, VA
Senior Infrastructure Engineer
$135,000 to $150,000

Ashby

Perchwell
Remote, US
Senior Site Reliability Engineer
$170,000 to $230,000

Thumbtack
Remote, US
Senior Software Engineer, Site Reliability Engineering
$210,800 to $272,800

Gamma
San Francisco, CA
Site Reliability Engineer
$230,000 to $310,000

A Place for Mom
Remote, US
Staff DevOps Engineer
$150,000 to $170,000

June 29, 2026
AI Coding Agents Are Pulling CI Feedback Into the Inner Loop

The traditional shape of CI/CD assumed humans worked in the inner loop and pipelines policed the outer one. AI coding agents are tearing that geometry apart. When code can be generated in seconds, waiting until a pull request to run validation, tests, code review and standards checks turns the outer loop into a backlog of preventable rework. The pipeline itself becomes the bottleneck — and the bottleneck shows up not just as time lost but as wasted tokens, half-finished features and engineering controls that fall behind the rate of change.

Rob Zuber, CTO of CircleCI, joined Mike Vizard to lay out how that pressure is reshaping CI/CD platform design. Zuber’s read is that CI quality work has to live where the work is actually happening — inside the developer and agent loop — so issues are surfaced and corrected before changes ever leave a working branch. That means pushing validation, testing, agentic review and deterministic guardrails right up against the moment code is generated, rather than catching problems hours or days later in a pipeline run.

Zuber and Vizard dig into what shifting that feedback earlier actually requires under the hood. Agentic review needs deterministic checks it can trust, observability has to extend to agent behavior, and feature flags become a far more central tool when changes are landing at machine speed. DORA metrics still matter, but Zuber argues the leaders are layering token consumption and pipeline efficiency on top to see whether AI is actually compounding throughput or just compounding cost.

The bigger shift is one of control surface. As AI amplifies how much software an organization can produce, the engineering controls around quality, security and release safety have to amplify in lockstep — or the gains evaporate.

June 26, 2026
AI Is Accelerating How Fast We Build the Wrong Thing

AI coding assistants have made it trivially easy to ship software faster — and that is precisely the problem. Human developers used to absorb the gaps in a vague spec by asking questions, reading between the lines and quietly steering toward something useful. AI tools do not work that way. They take incomplete requirements at face value and build them literally, which means small ambiguities at the front of the process compound into expensive mistakes at the back. The result is shops moving at AI speed in directions they never meant to go.

Jeff Keyes, Field CTO at Allstacks and a 30-year veteran of Microsoft, Atlassian and Planview, sat down with Alan Shimel to preview his Flowtopia 2026 talk on a new operating model for product management. His thesis is blunt: the smartest organizations he is working with have flipped the script and stopped treating AI transformation as an engineering problem. The real leverage sits upstream, in how product teams scope work and write requirements before a single line of code — human or machine — gets generated.

Keyes and Shimel work through why more than 80% of companies still cannot point to meaningful ROI from their AI tooling investments. The pattern Keyes keeps seeing is faster output stacked on top of unclear intent, which produces velocity metrics that look impressive while value realization stalls. He argues requirements quality has quietly become the highest-leverage variable in AI-assisted development, and that PM craft — not prompt craft — is what separates the teams pulling ahead from the ones spinning.

What that operating model actually looks like in practice is where the preview gets concrete. Keyes lays out how PM teams need to rebuild around clarity, traceability and feedback loops that hold up when an agent, not a junior dev, is doing the implementation. With AI accelerating both the right work and the wrong work in equal measure, the organizations that fix product management first will compound an advantage the rest of the field cannot catch.

June 26, 2026
How AI Is Transforming DevOps and Cloud Engineering
AI is rapidly transforming modern software infrastructure. While AI is often associated with chatbots or generative models, one of its most significant impacts is happening behind the scenes — within DevOps and cloud engineering.

As applications scale across distributed cloud environments, traditional DevOps practices are increasingly being enhanced by intelligent systems capable of predicting failures, optimizing resources and automating operational decisions.

This shift is giving rise to what many refer to as ‘AI-driven DevOps’ or ‘AIOps’, where ML helps engineering teams manage complex systems more efficiently.

The Growing Complexity of Cloud Systems

Modern applications rely on microservices, containerization and orchestration technologies to operate at scale. Tools such as ‘Docker’ and ‘Kubernetes’ have become foundational components of cloud-native platforms.

These technologies allow applications to scale dynamically, but they also introduce operational complexity. A single distributed system may contain hundreds of services generating massive volumes of logs, metrics and telemetry data.

For DevOps teams, manually monitoring these signals can quickly become overwhelming.

Traditional rule-based monitoring systems often struggle to detect subtle anomalies or predict system failures before they occur. As infrastructure grows more complex, organizations are increasingly turning to AI-driven operational intelligence.

AIOps: Bringing Intelligence to Operations

AIOps applies ML techniques to operational data such as logs, system metrics, traces and infrastructure events.

Instead of relying only on predefined thresholds, AI models learn normal patterns in system behavior and automatically detect anomalies.

For example, an AI model monitoring infrastructure metrics may detect unusual latency spikes or resource consumption patterns. Rather than waiting for a full outage, the system can alert engineers or automatically trigger scaling actions.

This proactive approach shifts DevOps from reactive incident management toward predictive infrastructure operations.

Real-World Experience: AI in Public Safety Infrastructure

The value of AI-driven operations becomes especially clear in mission-critical environments.

In one public safety platform I worked on, the system processed large volumes of real-time telemetry data from distributed IoT devices. The infrastructure relied on containerized services orchestrated through Kubernetes, allowing the platform to scale dynamically as new devices connected to the network.

However, as data streams increased, traditional monitoring tools began producing large numbers of alerts without clearly identifying the root cause of performance issues.

To address this challenge, ML-based anomaly detection models were introduced into the operational pipeline. These models analyzed infrastructure metrics and device telemetry patterns to detect unusual system behavior.

This allowed the system to automatically trigger scaling policies and notify engineers before system performance degraded. In environments such as public safety monitoring, where reliability and response time are critical, AI-driven operational intelligence can significantly improve system resilience.

AI for Infrastructure Optimization

Beyond monitoring, AI is also improving how infrastructure resources are allocated and optimized.

Cloud platforms often run with over-provisioned resources to avoid performance issues. While this improves reliability, it can also lead to inefficient resource utilization.

ML models can analyze historical workload patterns and recommend optimal infrastructure configurations. These systems can support:
- Predictive autoscaling based on demand patterns
- Intelligent workload scheduling across clusters
- Resource allocation optimization
This enables organizations to maintain high system performance while reducing infrastructure costs.

The Future: Autonomous Infrastructure

The long-term vision for AI-driven DevOps is autonomous infrastructure.

In this model, cloud platforms continuously monitor their own operational behavior and automatically apply corrective actions when anomalies occur.

Examples of emerging capabilities include:
- Self-healing infrastructure that replaces failing nodes
- Predictive scaling before traffic surges occur
- Automated root cause analysis across distributed systems
While fully autonomous infrastructure is still evolving, many organizations are already implementing early forms of these capabilities.

Preparing for an AI-Driven DevOps Era

As AI becomes embedded in DevOps workflows, engineers will need to expand their expertise beyond traditional infrastructure management.

Future DevOps professionals will increasingly work at the intersection of:
- Cloud infrastructure engineering
- Data pipelines and observability systems
- ML for operational analytics
Engineers who combine knowledge of distributed systems with AI-driven analytics will play a key role in building the next generation of intelligent platforms.

Conclusion

DevOps is entering a new phase where AI enhances how infrastructure is monitored, optimized and maintained.

By applying ML to operational data, organizations can detect issues earlier, automate responses and build more resilient cloud platforms.

Rather than replacing engineers, AI will empower them to design smarter systems capable of operating efficiently at scale.

As cloud environments continue to grow in complexity, AI-driven DevOps will become an essential foundation for modern software infrastructure.
June 26, 2026
Your Foundation Model is a Service. Operate it Like One
The FM life cycle is just the SDLC with more math and less mercy at 3 a.m.

You have survived canary deployments that took out 40% of prod instead of 5%. You have been paged because someone merged a config change on a Friday. You know exactly what it feels like when your monitoring tells you everything is fine, right up until the moment customers start tweeting.

Now your organization has a foundation model (FM) in production, and I promise you: The on-call playbook you have built over a decade does not cover what happens when your model starts confidently answering customer questions with plausible-sounding nonsense.

Most teams are still treating FM deployment as a data science handoff. Ship the model, write a README, move on. That is the same mistake we made in 2012 when we thought you could just rsync your app to a server and call it a release pipeline. We know how that story ends.

Here is what it actually looks like to run this stuff.

The Base Image Problem (and Why Your Data Team Doesn’t Get It)

Pre-transformer ML was tedious, but at least it was honest. You needed a labeled dataset per use case. Fraud detection, churn prediction, sentiment analysis: Each one a separate model, a separate pipeline, a separate thing to break. The maintenance overhead was brutal, but failure was contained.

FMs break that containment model. One base model, trained on a chunk of the internet, handles everything from code generation to customer support. The appeal is obvious. The operational risk is less so.

Think of pretraining as an extremely expensive compile step. It is where the model learns statistical relationships between concepts. If that compiler runs on low-quality, biased or outdated data, you have baked a hard ceiling into your system. No amount of prompt engineering digs you out of a bad base model. You are optimizing on top of a flawed binary.

If you need to update the model’s knowledge without a full retrain, you use continuous pre-training. Think of it as rebuilding your Docker base image with updated OS packages. Same core logic, refreshed context. It is not cheap, but it is cheaper than discovering your model has no idea what happened in your domain after its training cutoff.

What You Actually Need to Know About the Architecture (Without the PhD)

I am not going to walk you through the math. You do not need it. You need just enough architectural literacy to know where to look when production breaks.

Transformers and the Hallucination at 3 A.M.

Most LLMs you are deploying are transformer-based. The short version: Text becomes vectors, the model predicts what comes next and attention mechanisms let it weigh context across the entire input window. It is elegant until it is not.

When a transformer starts hallucinating, your instinct will be to check the deployment. Wrong move. The model is not broken. The failure mode is almost always a distribution shift: The input looks nothing like the data the model learned on. A financial services model trained on general-purpose data will confidently invent regulatory details. A support bot trained on last year’s product docs will answer questions about features that no longer exist.

Do not debug the container. Debug the input patterns. That is where the failure lives.

Diffusion Models and the Latency Trap

Diffusion models (images, audio, video) are iterative. They start with noise and denoise step by step until you get a coherent output. Every step is a GPU pass. If your generation latency is spiking, scaling horizontally to more H100s is the expensive reflex answer.

Check your sampling steps first. Tuning the inference configuration is almost always faster and cheaper than adding hardware. Understand what you are running before you spend money on compute, you may not need.

Optimization is a FinOps Decision, Not a Science Project

Every CFO who has seen an AI infrastructure bill for the first time has the same expression. You have three levers. They are not interchangeable. Using the wrong one costs time, money and in some cases, model stability.
1. Start With Prompt Engineering: It costs nothing in infrastructure, and you will be surprised how far you can get with it. Most teams skip straight to fine-tuning because it feels more technical. That is almost always the wrong call.
2. RAG is Your Sweet Spot as an SRE: You pull fresh, relevant data into context at runtime and inject it into the model’s prompt. The model stays current without touching its weights. There is no drift risk on the model itself, your retrieval layer is just another service you already know how to monitor.
3. Fine-Tuning is a Last Resort: You are literally modifying the model’s internal weights. You will need a full regression suite. Fix one capability, break three others. The risk profile looks a lot like a major refactor with no rollback plan. Never fine-tune until you have genuinely hit the ceiling with RAG and prompt engineering. The teams that fine-tune too early spend months chasing weight drift they introduced themselves.
The Feedback Loop is the Architecture

Here is what makes AI systems genuinely different from the services you have operated before: They fail without throwing a 500.

Your load balancer shows green. Your error rate is nominal. Response latency looks fine, and somewhere, your model is telling users things that are subtly, confidently wrong. You will not find out from your existing dashboards. You will find out when a customer escalates or when someone screenshots a bad response on LinkedIn.

You need a separate observability layer. Tools like LangSmith or Arize exist specifically because your existing APM stack was not built for this failure mode. Call them optional and you are not running a life cycle: You are hoping for the best with expensive compute.

What you need to monitor:
- Output Drift: Is the distribution of model responses shifting over time, even when inputs look stable?
- Input Anomalies: Are users querying patterns that fall outside the training distribution?
- User Feedback Signals: Thumbs down, correction requests and abandonment patterns are your real-world ground truth.
- Latency Per Component: Your retrieval layer, your model call and your post-processing should be metered separately.
If you are not collecting that signal and feeding it back into your next tuning cycle, you are not running a life cycle. The FM is just drifting and you will not know until the drift is customer-visible.

The Operational Reality

The FM life cycle is the SDLC with more math and more ways to fail silently. The engineers who ship production AI systems, which actually hold up, are not the ones who read the most research papers. They are the ones who treated the model like a service from day one: With runbooks, with observability, with a clear escalation path when outputs degrade.

Everything you already know about operating distributed systems applies here. The failure modes are just newer and quieter.

Treat the FM like any other critical dependency. Own the inputs, monitor the outputs and for the sake of your on-call rotation, do not fine-tune until you have exhausted every cheaper option first.
June 26, 2026
Modernization Is Not Migration

Modernization used to mean something simpler: Move the workloads, update the tooling, declare the project done. In practice, that approach meant engineers manually migrating hundreds of DataStage jobs one at a time — a process that was slow, error-prone and impossible to scale as platforms grew. The traditional model worked when volumes were low. It broke entirely when weekly release windows started carrying 500 jobs and the only way through was brute-force manual effort.

What changed the equation was not just cloud infrastructure, but a fundamentally different operating model. When a CI/CD-based promotion mechanism replaces manual steps — reducing what once required hours of coordinated effort down to a single parameterized execution — hundreds of jobs could migrate consistently, with less human involvement and a verifiable audit trail. That shift exposed a harder truth: The technology was never the bottleneck; the operating model was.

That distinction matters more than most modernization programs acknowledge. In regulated financial environments, a single poorly governed release, an undetected performance bottleneck or a monitoring gap that cannot identify which of hundreds of running jobs is consuming abnormal resources can cascade into compliance failures, SLA breaches and production incidents that take hours to diagnose. Migration moves workloads. Modernization changes how those workloads are released, observed and recovered. Organizations that confuse the two end up paying cloud prices for legacy-era operational risk.

The Release Bottleneck: Scale Exposes What Manual Processes Cannot Sustain

The scale problem became undeniable on Thursday release windows. With roughly 500 DataStage jobs queued for migration each week, a single Jenkins server connected to a Windows host via known_hosts authentication would spend close to two hours sequentially placing files from commit IDs into DataStage directories, then waiting on compilation and promotion to complete. The process was not broken. It was simply not built for the volume it was being asked to carry.

The fix was horizontal scaling applied to the migration layer itself. Three dedicated Windows migration servers — MIG servers hosted on OSV — were introduced to split the job queue and run promotion concurrently across all three nodes. Jenkins triggers the build, establishes the known_hosts connection and Git commands distribute the committed file changes across the MIG servers in parallel. Each server handles its share of the queue independently. Bulk migration dropped from two hours to 45 minutes. The same Thursday release window that previously consumed an entire afternoon now closes before the first standup of the day.

The architectural lesson is transferable. What looked like a tooling problem was a throughput problem, and the solution was treating the migration layer the same way any bottlenecked data pipeline gets treated: Parallelize it. Governed CI/CD pipelines with commit-level traceability, parameterized environment targets and approval gates tied to security groups and change records are not overhead. They are what make a high-volume, audit-ready release possible at enterprise scale.

The Observability Gap: Prevention Without Detection is Incomplete

The symptom was a network breakdown on OSV servers under load. The cause, once we could see it, was partition skew — DataStage jobs with uneven data distribution hammering specific nodes while others sat idle, driving CPU utilization past sustainable thresholds with no way to identify the responsible job until the platform was already in distress. With thousands of jobs running concurrently, the existing monitoring told us the cluster was under pressure. It could not tell us where to look.

This is one of the most underestimated failure modes in enterprise cloud modernization. When data traverses a network for distributed processing, uneven partitioning concentrates compute demand on a subset of nodes. Jobs that were not properly partitioned surge CPU instantly. Infrastructure monitors like Dynatrace surface the result — CPU utilization exceeding 90% — without surfacing the identity of the job causing it. The gap between the alert and the answer is where incidents live.

The solution is to build a second observability layer beneath the infrastructure monitor — one designed around job identity rather than cluster state. In one financial data platform implementation, a DB2 pipeline table was constructed to capture operational metadata directly from the DB2 server at the job level: Job name, volume of data processed, number of CPUs consumed, percentage of CPU utilization and execution timestamp. This metadata is ingested on a scheduled cadence into a BigQuery stats table, where it becomes queryable alongside the rest of the platform’s operational data.

On top of that stats layer, Looker reports run on an hourly schedule and apply a threshold rule: Any job with CPU utilization above 90% is flagged in red and triggers an automated notification routed directly to the responsible production support team and the L6 engineering escalation group. The alert is no longer ‘the cluster is hot’. It is ‘job X on node Y consumed Z CPUs at 02:23 p.m., processed N records and has now exceeded the threshold three cycles in a row’. That is the difference between a signal that opens a bridge call and one that closes an incident in minutes.

This architecture — infrastructure monitor surfacing the symptom, job-level telemetry pipeline identifying the cause, scheduled reporting enforcing the threshold and automated routing engaging the right team — is what targeted observability looks like in a regulated production environment. It turns performance management from an operations burden, reliant on institutional memory and manual log trawling, into a data-driven engineering discipline. The platform can now explain its own behavior under stress. That is what operational maturity requires.

Modern Regulated Data Architecture: Design for Operations, Not Just Delivery

In regulated financial data platforms, architecture should be evaluated not only by how data moves, but by how reliably the platform can be operated. A layered ingestion model may move data from upstream financial systems into cloud storage and processing tiers, with transformation logic in intermediate layers and curated exports sent to downstream reporting and compliance systems. But architecture alone does not create operational confidence.

What distinguishes a resilient platform is the operational layer around it: Automated promotion across environments, governed release controls, telemetry pipelines that capture workload behavior at regular intervals, cloud cost thresholds tied to workload patterns, schema management discipline and clearly documented recovery paths for production incidents.

Without these investments, cloud migration often produces familiar post-go-live problems: Unexplained cost spikes, slower incident response and audit trails that appear acceptable for delivery but fail under regulatory scrutiny. Architecture decisions matter. Operational discipline matters just as much.

Conclusion

Modernization worked only if the platform became easier to change, easier to understand and safer to run under pressure. That is not a philosophical position — it is a measurable one. The clearest proof is not an architecture diagram but a before-and-after comparison any leader can read: The same migration task that previously required manual coordination across multiple engineers now executes with a single trigger, no human intervention and a full audit trail. When execution moved from VM-based infrastructure to OSV servers, compute costs dropped by 40%. When the migration layer was parallelized across three nodes, Thursday release windows shrank from two hours to 45 minutes. When job-level telemetry was built on top of infrastructure monitoring, incident response stopped depending on who happened to know which job was misbehaving.

These are not modernization claims; they are modernization receipts. The organizations that will lead the next phase of cloud data platform maturity are the ones that can show their work — not just describe their architecture, but produce the cost curves, the time comparisons and the incident response metrics that prove the operating model actually changed. Cloud platforms are not modern because they run on managed infrastructure. They are modern when the numbers say so.

June 26, 2026
Why CI-Based Security is Too Late for Modern Node.js Projects

Most Node.js teams rely on CI pipelines to tell them whether their dependencies are secure. By the time that feedback arrives, however, the most important decisions have already been made.

A developer installs a package, writes code, commits changes and pushes to a repository. Only then does the pipeline run, a scanner execute and a report appear. At that point, the issue is no longer discovery; it is remediation under time pressure.

This is the core problem. CI-based security workﬂows are fundamentally delayed.

From Feedback to Friction

In theory, CI pipelines provide a consistent and automated way to validate security. In practice, they introduce a feedback loop that is too slow for how developers actually work.

A typical cycle looks like this:

A vulnerability is detected in CI.

The developer investigates the issue.

A ﬁx is attempted locally.

Changes are pushed.

The pipeline runs again.

A new issue appears or the original issue is only partially resolved.

This loop can repeat multiple times, especially in projects with deep dependency graphs. What starts as a simple ﬁx becomes a multi-step process spread across commits and pipeline runs.

The cost is not just time. It is context switching, delayed releases and growing frustration.

The Structure of Dependency Risk

Node.js applications are built on layered dependency graphs. A single package installation can introduce dozens of indirect dependencies, each with its own version constraints and potential vulnerabilities.

In this environment, not all ﬁndings are equal.

Some vulnerabilities are direct and can be ﬁxed immediately by upgrading a dependency. Others are transitive and require changes in upstream packages or more complex resolution paths.

CI-based workﬂows typically surface all ﬁndings together, without clearly distinguishing between what is actionable and what is not.

This creates noise.

Developers see a list of issues but lack clarity on what they can ﬁx right now versus what is effectively blocked.

A Case Study in Iterative Fixes

In a recent case study of the NestJS repository, a scan of package-lock.json revealed 1,626 resolved packages and 25 vulnerabilities. More important than the number was the structure.

Only 12 of those vulnerabilities were directly ﬁxable. The remaining 13 were transitive.

In one dependency path, resolving a single issue required multiple iterative upgrades as the dependency graph shifted after each change. Each step revealed the next required upgrade.

In a CI-driven workﬂow, this would mean multiple commits and multiple pipeline runs, each introducing delay.

In a local workﬂow, the same process can be handled in a single session by iterating through ﬁxes and verifying them immediately.

This is the difference between a multi-hour or multi-day process and one that can be completed in minutes.

Why CI is Too Late

CI pipelines are valuable for validation. They are not ideal for discovery.

By the time a vulnerability appears in CI, the developer has already moved on. The code is written, the dependency is introduced and the context is no longer fresh.

Security feedback is most useful at the moment a decision is made, not after the fact.

This is why CI-based security alone is insufficient. It introduces delay at the exact point where speed and clarity matter most.

A Better Workflow

What developers need is a workﬂow that aligns with how they actually build software.

That means moving dependency analysis earlier, closer to the developer and grounding it in the actual lockﬁle that deﬁnes the dependency graph.

A local scan-ﬁx-rescan workﬂow changes the dynamic completely.

The CVE Lite CLI was built around that exact gap. Instead of waiting for a CI pipeline to reveal dependency issues after code has already been pushed, the tool scans JavaScript and TypeScript projects locally from their lockﬁles and focuses on the questions developers actually need answered in the moment. It identiﬁes known OSV-backed dependency issues, separates direct from transitive ﬁndings, shows dependency paths and highlights ﬁxed-version guidance where available. The goal is not to generate a bigger report. The goal is to make the next engineering step clearer.

That difference becomes especially important when remediation is iterative. In the NestJS case study, some dependency paths required multiple upgrades as the lockﬁle changed after each install. In a CI-only workﬂow, that would have meant repeated commits and repeated pipeline runs just to discover the next required step. With a local-ﬁrst tool such as CVE Lite CLI, the developer can stay in the same session, apply a package change, rescan immediately and keep working through the dependency path until the result is clean. That is where dependency scanning stops feeling like delayed compliance and starts becoming part of normal engineering practice.

Developers can scan dependencies before committing changes, identify which vulnerabilities are actionable, apply ﬁxes immediately and verify results without leaving their working session.

CI then becomes a validation step, not the primary source of discovery.

Conclusion

Modern Node.js security is not limited by detection. It is limited by how quickly developers can understand and act on what they ﬁnd.

CI-based workﬂows solve part of the problem, but they introduce delays that make remediation harder than it needs to be.

When developers can see what is actionable, ﬁx issues immediately and verify results in the same workﬂow, security becomes part of everyday engineering practice rather than a separate process.

That is how dependency security should work.

June 26, 2026
These are 10 CI/CD Pipeline Mistakes That Slow Down Engineering Teams
Continuous software delivery in the digital age depends on CI/CD pipelines, which enable engineering teams to rapidly develop, test, and deploy code while maintaining high usability and consistency across environments. However, CI/CD pipelines — when systems start small and become complex — can themselves produce a source of friction. Pipelines set up to work for small projects have difficulty scaling with ever more repositories, larger test suites and larger development teams. Slow feedback cycles, increased infrastructure costs and decreased developer productivity are some by-products of bad pipeline design.

Here are 10 CI/CD pipeline mistakes that you’ll often come across as part of normal CI/CD pipeline activities in your own environment, based on common patterns found in a lot of engineering worlds.

Considering CI/CD Pipelines as ‘Set and Forget’

Some teams set up a CI/CD pipeline at the beginning of a project and then rarely touch it afterward. The pipeline of an application grows as it stacks up without a plan! Test suites get bigger, more build steps are introduced and repositories get bigger. However, without regular review, the pipelines slowly become slower and more complex.

Frequent monitoring to review pipeline performance should be a team responsibility. CI/CD pipelines should be reviewed and updated with the underlying software.

One Big Monolithic Pipeline

Another frequent mistake is to build a specific pipeline in charge of all the process of validation and deployment. Such pipelines frequently integrate: Linting, testing, packaging, security scanning and deployment. However, monolithic pipelines are slow and cannot be effectively maintained at scale.

Instead, the scalable route is to use distinct pipelines, designed for different purposes:
- Quick validation pipelines for pull requests
- Extended test pipelines for all merges
- Release deployment pipelines for releases
This structure allows developers to get prompt feedback and yet gain strong validation prior to production releases.

Inability to Monitor Performance of the Pipelines

Engineers put a lot of work into observing production systems; however, CI/CD pipelines can struggle to provide as much visibility. When pipelines slow down because they’re being taken down quickly or because they aren’t reliable, it can be difficult if not impossible for a team to answer questions such as:
- Which stage of the pipeline uses the most time?
- Which repositories trigger the most builds?
- Where do failures happen more often?
The addition of metrics, dashboards and logs to enable pipeline observability allows teams to identify performance bottlenecks and enhance pipeline stability.

Heavy Dependency Management is Responsible for Slow Build Times

Large projects can experience huge delays if they repeatedly download dependencies from external registries on pipelines.

Usage methods that improve dependency performance include: Dependency caching, Docker layer caching, artifact reuse between pipeline stages. Even a small step in caching can drastically reduce build time.

Balanced Testing is Often the Best

Too much testing or too few tests? The testing strategy has a direct effect on CI/CD pipeline performance. Some teams run the entire test suite on each commit, leading to lengthy feedback cycles. Others are not as good at CI testing, still need to do as much manual verification once they have CI.

It is recommended to use the following approach:
- Unit tests for every commit
- Integration tests performed on merges
- Full regression tests before release deployments
By this approach, developers can get quick feedback while maintaining software quality.

Ignoring Parallel Execution

Many pipelines will run tasks in this order because that was the default setup of the pipeline when it was created.

Modern CI/CD platforms support parallel execution of tasks such as running test suites, building microservices, performing validation checks. Teams can drastically reduce overall pipeline execution time and speed up development workflows through parallelism.

Creating Inconsistent Build Environments

Environment inconsistencies cause CI/CD pipeline failures most of the time. Build locally to one environment and CI on the other leads to avoidable unexpected breakdowns. This is when developers run builds locally in one environment and CI systems use another, and they find unexpected failures. Typically, that results in the common situation of code working in your local environment but failing on your pipeline.

A containerized environment or reproducible build environment will help keep the developer machine and CI systems consistent.

Resilient Pipelines Need to Improve

Builds can occasionally be interrupted by network instability, flaky tests or temporary dependency issues. Without resilient pipelines, there is a risk of failures being too frequent to prevent; instead, developers will only be forced to restart builds.

Retry logic for temporary failures, isolation of flaky tests and clear error reporting mechanisms can help in such situations. Enhanced pipeline resilience reduces unnecessary interruptions in an existing development workflow.

Poor Secrets Management Practices

Security is usually ignored early on in CI/CD build. Credentials might be stored directly in configuration files or environment variables as credentials (e.g., API tokens, service keys or database passwords) but without careful safeguards. When CI/CD pipelines share secrets with many services, including the infrastructure pieces, secure secret management may be quite critical.

It is possible to mitigate the security risk dramatically by implementing secret management tools and rotating passwords on-demand.

The Developer Experience is Neglected for Developers

CI/CD pipelines are developed to support developers and speed up the production of software to deliver it closer to market. When pipelines get slow, unpredictable or challenging to debug, they don’t just disrupt productivity, but they stall it.

Good pipelines do the following things: Fast feedback cycles, clear error messages, and predictable execution. When pipelines work well, developers rarely have to think about them.

Final Thoughts

CI/CD pipelines are often seen as support infrastructure, but they are essential to modern software delivery. Pipelines must evolve with the complexity of systems. Review of pipeline design, performance and reliability can lead to a dramatic increase in engineering productivity and system stability. Even the incremental benefits such as effective caching, better caching strategies, clearer pipeline stages and better observability, can create long-term benefits.
June 26, 2026
The Bottleneck Isn’t Coding Anymore. It’s Verification
Last month, one of our autonomous coding agents (not a copilot suggesting inline completions, but a system that reads a ticket, plans a multi-file implementation and opens a PR without a human touching the keyboard) analyzed a ticket, touched 37 files, updated two database migrations and opened a PR in 11 minutes flat. The diff looked clean. Tests passed. The reviewer approved it.

We found the problem at 2:47 a.m. on a Thursday, three days later, during an unrelated log audit. One of our SREs was tailing canary logs trying to trace an intermittent 401, and there it was: A staging secret, printed in plaintext, sitting in a log line the agent had added while ‘fixing’ a failing test. The agent had also introduced a token audience mismatch. The canary environment expected one audience claim; the agent had hardcoded another. Traffic was routing. Nothing was failing. The tokens were just being validated against the wrong audience, which meant our pre-production slice of live traffic had been running with a quietly broken auth contract for 72 hours.

I remember staring at the log line and thinking: This wasn’t written by a junior engineer who forgot to scrub a credential. This was written by a system that has no concept of ‘forgetting’. It did exactly what it was optimized to do. It made the tests green. The fact that it leaked a secret along the way wasn’t a bug in the agent. It was a gap in our process.

That incident rewired how I think about engineering leadership because correctness has shifted into gaps our tests and reviews were never designed to cover.

The Craft Transition: Most Teams Stuck Midway

Software engineering is shifting from a code-centric craft to an intent-centric operating model. Humans describe outcomes. Agents execute multi-step changes. This is happening whether your roadmap accounts for it or not. The only real question is whether it happens inside your governance framework or outside of it.

I see three stages in how organizations are adapting.
1. The first is the classic SDLC everyone knows: Humans write code, humans review code and humans ship code.
2. The second is what people are calling ‘vibe coding’. Prompt-driven, fast, informal. It feels incredibly productive because output goes up. But review capacity stays the same. Engineering incentives stay the same. What actually happens is you start accumulating verification debt: The growing gap between how fast you can generate change and how confidently you can prove that change is safe.
3. The third stage, and very few teams have reached, is spec-driven development. Structured, testable inputs that constrain what the agent can do before it does it. When diffs become cheap to produce, specs become the only scalable way to express intent and control blast radius.
Most organizations I talk to are stuck in stage two. They’ve adopted the tools. They haven’t redesigned the process.

The Data Tells a Story Nobody Wants to Hear

The DORA 2025 State of AI-assisted Software Development report surveyed nearly 5,000 technology professionals and landed on a finding that should make every engineering leader uncomfortable: AI adoption now correlates with higher throughput, but it also correlates with higher instability. AI acts as what the researchers called a ‘mirror and a multiplier’. In cohesive organizations with strong platforms and fast feedback loops, AI boosts efficiency. In fragmented ones, it exposes and amplifies every existing weakness. 90% of developers now use AI in their daily work. Over 80% say it has improved their productivity. However, 30% still report little or no trust in the code AI generates.

The Stack Overflow 2025 Developer Survey, which drew 49,000 responses across 166 countries, makes the trust problem even starker. 84% of developers now use or plan to use AI tools, up from 76% in 2024, with 51% using them daily. But trust in AI accuracy has dropped from 40% to 29% in a single year. More developers now actively distrust AI output (46%) than trust it (33%). Only 3% report ‘highly trusting’ what these tools produce. Experienced developers are the most skeptical: Their ‘highly distrust’ rate is 20%.

Here’s the number that keeps me up at night: 66% of developers say their biggest frustration is AI solutions that are ‘almost right, but not quite’. 45% say debugging AI-generated code takes longer than writing it themselves would have. Three-quarters of developers say that even in a future where AI can handle most coding tasks, the top reason they’d still ask a human for help is: ‘When I don’t trust AI’s answers’.

This isn’t a tooling problem. It’s an arithmetic problem. Generation cost has hit zero. Verification cost is fixed or increasing. If you haven’t redesigned your review process, you haven’t gained velocity. You’ve relocated the work. What used to be ‘writing code’ is now ‘hunting for subtle, agentic regressions’, and the second job is harder than the first.

Why Your PR Process is Already Broken

Here’s the arithmetic that should worry every engineering leader: Generating a thousand lines of code now has near-zero marginal cost. An agent can touch 40 files in minutes. Your senior reviewer, the one you’re counting on to catch problems, is now dealing with diffs they can’t fully reason about in any realistic timeframe.

So they start sampling. Reading some files carefully. Skimming others. Trusting the test suite to catch the rest.

Sampling is not a review strategy. It’s a coping mechanism.

Three things need to change to make agentic workflows survivable.
1. First, the review has to shift from syntax to intent. You can no longer audit every line, so the question becomes: Did the agent output actually satisfy the original constraints? The failure mode you need to worry about now isn’t ‘tests failed’. It’s ‘tests passed, but intent failed’.
2. Second, the spec review becomes your primary gate. You review the instructions before the agent runs, not after. By the time you’re reading the diff, half the damage is already done.
3. Third, command approval becomes your new privilege escalation. You don’t need to approve every command the agent runs. But you absolutely need gates on the high-risk ones: Migrations, secrets access, IAM changes, destructive operations, deploys.
The New Loop

Here’s what the SDLC looks like when you account for agents:

Spec → Agent Executes → CI Verifies → Human Approves → Deploy → Observe → Spec Updated

That loop only works if you can measure verification capacity, and you can start measuring it next week with three metrics.

Track the median PR review time, but segment it: Agent-generated PRs versus human-written ones. If there’s no meaningful difference, either your agents are producing trivial changes or your reviewers aren’t actually reviewing.

Track the agent failure rate separately. How often do agent-assisted changes get rolled back or cause regressions? This number should be going down. If it isn’t, your specs are too loose.

Stop tracking generic code coverage. Start tracking critical path coverage: Auth, payments, data writes. These are the systems where a missed edge case costs you money, trust or both.

Specs as the Control Plane

The spec needs to be a versioned artifact that lives in your repo. It gets reviewed before the agent runs. It gets updated after every change. If you’re familiar with the model context protocol (MCP) or NIST’s SP 800-218A (the AI community profile for the Secure Software Development Framework), these provide useful scaffolding for connecting agent tooling to policy engines. MCP is particularly interesting here because it allows your verification engines to feed live system state back to the agent, closing the gap between the IDE and the runtime. Instead of the agent operating on a stale snapshot of your infrastructure, it can query the actual state of your canary environment, your schema version, and your secrets vault permissions before it writes a single line of code.

Let me give two concrete examples of what spec-driven prevention looks like in practice. For auth constraints: You specify that token audience must match the environment. Dev, QA, canary, prod. Then you write an automated test that fails if the audience differs. If that spec had existed before our canary incident, the agent would have been constrained from introducing the mismatch in the first place, and the test would have caught it if it did anyway.

For migration safety: You require every database migration to ship with an automated rollback script and a pre-migration check that validates against a schema snapshot. The agent doesn’t get to run the migration without those artifacts present.

A Template You Can Actually Use

I’ve been iterating on a spec format with my team. Here’s roughly where we’ve landed:

Feature: [Name]

Context: [Intent and Business Logic]

Governance and Risk

Risk Tier: [prototype/customer-Facing/regulated]

Change Budget: [Max Files/LOC/commands Allowed Without Escalation]

Data Boundaries: [PII, PCI, GDPR or Specific Files the Agent Must Not Read]

Security Controls: [Automatic secrets redaction in logs, scoped time-bound tokens]

Contracts and Constraints

API/Data Contracts: [Endpoints, Inputs, Outputs, Error States]

Security Policy: [Auth Requirements, Secrets Scanning, Dependency Rules]

Verification and Observability

Acceptance Criteria: [Given/When/Then Scenarios]

Test Plan: [Unit, Integration and Negative Tests Required]

Rollback Verification: [What Specific Metric Proves a Rollback Succeeded]

Agent Execution

Files to Touch: [Scope]

Rollout Plan: [Feature Flags, Canary Steps, Manual Gates]

The ‘change budget’ line is the one most teams skip, and it’s the one that saves you. Think of it as the SRE error budget concept applied to agent execution: Just as an error budget caps how much reliability you’re willing to burn in exchange for velocity, a change budget caps how much codebase surface area an agent can touch before a human has to re-authorize. If you tell the agent it can touch a maximum of 12 files without escalation, you’ve put a ceiling on blast radius before anything runs. Just as an error budget is enforced by the platform (not by hoping engineers remember to check), a change budget should be enforced by a pre-submit hook that counts modified files and blocks the PR if the agent exceeds its allowance. If it’s not automated, it’s not a budget. It’s a suggestion.

What ‘Senior Engineer’ Means Now

Writing clever syntax is losing value fast. The differentiator for senior engineers increasingly is verification-driven design: The ability to write specs that constrain failure, critique agent output for intent mismatches and harden systems against the kinds of regressions that pass all your tests.

This needs to reach your hiring process. I’m not saying throw out your fundamentals interviews. Keep them. But add an exercise that actually tests agent-era thinking. Here’s what I mean. Give the candidate a feature: ‘Add a password reset flow to the user service’. Then show them two specs.

The first spec looks like this:

Feature: Password Reset

Context: Users need to reset passwords via email link.

Contracts:

POST /reset-request → sends email with reset token POST /reset-confirm → accepts token + new password

Test Plan:

Happy Path: Valid token resets password

Integration Test: Email delivery

That spec would have seemed perfectly adequate two years ago. An agent given this spec would produce working code. The tests would pass, and you’d have no idea whether the agent had introduced a token that never expires or stored the reset token in plaintext or allowed unlimited reset attempts from a single IP.

Now show them the senior spec:

Feature: Password Reset

Context: Users need to reset passwords via email link.

Governance

Risk Tier: Customer-Facing/Regulated (handles auth credentials) Change Budget: Max eight files. Agent must not modify auth_provider.go or session_store.go.

Data Boundaries: Agent must not log, print or persist the reset token in any form outside the tokens table.

Security Controls: All new log statements must pass secrets-pattern scan before merge.

Contracts

Post/reset-Request → Sends Email With Reset Token

Token: Cryptographically random, 256-bit, stored as bcrypt hash

Expiry: 15 minutes from creation, enforced server-side

Rate Limit: Max 3 requests per email per hour; return 429

Post/reset-Confirm → Accepts Token + New Password

Token is single-use; delete on consumption regardless of outcome

Password must meet policy (min 12 chars, breach-list check)

On Success: Invalidate all existing sessions for that user

Test Plan

Happy Path: Valid token resets password, old sessions invalidated

Expired token returns 410

Reused token returns 410

Brute Force: 4th reset request within 1 hour returns 429

Negative: Agent did not modify auth_provider.go or session_store.go

Negative: Grep codebase for plaintext token outside tokens table

Integration: Email delivery, token expiry enforcement

The gap between those two specs is exactly where every agentic regression hides. The first spec tells the agent what to build. The second spec tells the agent what it must not break. What you’re really testing for is negative space thinking: A junior engineer reads code and sees what is there. A senior engineer reads a spec and sees what is missing. The constraints that weren’t written. The failure modes that weren’t named. The files that should have been marked off-limits but weren’t. Ask your candidate which failure modes the first spec misses, and why the constraints in the second one exist. You’ll learn more about their engineering judgement from that conversation than from watching them invert a binary tree.

A 30/60/90-Day Plan for Leaders

If you’re an engineering leader reading this and wondering where to start, here’s a sequence that’s worked for me.

In the first 30 days, focus on guardrails. Define which domains agents are allowed to operate in (internal tooling is usually a good starting point) and which are off-limits (billing and payments). Standardize an AGENTS.md file for every repository that makes the rules explicit, and mandate audit logging for every terminal command an agent runs. You need the paper trail before you need anything else.

In the next 30 days, start piloting. Pick high-toil work that nobody wants to do: Dependency upgrades, boilerplate generation, repetitive refactors. Measure the verification gap, which is the time difference between reviewing an agent-generated PR and a human-written one. Put command approval gates on migrations and IAM changes. See what breaks.

By day 90, you should be ready to rewire the process properly. Integrate agent execution into your CI pipelines. Update your performance review criteria to reward spec quality and verification strength. Publish the spec-driven standard as the default for all new services.

The Bet You Are Making

The Stack Overflow numbers paint a picture of an industry that has adopted the tools but hasn’t solved the trust problem: 84% adoption, 29% trust. That gap isn’t closing on its own. The DORA data confirms that speed without stability is just accelerated chaos.

The teams that come out ahead in the next two years won’t be the ones that adopted agents fastest. They’ll be the ones that invested in verification capacity with the same seriousness they invest in production capacity: Tests, policies, audit logs, rollout controls.

The bottleneck has moved. Your process needs to move with it.
June 26, 2026
Mastering the Blast Radius, Deployment Without Fear- Progressive Delivery in Modern DevOps
Traditional software deployments are high-risk, all-or-nothing events. A single faulty release configuration can cascade into outages, increased error rates, customer impact and costly rollbacks. Progressive delivery changes this paradigm by introducing controlled, observable and reversible releases. The traditional ‘big bang’ release — where code is merged and deployed at 2:00 a.m. — is increasingly a relic of the past.

‘Progressive delivery’ is the modern evolution of continuous delivery, designed to reduce the blast radius of new features and decouple ‘deployment’ (moving code to production) from ‘release’ (exposing features to users).

In a progressive-delivery model, the goal is to move from a binary state (on/off) to a spectrum of availability. This is achieved through two primary pillars:
- Release Observation: Monitoring the health of a deployment in real-time.
- Release Control: Using technical levers to shift traffic or toggle features without redeploying code.
Instead of: ‘Deploy and Pray’, Move to: ‘Deploy, Observe, Decide and Expand’

Key Strategies for Risk Mitigation

To deploy without fear, organizations typically employ one or more of the following patterns:
1. Feature Flags: Wrapping code in conditional logic to turn features on/off remotely. Performed using dark launching, A/B testing and emergency ‘kill switches’.
2. Canary Releases: Routing new code to a small percentage (e.g., 1% or 5%) of traffic. Detecting regressions or performance bottlenecks early in routed users.
3. Blue/Green Deployment: Maintaining two identical environments; one is live, one is idle/backup. Zero-downtime cutovers and instant rollbacks to backup version.
4. Shadowing: Mirroring live traffic to a new service without sending the response to the user. Load testing and validating data accuracy under real stress.
The Automation Feedback Loop

Deployment fear usually stems from the uncertainty of new code exposure, mismatched config, logic misalignment but largely by untracked code changes. Progressive delivery eliminates this by automating the ‘promotion’ and ‘rollback’ phases based on service level indicators (SLIs).

For example, a typical automated workflow might look like this:
1. Deploy to a small subset of nodes.
2. Analyze error rates and latency.
3. Check Condition: If error rate <0.1% and latency_p99 <200 ms, then increase traffic.
4. Auto-Rollback: If metrics exceed thresholds, the system automatically flips the traffic back to the stable version
Progressive Delivery Cultural Shifts

Technical tools alone won’t solve ‘deployment anxiety’. Progressive delivery requires a shift in how teams view failure:
- Decoupling Deploy From Release: Engineers can deploy code on Tuesday morning, but product managers release the feature on Thursday when marketing is ready.
- Observability Over Monitoring: Moving beyond “Is the server up?” to “Is the user’s experience degraded?”
- Small, Frequent Batches: Reducing the size of a change makes it exponentially easier to debug if something goes wrong.
What This Means for Modern DevOps

Progressive delivery isn’t just a deployment technique; it fundamentally reshapes the role, responsibilities and mindset of DevOps teams. Instead of being gatekeepers of releases, DevOps evolves into enablers of safe, continuous experimentation at scale.

DevOps can now add ‘a layer of observability’ and adapt to changed releases.

Build → Test → Deploy → Observe → Decide → Adapt

What Changes
CI/CD integrates with observability systems. Deployment success is ‘metric-driven’, not just completed. Rollouts become ‘progressive and conditional’.

Automation Over Manual Intervention: Manual approval slows down delivery and introduces inconsistency. Instead of developing team managing releases, DevOps and SRE’s can take the processes in hand proactively.
- DevOps Owns Feature Release Strategy
Impact
- Developers Ship Code Anytime
- DevOps Controls When and How Users See It
Infrastructure as Experimentation Platform: Infrastructure is no longer static — it has become a dynamic experimentation layer.

DevOps enables:
- A/B Testing at Scale
- Gradual Feature Exposure
- Region-Based Rollouts
Risk Management Becomes Proactive: Instead of reacting to incidents, DevOps actively reduces risk before full impact.
- New Approach
- Small Blast Radius Deployments
- Early Anomaly Detection
- Continuous Validation
Metrics of Success Change

Old Metrics
- Deployment Success Rate
- Uptime
New Metrics
- Change Failure Rate
- Mean Time to Recover (MTTR)
- Rollback Frequency
- Release Observability
Integrating Agentic Ops With Progressive Delivery

Progressive delivery (canary, blue-green, feature flags) minimizes risk during deployments — but still relies heavily on human monitoring and decision-making. Agentic Ops introduces intelligent agents that can observe, analyze and act in real-time. It helps to combine both to create self-evaluating, self-healing deployment systems that can automatically promote, pause or roll back releases based on live system behavior.

Architecture With Agentic Ops
1. Observe (Telemetry Layer): Monitor metrics such as latency, error rate, saturation, logs + traces for deep debugging and business KPIs (conversion, drop-offs). Agents continuously monitor deployment health before, during and after progressive deployment.
2. Analyze (Decision Engine/Agent Brain): Compare against SLOs and baselines, detect anomalies (spikes, regressions) and correlate issues with recent deployment. Analyze historical patterns, recover and prepare a postmortem recovery report. For example:
3. Error rate increased by 12% after canary rollout.
4. Latency regression tied to new API version.
5. Act (Progressive Delivery Controls): Agent triggers a pause, rolls back the deployment, promotes the canary to a full release and adjusts traffic weights dynamically.
6. Report (Explainability Layer): Slack/dashboard updates, RCA summaries, evidence-backed decisions, builds trust in autonomous systems.
Advanced Patterns With Progressive Delivery

1. Adaptive Canary

Agent dynamically adjusts traffic based on real-time health.
5% → 10% → 25% → 50% → 100%

2. Multi-Metric Decisioning

Instead of just error rate, combine latency, errors and business KPIs.

3. Predictive Rollbacks

Agent rolls back before user impact escalates.

4. Learning Systems

Agents improve overtime with past incidents, deployment history and failure patterns.

Key Challenges to Watch For

Non-Determinism: Unlike code, an agent might pass a canary test once and fail it the second time. You need multiple runs per canary stage to gain statistical confidence.

Stateful Memory: If an agent learns or has a memory store, a rollback is harder. You must ensure your progressive delivery strategy includes vector database versioning.
June 26, 2026