devopsy.kinsta.cloud

Tag: infrastructure automation

How to Eliminate DevOps Toil Using Automation Scripts
Engineering teams today are facing under-the-hood productivity killers that are affecting job satisfaction and work efficiency. These productivity killers are called toils. Toils are repetitive, manual, and mundane tasks that provide no long-term value to your application or infrastructure.

Toils are experienced in countless ways in modern development operations. It can be the daily routine of manually deploying code to various environments, or the weekly routine of restarting all services that should self-recover, or the monthly server patching of multiple machines that requires manual management, and the constant administrative headaches that could be prevented with properly managed scripts. All these practices share similar characteristics, which are that they’re manual, repetitive, and scale linearly with system growth.

The overtime accumulations and impact of these DevOps toils are more productivity impactful than what most organizations realize. According to a recent survey, developers and DevOps professionals spend approximately 15 hours per week managing manual tasks such as debugging pipelines and waiting for tests and builds to complete. This represents nearly 40% of a typical work time that could be redirected toward innovation, system improvements, and strategic initiatives.

The solution lies in script automation, rather than the ad-hoc processes that often create more problems than they solve in DevOps workflows. When strategic script automation is adopted as part of the DevOps workflow, it brings about a streamlined workflow that cuts down the repetitive, manual, and mundane tasks, which saves your team valuable time.

This article will show you a practical framework for eliminating DevOps toil through automation. You’ll learn how to find the highest impact automation opportunities, choose the right technical approach for your environment, and implement sustainable solutions that reduce long-term maintenance burden, not add to it.

The Strategic Advantages of Script-Based Automation

There are numerous approaches to solving DevOps toil, ranging from purchasing enterprise solutions to redesigning DevOps workflows. However, script automation gives the most practical and immediate actionable solution for most teams. Generally, scripts offer flexibility, accessibility, and direct implementation, which makes them the first choice to eliminate toils in DevOps workflows.

Scripts have several advantages over more complex automation solutions. First, they use existing knowledge of your existing team. Most DevOps professionals already know Bash, Python, or PowerShell, which can be leveraged on the fly without following the learning curve of specialized tools. Secondly, they can be implemented incrementally, so you can automate one process at a time rather than having to change the whole workflow. Thirdly, they’re transparent and debuggable, so maintenance and troubleshooting is easy even for team members who didn’t write the original script.

Most importantly, scripts can address the specific quirks and edge cases in your environment. While commercial automation platforms force you to adapt your processes to fit their capabilities, custom scripts can be tailored to work exactly how your systems work, not how they should work.

Successful script automation isn’t just about writing code that works. It’s about creating sustainable solutions that genuinely reduce long-term maintenance burden. The most effective automated solutions follow four core principles:
1. Idempotency: Scripts should produce the same result no matter how many times they’re run. Running a deployment script twice shouldn’t create duplicate resources or system conflicts. This principle ensures scripts can be safely re-run during troubleshooting without creating more problems.
2. Observability: Every automated process should provide clear feedback about what it’s doing and whether it succeeded. This means comprehensive logging, meaningful error messages, and status indicators that help operators understand system state without manual investigation.
3. Fail-Safe Design: Scripts should be designed to fail gracefully and provide clear recovery paths. When something goes wrong and it inevitably will, the script should leave the system in a known, recoverable state rather than an undefined failure condition.
4. Documentation Through Code: The best scripts are self-documenting, with clear variable names, logical structure, and embedded comments that explain not just what the code does, but why specific decisions were made.
The most successful automation initiatives begin with what industry practitioners call the “15-minute rule”. If a task takes longer than 15 minutes to complete manually and needs to be performed more than once per month, it’s an ideal candidate for script automation. This threshold ensures you’re targeting genuinely repetitive work while avoiding the trap of over-engineering solutions for infrequent edge cases.

Common examples of 15-minute rule candidates include database backup verification, log rotation and cleanup, service health checks, configuration file deployment, and security certificate renewal processes. These tasks are frequent enough to justify automation effort but simple enough to script successfully without extensive infrastructure changes.

The key insight here is that small, focused scripts compound their value over time. A five-minute script that saves 30 minutes per week represents a 156-hour annual time savings for your team. This time can be redirected toward strategic improvements rather than operational maintenance.

Choosing Your Automation Approach

Your DevOps toil elimination success depends on choosing the right automation way for your environment and team. Don’t jump straight into scripting. Instead, take the time to evaluate your options to avoid costly mistakes and get sustainable solutions.

Language Selection Strategy

Your scripting language should align with three things: team expertise, infrastructure requirements, and long-term maintainability. The best teams standardize on one main language and allow exceptions for specific use cases. This way, you balance consistency with practical flexibility, so most team members can maintain and extend existing scripts and accommodate special cases.

Simple Scripts vs Orchestration Platforms

Start with simple, focused scripts that solve specific problems before going for a broader orchestration solution. A 20-line bash script that automates database backups provides immediate value and minimal maintenance. Orchestration platforms like Ansible or Terraform are great for managing large infrastructures, but they introduce overhead that may not be justified for basic toil.

The decision point is often scope and complexity. If you’re automating tasks within a single system or a few servers, scripts are usually the way to go. If you need to coordinate activities across multiple infrastructures or manage complex dependencies, orchestration tools are more valuable.

Building for Your Team’s Reality

Consider your team’s current skills, time to learn new tools, and existing infrastructure constraints. The best automation way is the one your team can implement, maintain, and extend over time.

Practical Implementation

Moving from theory to working solutions requires a systematic approach that gets quick wins while building towards toil elimination. The best implementations follow a structured methodology that minimizes risk and maximizes early value.

The 15-Minute Rule for Prioritization

Start by identifying tasks that take more than 15 minutes and occur at least monthly. These are your highest value automation targets—frequent enough to justify development time, complex enough to provide real relief. Common examples are log cleanup routines, service health checks, backup verification, and configuration deployments.

Document these tasks before automating them. Understanding the manual process completely prevents automation scripts from missing critical edge cases or error conditions that experienced operators know by instinct.

Essential Patterns

Every automation script should follow four patterns. First, have a comprehensive success and failure log. Second, write scripts that can handle interruptions by either resuming cleanly or giving clear directions for how to recover. Third, validate the script achieved what it was supposed to before marking it complete. Fourth, include rollback for any script that modifies state.

Testing and Validation Framework

Never deploy automation scripts to production. Create isolated test environments that mirror production so you can test script behavior in realistic conditions. Start with dry-run modes that show what the script would do without actually running.

Build up your confidence by rolling out slowly, starting with non-critical systems and then moving on to production infrastructure.

Measuring Success and Scaling Up

Showing the value of your automation efforts requires metrics that matter to both engineering teams and leadership. Focus on concrete improvements like time saved, errors reduced and system uptime rather than vanity metrics. For example, if automating database backups reduces a 45-minute manual process to 5 minutes of automated execution, that’s a 40-minute improvement per occurrence. Multiply that across daily, weekly or monthly to calculate total time recovered for your team. Also, monitor error reduction rates and incident response times as automated processes are more consistent and faster than manual ones.

Beyond individual script performance, successful automation initiatives build momentum through cultural adoption. Survey your team about workload reduction and job satisfaction improvements – high-performing teams consistently report that automation allows them to focus on strategic projects rather than operational firefighting. Track team adoption rates of new automated processes and measure the time between identifying automation opportunities and implementing solutions as this will show your team’s automation maturity.

Once initial scripts are working, scale by expanding into adjacent processes and sharing successful patterns across teams. Create reusable script libraries that standardize common operations while allowing for customization for specific requirements. Establish automation guidelines to ensure consistency and maintainability as your script portfolio grows and create internal documentation so team members can build on each other’s automation work rather than starting from scratch each time.

Wrapping Up

DevOps toil doesn’t have to be part of your team’s daily grind. By identifying and replacing manual tasks with well-designed automation scripts you can free up a lot of time for more strategic work. Start small with simple, high-impact scripts that solve obvious pain points and then gradually expand as your team gets more comfortable. Remember, it’s not about automating everything; it’s about eliminating the toils that consume the most time and attention of your team.

When done right, script automation becomes a force multiplier that not only eliminates toil but enables your team to tackle more complex problems and deliver more value to your organization.
October 10, 2025
env zero Revamps Infrastructure Automation Platform for AI Era

env zero (formerly env0) today revealed it has revamped its infrastructure automation platform to add artificial intelligence (AI) capabilities, including beginning next month a Static Code Analyzer Agent to identify and troubleshoot issues.

Chris Graham, chief marketing officer for env zero, said this iteration of the company’s Cloud Governance Platform will provide a control plane for automating and governing infrastructure-as-code that now takes advantage of AI to better connect repositories and cloud computing platforms using a Model Context Protocol (MCP) server. Originally developed by Anthropic, MCP is emerging as a de facto standard for exposing data to AI applications and agents.

The core capability is also making it possible to build a Static Code Analyzer Agent that detects who is accessing services, the level of compliance, data protection, security and protection of secrets that is being achieved and maintained, which can then be remediated via a pull request that is generated via a single click using the Cloud Governance Platform.

Compatible with Terraform, OpenTofu, Pulumi, CloudFormation, Terragrunt and Kubernetes tools, the Cloud Governance Platform is now making it simpler for AI agents to be used to programmatically provision cloud infrastructure in a way that enables policies to be enforced as code, said Graham.

The ultimate goal is to dramatically increase the pace at which IT infrastructure can be safely provisioned using the context that can be uniquely provided by the Cloud Governance Platform developed by env zero without requiring DevOps teams to master a completely separate user interface, he added.

It’s not clear to what degree organizations are embracing IT infrastructure automation platforms, but the rise of AI agents might soon force the issue. AI agents, while improving productivity, will need a set of guardrails to be in place that prevent them from exceeding the scope of the task that they have been assigned. Otherwise, it becomes probable that sensitive data will be exposed to either an AI agent or the application development team employing them in a way that runs afoul of any number of compliance requirements.

Ultimately, AI agents invoking MCP servers should also reduce the number of scripts and application programming interfaces (APIs) that DevOps teams currently need to support. As the overall complexity of DevOps workflows is reduced, it then becomes feasible to provision more infrastructure at scale at a time when AI is also dramatically improving the pace at which applications are being developed.

Each DevOps team will need to determine to what degree to rely on AI agents to automate tasks, but, for now at least, those teams would be well advised to review any code created by an AI agent. In the meantime, DevOps engineers would be well-advised to start identifying more tasks than can be potentially performed by AI agents as part of a larger effort to reduce the total amount of manual toil that, in addition to burning out teams, also creates more bottlenecks that slow down the pace of application development more than anyone cares to admit.

September 30, 2025
AI Agent Onboarding is Now a Critical DevOps Function
You wouldn’t throw a new engineer into production with no documentation, no context, and no support. Yet that’s exactly how many teams are treating AI agents. As autonomous systems and LLM-based agents begin taking on responsibilities once reserved for humans, their onboarding is emerging as a DevOps function just as vital as CI/CD, observability, or access control.

AI agents aren’t magic. They need well-defined environments, scoped permissions, data pipelines, and feedback loops to succeed. If DevOps doesn’t take ownership of onboarding them, you’re not deploying intelligence — you’re releasing liability.

AI Agents are Entering Production Faster Than You Think

AI agents are already helping teams triage tickets, summarize logs, provision environments, run basic tests, write code snippets, and answer developer questions. But here’s the rub: most of them are being deployed without formal ownership. A developer experiments with an LLM wrapper, gets useful results, and suddenly it’s a critical part of the toolchain. No review. No plan. No infrastructure.

That’s a problem.

Once an agent starts interacting with your real systems, the risks escalate quickly. Is it

rate-limited? Does it have access to sensitive environments? What happens when it fails silently or starts making invalid assumptions based on outdated context? Who logs the output? Who reviews the audit trail?

These aren’t academic questions. Teams are already running into issues where agents overwrite configs, cause noisy retries, or misroute customer support requests—not because they’re flawed, but because nobody treated their onboarding as a formal deployment.

Remember the Replit.ai incident with the deleted database? Exactly?

For better or for worse, DevOps is the only team with the discipline and systems thinking to treat AI agents as first-class deployable units. Without that lens, you’re building a house of cards with great demos and terrible ops.

Treat Agents Like New Engineers — Because They Are

If a junior engineer joined your team tomorrow, would you give them root access and hope for the best? Of course not. You’d give them a sandbox, read-only access to production logs, limited write privileges, a checklist of onboarding tasks, and clear escalation paths.

That’s exactly how AI agents need to be onboarded.

They need structured inputs, constrained actions, guardrails, monitoring, and progressive trust. They need to be versioned, tracked, and retrained when the environment changes. You need to know which model version is responsible for which action, and be able to reproduce its decision path.

DevOps teams already do this for humans. They use identity and access management (IAM), canary deployments, secret rotation, observability stacks, alerting rules, and incident response playbooks. Those same tools should be extended to agents.

Onboarding is an Operational Contract

Too many orgs assume that because LLMs are pretrained, they don’t need onboarding. Wrong. You’re not training the model, you’re configuring the agent.

Onboarding is the moment where an AI system stops being a general-purpose model and starts being a specific actor in your environment. That involves:
- Connecting it to the right APIs and systems
- Defining the scope of what it should and shouldn’t do
- Providing context, it can be referenced reliably
If the system changes, the agent needs retraining or re-prompting. If the security policies evolve, its access needs auditing. If an outage occurs, it needs logging hooks and rollback support.

DevOps already knows how to build and maintain contracts between services, infrastructure, and teams. The same rigor should apply to agents. If your onboarding process for a human engineer takes three weeks, but your onboarding for an AI agent is a Slack message with an API key, you’re ignoring the operational debt you’re accumulating. And it will come due.

Observability and Feedback Loops are Non-Negotiable

Deploying an agent without observability is like flying blind in a storm. You need to know what it did, why it did it, and what the outcome was—in real time.

That means structured logs, traceable decision paths, and dashboards that capture the agent’s behavior over time. Not just success/failure metrics, but patterns: How often does it repeat tasks? Where does it hesitate? What data sources lead to hallucinations, and what conditions exist when that data is being extracted?

Even more critical: you need feedback loops. Human-in-the-loop review, reinforcement learning from human feedback (RLHF), or at the very least, a way to flag and annotate failures. An agent that never receives feedback is an agent that will never improve.

Just like CI/CD systems rely on test coverage, linting, and rollbacks, AI agents need regression checks. Changes to prompts, context, or dependencies should be tested in pre-prod, versioned, and approved before rollout. And yes, rollback mechanisms should exist if something goes sideways.

You can’t control everything a generative agent might say or do. But you can control the infrastructure it operates in. Observability turns black-box behavior into manageable risk.

DevOps Should Own the Agent Lifecycle

The AI agent lifecycle isn’t a hackathon project anymore. It’s a real deployment pipeline. That pipeline needs owners, automation, and safeguards.

DevOps should own that lifecycle.

That includes staging, testing, promoting, and retiring agents just like any other service. It includes documenting context sources, prompt templates, version numbers, and access patterns. It includes maintaining CI/CD pipelines for updates, retraining, and observability checks.

You wouldn’t let a new API go to prod without a runbook and incident policy. Why should an agent be any different?

This shift is already underway. Platforms like LangChain, AutoGen and Microsoft’s Autogen Studio are bringing software engineering discipline to the agent space. But platform features alone don’t guarantee maturity, despite all the fears about DevOps specialists being one of the professions that’ll disappear due to AI. More than just the operational part, DevOps is the cultural layer that ensures tools are used responsibly.

As generative systems become persistent actors in cybersecurity and infrastructure in general, they deserve lifecycle management, not just one-off deployments. Ownership isn’t optional. The risks are too high.

Closing Thoughts

DevOps is now about shaping how intelligent agents interact with systems, data, and people. That means new responsibilities—but also new leverage.

Done right, agent onboarding becomes a competitive advantage. Teams that treat agents as real actors, with structured onboarding, scoped permissions, and rigorous observability, will ship faster and break fewer things. They’ll identify hallucination risks before they hit customers.

They’ll iterate on prompts with the same precision they bring to tests.

What’s more, if your DevOps team isn’t planning how to onboard and manage AI agents today, you’re already behind. Start building the scaffolding now. Because your next teammate might not be human — but they’ll still need your help to succeed.
August 28, 2025
System Initiative Adds AI Agents to Infrastructure Automation Platform

System Initiative today announced it has added autonomous artificial intelligence (AI) agents to its infrastructure automation platform

Company CEO Adam Jacob said AI agents will now be able to interact with digital twins of IT infrastructure environments that System Initiative creates to enable DevOps teams to propose and safely execute changes to IT infrastructure.

DevOps engineers using natural language prompts can simply tell the System Initiative platform what outcome they want and the AI agent determines how best to accomplish that task. DevOps engineers then review that plan, which, if approved, will be executed by the AI agent, said Jacob.

AI agents will be able to complete tasks in a matter of minutes that previously would have required weeks for DevOps teams to manually perform, he added. AI agents will also be able to propose validated changes to IT infrastructure environments to optimize workflows and eliminate bottlenecks, said Jacob.

Additionally, DevOps teams can import existing infrastructure into the System Initiative platform to create a real‑time digital twin of every element of their IT environment that they can then use AI agents to help them explore and uncover hidden relationships between resources, noted Jacob. The digital twin platform created by System Initiative essentially functions as a knowledge graph that AI agents can use to understand the relationships between resources, he added.

DevOps teams will also be able to define custom guardrails and rules to limit the scope of the functions any AI agent is allowed to perform, noted Jacob. Built‑in policy checks and tests ensure every change adheres to compliance and operational requirements, he said.

In effect, AI agents will be integrated into existing DevOps teams without requiring any changes to existing workflows, added Jacob.

Ultimately, the goal is to make it much simpler for DevOps teams to employ AI agents to consistently employ best infrastructure management practices in a way that, for example, eliminates misconfigurations that might be exploited by cybercriminals, said Jacob.

As more menial infrastructure management tasks are automated, DevOps teams should also have more time to research how to optimize their environments for complex applications, he added.

It’s not clear to what degree DevOps teams are ready to rely on AI agents to manage IT infrastructure, but it’s apparent that, given the number of misconfigurations that exist, existing workflows are deeply flawed. It’s simply too easy for developers using tools such as Terraform to make a mistake. Of course, AI agents might also be prone to making mistakes, but hopefully, there will be fewer of them, especially as those AI agents become more familiar with the specific IT environment they have been tasked to automate.

No matter how IT infrastructure is provisioned, there will always be a requirement for it to be reviewed by a DevOps engineer. The challenge and the opportunity now is to automate the management of IT infrastructure at a level of scale that might not have not too long ago seem unimaginable.

August 27, 2025
Cultural Debt vs. Technical Debt in Infrastructure Automation
Infrastructure is humming, deployments are fast and dashboards are green. But something still feels off. Despite the technical efficiency, teams might be slowed down by recurring issues, misalignments, or resistance to change. These aren’t always rooted in code.

While technical debt, the accumulated cost of quick fixes and legacy code, is a familiar concept, cultural debt often goes unnoticed. It surfaces through unclear communication, misaligned incentives, or outdated norms that no longer support evolving goals.

Both forms of debt can quietly undermine DevOps practices, as they limit the very automation and agility they’re meant to enable.

In this article, we’ll explore the differences between technical and cultural debt, how each impacts DevOps and infrastructure automation and practical approaches to identifying and addressing them.

Defining the Terms

Technical debt means the trade-offs made during development to prioritize speed over long-term quality. These shortcuts might be intentional or the result of time pressure, resource constraints, or evolving requirements. While they can accelerate delivery in the short term, they often lead to brittle systems, increased maintenance burden and operational inefficiencies down the line.

Technical debt can manifest in many ways: hardcoded configurations that don’t scale across environments, inadequate or missing monitoring, outdated provisioning scripts, or reliance on manual processes that should have been automated. Over time, these issues can drag on velocity and complicate troubleshooting.

On the other hand, cultural debt is less tangible but equally impactful. It represents the accumulation of unresolved issues in team interactions, which can mean mismatched expectations, ineffective communication and outdated cultural norms that no longer support the team’s goals.

Examples include resistance to adopting new tools, siloed teams with poor cross-functional collaboration and unclear ownership of infrastructure components. Like technical debt, cultural debt builds silently and can impede progress if left unaddressed.

Why Cultural Debt is the Silent Killer in DevOps

DevOps is more than pipelines and automation. At its core, it’s a cultural transformation.

It’s about breaking down silos, fostering collaboration, building fast feedback loops and promoting shared responsibility across development and operations. Even the most advanced tools and workflows struggle to deliver their full value without the right cultural foundation.

This is where cultural debt becomes particularly insidious. Unlike technical debt, which is often visible in logs or performance metrics, cultural debt hides in team dynamics and day-to-day interactions.

It shows up when teams resist adopting new tools, cling to “the way we’ve always done it,” or hesitate to take ownership of shared responsibilities. These patterns create friction, slow onboarding and make scaling CI/CD efforts far more complex than necessary.

Moreover, cultural debt tends to compound quietly. A single unclear decision-making process or a lack of psychological safety can ripple outward, leading to missed handoffs, redundant work, or breakdowns in trust. Over time, this erodes the very principles DevOps relies on: agility, transparency and continuous improvement.

Addressing cultural debt means creating an environment where teams can adapt together. Without this cultural alignment, the technical side of DevOps is left propping up an unstable foundation.

How Technical and Cultural Debt Feed Each Other

A dysfunctional team culture, marked by poor communication, unclear ownership, or fear of failure, often leads to rushed decisions and technical shortcuts. These quick fixes accumulate into brittle systems and fragile automation.

As technical debt grows, so does the hesitation to make changes. Teams avoid touching fragile pipelines, unsure of the impact or who’s responsible. This reluctance reinforces siloed behavior and deepens mistrust, further degrading the culture.

It’s a vicious cycle: bad culture leads to poor code, and poor code makes the culture worse.

Breaking the loop requires tackling both, cleaning up systems while fostering a culture that supports shared responsibility and continuous improvement.

Diagnosing and Measuring the Debts

Recognizing technical and cultural debt early is key to preventing long-term drag on DevOps performance. While technical debt often leaves a visible trail, cultural debt tends to require a more observational, qualitative approach.

Signs of Technical Debt

These indicators often show up in system performance and team workflows:
- Frequent outages or instability after changes
- Long release cycles or painful deployments
- Manual tasks where automation should exist
- High effort to onboard new infrastructure or services
Signs of Cultural Debt

Cultural issues can be more subtle, but they have a tangible impact on collaboration and growth:
- A culture of blame instead of shared accountability
- Knowledge hoarding or unclear documentation
- Avoidance of retrospectives, feedback, or experimentation
- Siloed teams with limited cross-functional interaction
While tools like DORA metrics can help measure delivery performance (e.g., deployment frequency, lead time for changes), cultural debt often requires listening to team dynamics, observing behaviors and surfacing what’s not being said.

Technical problems may flag a deeper cultural issue, and addressing both starts with awareness.

Strategies to Address Both Debts Together

Addressing technical and cultural debt in isolation only offers a temporary fix. Both must be tackled in tandem to build sustainable DevOps practices because culture drives how technology is built, maintained and evolved.

Start with the code, but don’t stop there.

Make space for regular refactoring and cleanup cycles, as they are excellent opportunities to reassert quality standards. Strengthen test coverage and invest in observability, not just for performance but also to reduce fear when deploying changes. And when automating, be deliberate. Automation should solve real problems, not just shift complexity around.

But even the cleanest codebase won’t thrive in a broken culture. Creating psychological safety is foundational: people need to feel safe speaking up, admitting mistakes and asking questions.

Leaders should actively model DevOps values like transparency, shared responsibility and continuous improvement. Practices like cross-functional pairing and meaningful retrospectives help reinforce collaboration across silos.

Tools can support this work, platforms like Terraform Cloud, Spacelift, GitHub Actions, or Jenkins enable better infrastructure management and automation. But tools don’t solve culture; they amplify it. A strong foundation of trust and clear communication is what allows these platforms to truly accelerate delivery.

Wrapping up

Technical and cultural debt both undermine DevOps success, but cultural debt often lays the groundwork for technical shortcuts and long-term inefficiencies. While it’s easier to spot failing pipelines than failing team dynamics, it’s usually the culture that shapes how systems evolve.

Investing in tooling and automation is important, but without a healthy culture to support it, those investments won’t scale. Treat culture as a core part of your infrastructure strategy, as it’s how you build and grow.
June 9, 2025
Understanding the Infrastructure Automation Maturity Model

Nearly every company wants to evolve its digital transformation and increase the pace of software delivery and operations. And infrastructure automation is a new means to achieve these ends. It supports the emerging discipline of platform engineering and can increase scalability and reactivity to unforeseen events, enabling software ecosystems to be more anti-fragile. Increased infrastructure automation also helps free up engineers from focusing too much on managing DevOps when they really should be building out new features.

But introducing more infrastructure abstraction can be met with certain challenges. This is often because departments have a wide range of cultures, processes and tooling already in place. With varying maturity levels across teams, it can be challenging to gauge where a company is before attempting to introduce new automation across the board. And as further evolution may require significant changes, having a baseline is key to planning for the future.

I recently chatted with David Williams, senior vice president of market strategy at Quali, about an internal technique they’ve been using at Quali to gauge the maturity of their clients’ infrastructure automation. Their infrastructure automation maturity model breaks down the journey into four unique stages, moving from the least-mature ad-hoc automation to the most-mature autonomous solutions. Below, we’ll review this model and consider what questions need to be asked to ascertain where your team sits on the spectrum. We’ll see how enterprises can leverage such a model to hone in on improving their software operations.

Four Questions to Determine Infrastructure Maturity

DevOps is still a wild west in terms of how people approach it, said Williams. So before attempting to mandate sweeping cultural changes, it helps to first determine the base level of DevOps maturity within an organization. Here are four questions that can quickly assess the state of infrastructure automation within an organization.

1. What Culture Already Exists?

First, it’s good to evaluate the culture that already exists. How do teams deliver value and how is this value measured? Perhaps throughput or code quality is measured with specific metrics or SLAs. Understanding the DevOps attitude (and aptitude) within the organization will highlight pre-existing capabilities and show how teams collaborate.

2. How Are People Organized?

The genetic makeup of teams can dictate much about development and operations. How are teams organized? Are they large, siloed teams? Or are they two-pizza-sized teams? Understanding team organization and communication patterns are necessary to identify bottlenecks.

3. What Processes Are Already in Place?

Next, consider the development timelines and procedures already established and in use. Are teams applying SAFe, Lean, Agile or Scrum methodologies? What are these exact processes and how mature are they? Answers to these questions will demonstrate general release cadences and how teams drive productization.

4. What Infrastructure Automation is Already Being Used?

Lastly, you’ll need a window into what level of infrastructure automation, if any, is already in use today. How is it delivered? How is it managed? How is the success of this automation measured, and what is the overall outcome? You will likely find that automation is typically quite siloed and teams have differing approaches. This auditing is vital to identify what works well and what doesn’t.

The Four Levels of the Infrastructure Automation Maturity Model

Asking the above questions will quickly determine an organization’s “readiness state,” said Williams. But how do these answers map to the overall maturity of the organization’s infrastructure automation? Well, he shared four basic levels of infrastructure automation maturity that are commonly found in practice throughout today’s software development environments, ranging from bespoke scripts to more robust self-defining automation.

Level 1: Ad-hoc

The most basic level of infrastructure automation is ad-hoc integrations involving a mix of old and new technology. This level is chaotic and fragmented, with varying processes used throughout the organization. These bespoke integrations and scripts also may not have excellent visibility, limiting the potential for reusability within an organization.

Level 2: Automation

Next is having automation that is specifically tied to a certain technology stack. Nowadays, organizations are commonly hybrid multi-cloud, having adopted automation piecemeal with each new stack. This often results in multiple silos of automation stacks that are blind to each other. At this maturity level, one automation might fire off another automated procedure and so on, but engineers aren’t 100% aware of how they influence each other.

Level 3: Frictionless

The next level of infrastructure maturity begins to take the reins with more intentional automation adoption. At this stage, organizations embrace an internal developer platform, similar to Spotify’s Backstage, that helps engineers deliver and manage infrastructure more seamlessly. This tooling is more advanced than other levels integrating key commands into the developer’s IDE. Doing so eliminates the need to switch between environments and reduces friction.

Level 4: Self-Defining

Finally, the most advanced form of automation involves deeper intelligence within application operations, including the ability to dynamically optimize infrastructure. At this stage, infrastructure is self-defining, meaning production applications can automatically adapt, be self-governed and enable bursting and non-bursting capabilities. Such intelligence is typically proprietary to cloud providers, said Williams, yet companies often desire this intelligence inherent at the application level. Williams describes this stage as the “intelligent allocation of resources in line with application life cycle requirements.”

Benefits of Self-Defining Infrastructure Automation

Self-defining infrastructure automation has many benefits for companies, including having the infrastructure in line with application needs, scalability and measuring usage to optimize cloud spending. Many points within the software development life cycle could warrant increased automation, such as infrastructure-as-code (IaC), deployment release engines, configuration, traffic scalability, cloud management, containerization and documentation. But it’s not until you open conversations around infrastructure automation maturity and understand who is responsible for managing it that you can hone in on what areas can be improved.

February 14, 2023
Enabling Developers with Infrastructure Automation

Enabling developers with infrastructure automation

Being first to market with new and improved applications and services is how successful organizations grow and differentiate business offerings. Continuous software delivery means everything is fluid and constantly changed to enable the speed and agility required to ensure applications deliver on increasingly higher consumer satisfaction expectations.

Everything released and rolled out needs to be evaluated and updated frequently with new capabilities and features added as requirements change. DevOps, continuous integration and delivery practices play a growing part in the streamlining of software pipelines and accelerating software releases, however for enterprises, the infrastructure supporting the pipelines remains siloed, provisioned (potentially, multiple times) at each stage of the software delivery pipeline.

While infrastructure remains a skill intensive, stage based practice, there is no greater inhibitor to pipeline execution speed and agility and the requirement to scale DevOps to meet the needs of the business.

The result is that IT organizations cannot keep pace and struggle to provide development and testing teams with the infrastructure environments they need, while maintaining security, compliance and efficiency. To help achieve speed and control, many DevOps-focused companies are turning to infrastructure automation. The goal of such an approach is to give DevOps teams access to the environments they need in a timely manner, all while maintaining control and oversight of costs and governance.

The way businesses operate today, DevOps needs production environments available immediately. Developers cannot afford to wait days or longer when starting new projects. Some businesses try to address the challenge by building automated scripts to set up environments.

The problem is that automated scripts are typically custom, hand-coded and require increasingly scarce skills. When changes to environments are needed, custom scripts need the same high level of skills to understand, evaluate, change, and test environments taking crucial time from pipeline execution.

DevOps teams need instant access to infrastructure. The solution must be, application pipeline centric, automated, on-demand and self-service. It must be a platform that not only delivers and manages environments throughout the pipelines but integrate with existing infrastructure tools, subsuming what they create to ensure enterprises meet the demands of the business without inhibiting application software development innovation.

The role of immutable infrastructure

Enabling developers with infrastructure automation is the concept of reusable automation and immutable infrastructure. The concept for immutable infrastructure is to build the infrastructure components to an exact set of specifications. If a change to a specification is required, then a whole new set of infrastructure is provisioned based on the updated requirements, and the previous infrastructure is taken out of service as it is obsolete.

How does that play into automating infrastructure for DevOps teams? Continuous delivery uses automation to make deployments routine and repeatable. To achieve this, delivery is built on the concept of versioning and automation to push deployments to variable environments on a development pipeline.

Immutable infrastructure comes into play in that IT-heavy infrastructure chores, such as implementing updates and installing patches, are eliminated. If an infrastructure element needs an update or must be replaced, the old elements are replaced by new entities that have already made these changes and been incorporated, tested and verified.

Bottom line: Reusable immutable building blocks make it easier to deliver the resources developers really need.

Addressing governance

Infrastructure automation should be used to ensure governance. However, traditional cloud governance approaches are limited by features, with infrastructure automation achieved through complex scripting. This results in the automation becoming a major maintenance challenge and a time commitment. With multi-cloud environments, complexity multiples.

The smart approach is to focus infrastructure automation on managing governance across common cloud resources. Putting infrastructure automation in the hands of DevOps practitioners does not mean relinquishing governance control, but ensures corporate compliance is maintained and enforced with policies controlling usage, security and cost. This prevents cloud sprawl without slowing down development cycles.

Summing up

Meeting todays growing DevOps infrastructure needs is a time-consuming task for IT organizations. It slows down every stage of the application development pipeline. The need is for a solution that provides developers and DevOps practitioners access to the cloud infrastructure and application environments they need on demand. This is accomplished using an Environments as a service (EaaS) platform, accessible through a GUI portal, command line interface and existing CI/CD tools.

A modern environment automation platform must be designed for all team members of a DevOps practice to automate the requesting, provisioning and lifecycle management of infrastructure environments. The platform must integrate without impacting efficiency or limiting innovation. It must deliver environments on-demand with self-service automation capabilities and oversee and enforce governance compliance. This solution is the answer to organizations seeking to streamline the continuous software delivery pipeline for application development, testing and release to production.

November 15, 2021
IT Automation Tools Are No Longer Enough

As Chris Gardner—who conducted the Forrester Wave on Infrastructure Automation Platforms—stated, “Infrastructure automation isn’t just on-premises or in the cloud. It’s at the edge and everywhere in between.” And that’s because, well, it needs to be.

It shouldn’t come as a surprise to anyone reading this that IT automation is a must-have for organizations today. With infrastructure spread across on-premises data centers, multiple clouds and a multitude of technologies, the complexity of IT long ago outgrew manual management. Automation is the only way to keep pace with constantly changing environments that need enhanced security, greater compliance and be easily adaptable to changing business and market conditions.

Because of these needs, the market has been relatively quick to adapt and embrace automation technology. According to a recent study by Forrester Research, 56% of global infrastructure technology decision makers have implemented/are implementing or are expanding/upgrading their implementation of automation software. Many of these IT organizations have a trusted tool in place and have been automating scripts and tasks for years.

Along with outgrowing managing infrastructure by hand, the industry is also quickly outgrowing automation tools. The automation market has now reached a point of maturity where an automation tool alone is not enough. In order for automation to continue to have an impact on an organization and keep pace with the speed IT changes, organizations need a platform that allows them to create a culture of automation.

Automation as a Trait, Not a State

Typically, when an organization acquires a new technology, they try to adapt it to their needs and how they work. Automation technology, along with most technology tools, does exactly that—it lets you adapt it to how you want to use it, and performs tasks as desired. But ultimately, successful automation comes down to more than just supporting task execution. Enterprises have realized that they need to support the behavior of automation in order for it to continue to scale.

As opposed to automation being a state, use automation as a trait—something that is done naturally without thought. The analogy we like to use is one of learning a new language. When you learn conversational Spanish, your mind still thinks in English, then translates what you want to say into Spanish based on what you have learned. Yes, you improve over time, but when you start to dream in another language, that is when it has become second nature. It is no longer something you have to do, you just simply do it.

When automation becomes second nature, that is when it can make the biggest impact and will help address a continually evolving infrastructure. As a discipline, automation is not technology-specific. As much as we talk about the technology behind it, it actually has little to do with automation. The technology itself is the easy part, the part that is reliant on a tool. Many have reached a point of maturity with using tools and are now working toward IT staff viewing automation akin to testing or documentation—while we may not always do it, we know we should.

Having automation become second nature is easier said than done, and something organizations are struggling with, realizing that a tool alone will not support this effort. It doesn’t provide the coordination and collaboration that is desperately needed to unlock the next level of automation capabilities. We need more than a tool to continue to scale automation and spread it to new areas of IT.

The Next Phase of Automation and What to Look for

There are multiple personas within an organization that are paramount to successful business-wide automation, and, to be blunt, automation must cater to all of them. Between developers, those trying to simply consume automation or sys-admins executing automation of mundane tasks, they all need something that works for them—but can also work consistently across these personas. Organizations need a common automation language across the entire enterprise and the hybrid cloud that creates a standardized experience for solving problems (and solving them only once).

Not only that, but they need a repository or catalog where everyone can find an answer to what they need, along with easily accessed integrations. Part of why automation is growing so rapidly is because of how many technology vendors have embraced the concept and, in turn, are working with automation vendors to integrate their technologies and make them easier to use.

To continue to spread automation to new areas of a business, organizations need a single repository to more securely search and find necessary modules. Additionally, having a shared repository helps create an automation community within an organization. It brings different personas together to find what they each need, fosters collaboration and ultimately encourages sharing automation specific to their organization. It also provides a place to find integrations that they may not even have been aware they needed.

When preparing for the next step in their automation journey, organizations should look for automation platforms that provide access to analytics, specifically those that can tell you how your automation is being used and where. As the adage goes, knowledge is power, and the more you know about your automation, the more successful it can be. More importantly though, if you are trying to create it as a behavior, analytics let you track and measure progress. It allows you to better understand what is working, what groups are thriving and what areas need improvement. Using automation analytics enables organizations to create a system for enabling standardized practices.

On the whole, a successful automation initiative needs a supporting automation platform to provide coordination and embrace collaboration between different groups. Simplification is also essential for making automation more accessible across these groups, especially in the face of competing needs and varying levels of engagement. Scaling automation successfully applies both to technologies, as well as the users that want to benefit from automation.

Strategy for 2020

There is no such thing as a perfect tool, and don’t let anyone try to tell you otherwise. The good news is it’s not about having a perfect solution, it’s about the experience that can be gained from a solution. Every organization is unique and there is no one-size-fits-all option.

What is needed when developing an automation strategy that takes organizations into the future is a foundation for growing that automation. Automation platforms offer so much more than a tool alone, allowing it to be more easily scaled and being leveraged how the organization needs. It may never solve all problems but will take away a lot of the efforts. You need to find a vendor that is not selling a tool—but selling automation as a whole.

— Richard Henshall

March 23, 2020
Speaking the Language of Applications
I arrived early to a meeting, and the pleasant security guard at AT&T suggested I take a look at their innovation museum while I wait. Among motion picture sound technology and life-size photos of switchboard operators on roller-skates stood a transistor prototype. The transistor was invented in Bell labs in 1947, and it changed the world. Steve Wozniak considered it the greatest innovation of all time in his 2014 article.

But almost nobody needs to think about transistors anymore. The transistor had to become transparent and abstracted, so the next generations can build great things in higher level. The transistor made a huge influence on our lives as an enabler, but if people had to keep talking about transistors it wouldn’t be possible.

Abstracting technologies is key to advancing to a higher level, making it possible to solve real business problems.

The language of a client-focused business is applications. If we want to make a business execute faster these days, we need to make the development and release of its applications execute faster. But one thing stands in our way: the infrastructure. Applications reside on infrastructure–they are installed on servers, bundled in containers, on cloud instances, in networks, using storage.

If we want to speak the language of applications, we need to abstract them from the infrastructure, and make the infrastructure transparent.

The concept of abstracting applications is what stands at the basis of every new approach to application development and release around us.
- Containers are all about abstracting services from specific infrastructure and making them mobile.
- Serverless takes it to the extreme.
- Public cloud makes it easier not to worry about owning physical infrastructure.
Technology screams, “Think applications, soon you will not have to worry about infrastructure!”

And yet, most of us mix applications and infrastructure in an unhealthy way when we try to automate them. It is unhealthy because mixing applications and infrastructure prevents reuse of automation assets and makes it very difficult to connect infrastructure usage to a business context.

Understanding how mixing applications and infrastructure is manifested requires looking back into the evolution of infrastructure and application automation.

The Evolution of Infrastructure: Manual Deployment and the Inception of Its Automation

In the beginning there was manual. We got a server; it was sitting on a shelf. We got the application artifacts, maybe on a CD or a USB stick. We installed the application on the server. Our application was ready for use. When we needed to upgrade, we went back to the shelf and installed the update on our server. It was very hard to make this operation faster.

This is where automation of applications and infrastructure started.

Behind any automation tool or standard stands a model. This model reflects how its creators originally perceived the problem they were trying to solve. It consists of the entities they defined as first-class citizens, the operations they wanted to apply on these entities, their attributes.

“All models are wrong, some are useful,” said George E. P. Box in 1976. This is true for automation models too. Automation models are usually nice and clean when we start off. But as the problems we try to solve transform, our models need to grow and adapt. Until at some point they are not useful anymore and we need to replace them with next generation modelling approaches.

The Configuration Management Model

Configuration management tools such as Puppet and Chef were originally built to manage application deployment and configuration on static infrastructure. This new approach to Ops optimized manually updating applications on servers. Better network connectivity made it possible for this model to thrive, with the ability to configure applications on servers remotely.

But within a decade we witnessed the fast rise of immutable infrastructure. Immutable infrastructure is an approach to deploying applications where the infrastructure components are replaced instead of updated. This approach somewhat pulled the rug from under the configuration management model. With each update or change, the application must be re-deployed. This forced adding dynamic infrastructure provisioning to the configuration management model. Combining infrastructure automation and immutable infrastructure concepts into the configuration management model is a difficult task, since this model was never designed to accommodate something like that.

The Infrastructure-as-Code Model

Pure Infrastructure-as-Code tools were built to provision infrastructure dynamically. Terraform is an example for an open source tool that does this well. This model is created with infrastructure in mind.

Whether we look into Terraform, AWS, CloudFormation or ARM templates, we will see the infrastructure is the starting point (defining network connectivity, VM instances and services) and the applications are an implicit addition. They are artifacts that happen to be deployed on and are tightly coupled to the infrastructure. We may look at a component that says “App A” but it actually stands for “specific infrastructure with app A installed on it.”

This seems natural, because it’s the same way it worked in the good old physical-infrastructure world: We fetch a box, we install something on it. We focus on the infrastructure, because this is our starting point. The application is the artifacts we put on the infrastructure and scripts we run after we provision it. Presumably, the application cannot live without something to be on.

Infrastructure-as-Code tools evolved the way open tools do–their base model was extended by the community because there was a need to automate applications deployment and not just infrastructure. But it is becoming difficult to extend this model to reflect a world where applications are built to move between clustered servers, and the starting point is no longer the infrastructure, but rather the application’s requirements.

Mixing Infrastructure and Applications

Trying to fit infrastructure provisioning and application deployment in a single model that was originally designed for just one of them creates issues. Infrastructure is lower level than applications. Applications reside on the infrastructure. The same app can reside on different infrastructure configurations. This critical relationship is completely flattened if you mix them.

Moreover, from a business perspective it is important to make applications explicit. It’s our product. Our website. Our mobile app. The applications need to be the center of the attention and discussion. This kind of modelling puts applications in the shade of infrastructure.

Four Great Reasons to Model Applications as a First-Class Citizen Right Now

The models we use today are broken, and it is time to evolve. No matter if your applications reside in containers, or on cloud instances, or include native cloud services or a mix of them. Applications should be a first-class citizen, and not some artifact installed as an afterthought on the infrastructure we started from.

The notion that applications should be abstracted from infrastructure doesn’t eliminate all the challenges of infrastructure provisioning and application deployment, but it creates a foundation for better and more scalable automation.

1. Achieve Application Re-Use

Modelling applications as a first-class citizen is the only way to make applications a re-usable element throughout the development and release pipeline.

If there’s one thing I learned in my automation journey, it is that if automation doesn’t get frequently and consistently reused, you don’t get value for your money.

2. The Path to Multi-Cloud

Abstracting application is also the best approach for moving application workloads between cloud providers. If everything starts from infrastructure, we need to start from scratch when we move. If our applications are abstracted and defined separately—this definition stays the same when we move, only the infrastructure changes.

3. The Path to Breaking Silos

The vision of Dev, Sec, and Ops all dealing with the same language (code) is wonderful. But bringing everyone to the same code pool is not always easy. Applications and infrastructure are not the same thing, even though we can code both. Mixing them makes them unnecessarily hard to build and troubleshoot. It forces us to deal with fuzzy boundaries when it comes to responsibility for this code. Basing automation on a model that organizes the world in a way the different stakeholders can understand it, can help overcome this barrier.

4. Connect Infrastructure to Business Needs

Infrastructure-as-Code is a huge breakthrough, but the bigger story is about making application development and release faster. This doesn’t undermine the importance of infrastructure; infrastructure is the foundation of everything. But if we want to solve business problems, we need to talk the language of the business, and that’s the language of applications. Our automation should be built in a way that reflects this.

— Maya Ber Lerner
July 26, 2019
Using Automation to Control Cloud Costs
By building in cost control automation from the very beginning, you can keep your cloud costs low while encouraging innovation.

Infrastructure automation has become vital for every modern sys admin and operations team. Applications and systems are evolving from single-host mainframes to distributed microservices architectures. This automation has gone from basic operating system installation and setup to full-blown multistep deployments of production code from a single developer’s commit. By automating these mundane processes and eliminating the element of human error, production systems are much more stabile than ever before.

But why stop at automating deployments? There are other elements that need to be automated, too. One of the main ones is cost control.

Continually rolling out new infrastructure without ever taking a step back to analyze the cost often leads to the panic-driven cloud-bill-based phone calls from your finance department. Instead, applying automation to your cloud costs can help you to save money, so you never have that unpleasant surprise of a major bill.

Scaling Up Without Ever Spinning Down

Developers and operations teams often use infrastructure automation early in application development and deployment processes to get servers and databases deployed and functioning. Modern automation tools aren’t just powerful, but they’re also quick to deploy and fit into your current workflow. This is fantastic, but the problem is the automation effort can taper off once the environments are running—too often, users and teams move on to the next project before figuring out a way to keep costs from getting out of control. Then it’s too late, and users simply accept that money needs to be dumped into the deployment pipeline to keep everything on task.

Easy-to-use automation is the key to spinning these environments up efficiently. It also can be key for keeping the costs of these environments low. Sure, you may need to keep the production systems scaled up for maximum application performance and customer satisfaction, but what about the test lab, sandbox environment, dev systems, UAT servers, QA deployments, staging hosts and other pre-production workloads? Having giant environments with system sizes that match production can be useful for some testing, but leaving it all running can be easily doubling your cloud costs. This is true for each environment you have, including for elements that are only used for a fraction of the time.

As your infrastructure automation toolkit grows and evolves, there are a few things you should start building into all of your applications and deployments. These include:
- Logging
- Scalability
- Security
- Monitoring
As this list grows, there’s one more thing you need: continuous cost control.

By building in cost control automation from the very beginning, you can keep your cloud costs low while maintaining the flexibility required to keep up the pace of innovation. Without this, your costs are destined to rise faster than you intended, which will then cause headaches (and endless meetings) in the future. It may not be coming out of your bank account directly, but saving money at an enterprise organization is everyone’s job. Automating this is the key.

— Jay Chapel
July 10, 2019