devopsy.kinsta.cloud

Tag: network automation

From Operator to Agent Manager: The Real Shift in Network Engineering

I’ve been saying for ten years that network automation was three years away from being the only way to do things. Every year I moved the goalposts, because the data never caught up to the rhetoric. After a decade of scripts, YAML, Ansible, Terraform, and every NetDevOps framework we could throw at the problem, roughly 70% of enterprise networks are still managed by hand.

The industry’s explanation for that has always been some version of “those teams just haven’t gotten with the program.” I don’t think that’s right. I think we built the on-ramp wrong.

The On-Ramp Was Never Going to Work

To automate the network the old way, we asked a 25-year network engineer to also become a proficient software developer: fluent in a programming language, comfortable with source control, capable of writing automation that survives contact with production. We asked them to pick up a second career on top of the one they already had, which was keeping the network so stable that nobody noticed it existed.

The best day a network engineer can have is the day nobody calls them. We built the entire discipline around stability and predictability, then asked the same people to ship production-grade software on the side. The math was never going to work, and blaming the engineers for not doing the math was never fair.

Writing Configs by Hand Is a Solved Problem

What changed isn’t the underlying capability. It’s the interface. When an engineer describes what they want in plain language and an agent handles the translation, the expertise doing the real work is still the engineer’s. Fifteen or twenty years of judgment about how a network should behave doesn’t move to the model. It just stops getting filtered through a programming language the engineer never trained in.

That matters because every hand-typed change is a chance to introduce a human problem that has nothing to do with engineering skill. A fat finger. A missed step. A redundancy pair only half configured because the second command got typed at 2 a.m. and nobody caught it until morning. Automation was always supposed to take that risk out of human hands. We just never made the on-ramp easy enough. We do now.

The Job Isn’t Disappearing. It’s Becoming Management

The next decade of network engineering isn’t about engineers being replaced. It’s about engineers running a fleet. One agent handles documentation. One handles compliance checks. One triages tickets. One sits read-only, gathering the data the others need. The engineer’s job shifts from typing the change to defining the agent, setting its guardrails, watching its work, and deciding when it’s earned more scope.

NVIDIA CEO Jensen Huang described this as IT eventually needing something like an HR department for agents, and I think network engineers are better positioned for that role than almost anyone else in IT. We already troubleshoot in layers. We already design for redundancy. We already operate in environments where a single mistake takes down a business. Managing a fleet of agents is going to feel more familiar to a senior network engineer than it will to most other technical roles.

Trust Has to Be Graduated, Not Granted

Nobody should hand an agent production access on day one, and nobody has to. Start with read-only work: documentation, compliance checks, source-of-truth reconciliation, testing. None of it touches production, and all of it delivers value immediately. If you’ve ever put a new hire’s first two weeks toward updating the network diagrams that never matched reality, you already know this playbook.

Let trust build the same way it would for a person. A quarter of clean documentation and compliance runs might earn the agent the ability to propose a change a human approves. Another quarter of that might earn it the ability to make the change and prove it worked. Eventually, routine tickets close without a human in the loop at all. It’s the same onboarding curve you’d use for a junior engineer, applied to software instead of a person.

Leverage, Not Loss

When Excel arrived, accountants didn’t disappear. The manual reconciliation and line-by-line calculation moved to the spreadsheet, and accountants moved up into work that required judgment. I think the same shift is coming for network engineering, and I think it happens faster than most people expect.

If you’re an engineer who has been waiting for the right moment to get into this, the moment is now, not because you’re behind, but because almost nobody is ahead. Spec-driven development is roughly six months old. MCP is barely a year old. Pick one read-only use case your team has been meaning to get to for years and build an agent that does that one thing. Watch it work. Then build the next one. That’s how the role changes: one agent at a time, not all at once.

August 31, 2026
NetDevOps Isn’t Stalled, it’s Stuck on the Wrong Problem

While it is true that network engineers are taking on NetDevOps roles to advance stalled automation efforts, the barrier to NetDevOps isn’t technical; it’s people.

The 44% Problem

The 2025 State of Network Automation Survey from the Network Automation Forum, which gathered responses from 681 network professionals across 58 countries, paints a clearer picture. When asked about barriers to automation, only 10% cited technical challenges. Meanwhile, 44% pointed to people problems: Skills gaps, organizational dysfunction, cultural resistance and yes, sometimes, just personalities that don’t mesh.

Let that sink in: The tools work. Python is mature. Ansible is everywhere. Source-of-truth platforms are production-ready. The tech isn’t the bottleneck — people are — and this aligns with something else the recent NetDevOps article touched on: Nearly half the organizations have no formal measurement of automation success. You can’t fund what you can’t prove, and you can’t prove what you don’t measure.

This is the real challenge in NetDevOps adoption. It’s not about writing better Python. It’s about organizational change management. It’s about communication. It’s about building trust between teams that have historically operated in silos.

You Don’t Need Unicorns — You Need Evolution

Here’s where I want to push back on the industry narrative a bit.

There’s this persistent idea that NetDevOps requires mythical unicorn hires, people who are simultaneously expert network engineers and expert software developers and expert DevOps practitioners. These people exist. I’ve met a few. But building your automation strategy around finding them is a losing game.

Research from theCUBE shows that organizations prioritizing internal upskilling outperform those relying on net-new hires. But I want to amplify this point: You already have the people you need. They just need to evolve.

When I was at Dell EMC, I transitioned from principal network engineer to consultant software developer, because automation became necessary and I kept pulling on that thread. One script at a time. One playbook at a time.

That’s the path for most of us. You don’t wake up one day as a NetDevOps architect. You evolve into one.

The automation survey data backs this up. When asked who builds automation in their organizations, 92% said “network engineers with automation skills.” Not software developers (44%), not DevOps teams (32%) — network engineers who learned new tricks.

So how do you create the conditions for that evolution? Here are a few patterns I’ve seen work:

Permit people to learn on the job. Automation skills don’t come from weekend certification cramming. They come from solving real problems with new tools. That means accepting that the first few automation projects will take longer than doing manual work, and that’s okay.

Pair, don’t silo. When you do have someone with stronger automation skills, don’t make them the automation person. Pair them with network engineers on real projects. Transfer skills through collaboration, not documentation.

Celebrate the small wins. The engineer who writes their first working Python script to pull interface stats has taken a bigger step than it looks. Recognize it. That momentum compounds.

The Data Foundation

Traditional network operations treat documentation as an afterthought, something you update after you make changes (if you remember). NetDevOps flips this. You make changes in the data first, then push those changes to the network through automation.

Intel’s Greg Botts (a Network to Code customer) puts it perfectly: “Start with your data, and everything falls out of that. Network config is a byproduct of the data.”

This is a mindset shift that trips up a lot of organizations.

EMA’s McGillicuddy notes that organizations often spend a year or two just establishing their network source of truth. That sounds like a long time, but it’s foundational work. You’re consolidating information that lives in 10 different silos, cleaning up years of accumulated drift and building the authoritative data layer that makes everything else possible.

Organizations with a clearly defined source of truth are nearly three times more likely to achieve consistent automation outcomes. Yet 56% are still flying without one — and if you’re trying to do NetDevOps without a source of truth, you’re building on sand.

From Documentation to Operations

Here’s where intent-driven methodology changes everything.

Without NetDevOps, without an intent-driven approach, your network data is just documentation. It’s a record of what you did, updated after the fact (if you remember) and its only purpose is to help the next person understand what’s out there. It’s passive. It describes the past.

With an intent-driven approach, data becomes operational. Your source of truth doesn’t just document what the network looks like; it defines what the network should look like. Data becomes the way you operate the network. You change the data, and automation makes reality match.

This flips everything. Documentation stops being a chore you do after the work is done. It becomes a natural byproduct of doing the work itself. You don’t update the spreadsheet after you make a change; you make the change by updating the source of truth. The documentation is the operation.

That’s a profound shift, and that’s why organizations that embrace it see such dramatic improvements. The act of automating forces you to clean up your data, and clean data makes better automation possible. But more than that: Data finally has a real purpose. It’s not busywork. It’s how you run the network.

AI Can Help Today, if You’re Ready

AI can help you today. Natural language queries against network data, configuration analysis, optimization recommendations, and even generating automation code, all of this works right now. The problem isn’t the AI. The problem is that most teams aren’t ready to use it effectively.

Think about how we approach Python in network automation. We don’t just write scripts and push them to production. We have linting. We have a code review. We have testing frameworks and CI/CD pipelines. We build structure around how we use Python so that it’s safe, repeatable and maintainable.

AI needs the same kind of structure. The good news is that the industry is starting to define it — frameworks such as the NIST AI Risk Management Framework and ISO 42001 for AI management systems, plus LLM testing tools such as DeepEval that bring automated validation to AI outputs. However, most organizations haven’t built these guardrails yet.

The automation survey shows that 45% of network professionals have no current plans to use AI/LLM in network operations. Only 3% have actually deployed it. I don’t think that’s because AI doesn’t work; it’s because experienced network engineers are appropriately cautious. They’ve seen what happens when you push untested changes to production. They’re waiting for the guardrails.

The Road to AI Runs Through Automation

Here’s the thing: AI is only as good as the data you feed it. If your network data is scattered across spreadsheets, tribal knowledge and half-updated CMDBs, AI will confidently give you wrong answers.

This is why the road to AI in NetDevOps runs through automation, not around it. Think of it as a flywheel:

First, you build your data foundation. You establish a source of truth and create authoritative records of what your network should look like.

Then, automation starts improving that data. Every automated change gets logged. Every drift gets detected. The act of operating the network through data makes your data better.

Finally, with solid data and mature automation practices, AI has something real to work with. It can query accurate inventory. It can analyze consistent configurations. It can learn from a history of well-documented changes. The foundation you built for automation becomes the foundation AI needs to be useful.

Skip the early steps and you’re asking AI to make sense of chaos. That’s not a technology problem; it’s a sequencing problem.

As Intel’s Botts says, “AI is going to enable us to uplevel our workforce. It’s not going to take these jobs away; it’s going to help us do our jobs better.” But better assumes you’ve done the work to be ready for it.

The Bottom Line

NetDevOps is real. It’s happening — and the organizations that figure it out will operate more efficiently, respond faster and frankly, be better places to work.

But let’s be honest about what figuring it out actually requires. It’s not about finding the perfect hire or buying the perfect tool. It’s about evolving your team, fixing your data and doing the hard organizational work that nobody wants to talk about.

2026 is a turning point. It will be defined by the people who do the work — and most of those people are already on your team. They just need the opportunity to evolve.

May 11, 2026
The Rise of NetDevOps and CI/CD Pipeline Solutions

Over time, as organizations, networks and technologies have changed, the way enterprises think about managing networks has shifted as well. Enterprises are continuing to evolve their networking strategies, specifically with regard to how they expand their efforts to adopt infrastructure-as-code (IaC) practices. Looking ahead, the industry will continue to see the proliferation of solutions that help teams adopt a modern network operating model that delivers an agile, data-driven change management process. In most organizations, the software development team members and the networking team members have operated independently, with separate tools and processes. Developers would leverage pipelines to test and deploy code and then open a ticket for network changes such as VPNs or firewall changes. The network team would implement the changes, and end users could then use the application. The handoff between dev and network could be immediate or, more likely, it could take a few days–delaying the ability of the end users to access the new functionality. NetDevOps is focused on addressing this situation, which has become one of the market’s more critical infrastructure questions: How do you build an organization that can function smoothly across different skill sets, different technologies and different processes? Those same enterprises will be looking to leverage continuous integration/continuous delivery (CI/CD) practices from their software development best practices to help them overcome some of the challenges involved with implementing automation and orchestration into their networks.

CI/CD Accelerates the Automation Process

Let’s start with CI/CD. Adoption of network automation and orchestration will accelerate as network operators change the way they think about network updates and configuration management. According to Cisco’s recent white paper on how “Automation is driving network engineer skills transformation,” the reasons we’re seeing an increase in automation adoption is due to organizations relying on continuous connectivity, a greater need for the network to become more dynamic and the increased rate of changes both in the service supply side (applications) as well as in the service demand side (application consumers).

However, some enterprises trying to build their own automation infrastructure struggle with resourcing, skill sets and maintenance effort struggles. To solve this, many of them are looking to incorporate the traditional DevOps practice of building and updating code with traditional continuous integration/continuous delivery (CI/CD) pipelines into their network automation and orchestration initiatives.

Establishing a CI/CD pipeline is a workflow best practice for DevOps teams to follow. Today, organizations rely on CI/CD pipelines to help minimize bottlenecks in the software development and delivery process, while continuous integration optimizes building, integrating and testing code within a development environment.

By integrating both CI/CD with network automation, teams will gain greater velocity and improve delivery. And by integrating network automation with an application CI/CD pipeline, they can reduce the time to perform network changes and enable secure access between the application and an in-house sales database.

Why NetDevOps?

NetDevOps brings the culture, technical methods, strategies and best practices of DevOps to networking.

When DevOps principles are applied to network operations, organizations minimize operator interactions and optimize programmable network processes to configure, deploy and manage the environment, increasing efficiency and ensuring network availability, quality and reliability.

NetDevOps takes those principles and applies them in a networking environment, offering tools to empower teams to collaborate and build an organization that can function across different skill sets and different processes — enabling everyone to participate in network orchestration.

And it’s a movement that’s taking hold. According to Cisco’s Hybrid Cloud Trends report, 55% of those surveyed have created a cross-functional team with technical and business representation and 50% have a centralized CloudOps and NetOps function to help ensure their organization’s hybrid cloud strategy meets business objectives.

Network Management Tools Will Continue to Evolve

In today’s networking environment, where the expectation is to have an “always on” approach, teams are expected to build and release new applications at a faster pace than ever before. And every team has its own processes and methods for doing that which can create delays in the time it takes to execute the application. Incorporating network automation into the pipeline can empower teams to make application changes and deploy them to the network in seconds instead of the multiple days/weeks required for manual network changes. By integrating CI/CD pipelines and NetDevOps principles with network automation and orchestration, organizations can reduce the network change bottleneck even more.

As we face the new year, we expect the way in which teams consume network automation and orchestration services as part of their software pipelines to become more sophisticated, with NetDevOps and CI/CD practices becoming a standard. And as more services become available, the adoption of IAC becomes easier. By abstracting and tweaking to abstract the traditional CI/CD pipeline, organizations of all sizes can accelerate the use of network automation and orchestration.

April 18, 2023
The Power of Automation and Distributed Sources of Truth

Enterprise networks are vast, intricate systems with multiple methods for storing configuration and state information. The configurations of those networks are also constantly changing as new applications, services or other updates are added. When you consider the complexity and dynamic nature of an enterprise network, it may be alarming to learn how much data–including most network configurations–is still updated manually, often in the middle of the night. To make that process even less efficient, the people doing the updating are required to manually log those updates on spreadsheets or into a custom-built database.

Relying on manual processes to maintain and manage the state of an enterprise network not only leaves the door open for errors but can also lead to the network spreadsheets or databases being out of sync with the actual current state of the network. Is the network correct? Or is the spreadsheet correct? Should we make the network look like the spreadsheet? Or should we update the spreadsheet to reflect the network? What is the actual source of truth?

The good news is that the reality of modern networking has changed the paradigm. As programmable networks are deployed more broadly, manual CLI interfaces are no longer necessary. Reading and writing network configuration and reading network state can be accomplished quickly and easily with machines and automation.

The Source of Truth

More enterprises are embracing network automation because they recognize the efficiencies it can deliver when managing complex networks and the benefits those efficiencies can add to the bottom line. But if you’re going to automate your infrastructure, your automation solution will need to gather authoritative information about it from somewhere. This source of truth holds the knowledge needed to enable automation to make changes to the network.

Many enterprise networking teams feel they need to rely on a single source of truth to know the true state of the network. They believe that building a single system/server/database that can synchronize with the configuration and state information of all devices across the whole network is a viable solution. But such a solution doesn’t currently exist. Given the multifarious nature of today’s networks, relying on a single source is both impractical and potentially expensive.

Rather than asking “What is the single source of truth?” perhaps the better question is “What will provide better data: Creating a single source of truth that tries to synchronize to the entire network? Or accessing distributed sources of truth needed to accomplish the automation in real-time?”

Instead of relying on a single source of truth, enterprises should focus on accessing multiple sources of truth to conduct proper and meaningful automation.

Cast a Wider Net

If you’re running a service provider network, for instance, your business is your network. If you’ve spent millions, or even a billion dollars on your network assets, it’s worth your time to have a system in place that accurately manages and maintains those assets. A single source of truth won’t provide that.

Also, as the network scales in size and complexity, the time it takes to synchronize a source of truth to the network also grows, limiting how often you can synchronize and leading to a greater time delta between synchronization states.

Finally, the process of trying to create a single source also doesn’t address the real issue, which is that enterprises are still relying on manual processes that will almost always result in the source of truth databases and the actual network status being out of sync at some point. And that can have serious negative effects on a business as well.

The case for accessing multiple sources of truth is a relatively simple one: it allows for greater flexibility and enables enterprises to more easily leverage data to benefit their business.

With the advent of programmable networks–machines talking to machines–the most current automation solutions can access multiple distributed sources of truth–different APIs and databases that are responsible for being the source of truth for different parts of the network – and federate and sync data from these systems in real-time, providing a truer window into the status of the network and more accurate, actionable data that delivers business value back to the enterprise.

Automation Hesitation

So why are some enterprises still hesitant to commit to a full network automation project? There are various reasons for this, but clearly, there’s concern that not having a single source of truth for network data – or data that may not be 100% accurate – will compromise the automation process. It’s a classic “fear, uncertainty and doubt” thought process: “What if the data is only 70% accurate? If we try to automate with bad data, we’re going to break the network 30% of the time.”

Some network teams may believe they can’t automate anything until they can combine their five or six sources of truth into one huge database. At the same time, leadership may be pushing those network teams to move forward with automation to get more value from the network. By choosing to wait until they finally have a “pristine” database, they could lose months or even years of time, to the benefit of no one.

The solution is to take an incremental approach. Look at all the various network assets–programmable, cloud, legacy, etc.–and start the automation process in the areas where good data exists and where a source of truth doesn’t need to be built. In parallel, you can be cleaning up the other data and add automation when that data is ready.

Being able to utilize multiple sources of truth is better than worrying about having a “perfect” source of truth and holding back on an automation initiative. In the long run, it will be beneficial to both the network team and the business.

Just as important as committing to an automation solution that can integrate with multiple sources of truth to make decisions, is deploying the right automation solution. The most effective automation solution will embrace:

● API-first–The only way to integrate multiple sources of truth into a unified one is through an API-first approach, allowing systems to talk to each other.
● Data federation and transformation–Having unified sources of truth is only impactful when teams can fully integrate the data together while ensuring it speaks the same language.
● Unified view–An abstracted and federated view of the data, processes and logic from integrated systems enables a single pane of glass across an entire network, as well as management tools for simplifying automation across the organization.

Multiple Sources Equal More Actionable Data

With today’s modern, programmable networks–and the absolute necessity for enterprises to have access to accurate, real-time network data–relying on a single source of truth is based on a flawed assumption that we can always have a synchronized database, and therefore is not a viable strategy.

By adopting network automation, organizations can adopt a distributed source of truth solution by enabling the multiple systems of record, and their collective data, to act as the source of truth, mitigating data quality issues and manual errors.

As an enterprise implements its automation strategy, having multiple sources of truth within the network will result in more accurate, timely and actionable data for automation to succeed, which will increase efficiency and drive the business forward.

August 29, 2022
A Full Pipeline: CI/CD for Network Automation and Orchestration
The intersection of CI/CD and network automation has grown substantially over the past few years. Network teams have seemingly reached the point in their automation and orchestration journey where it has become critical to implement a strategy to manage the testing, version control and deployment of both network changes and automation assets.

While CI/CD has grown in popularity, there are—at least on the surface—many differences in the types of pipeline implementations to which network teams want to apply the technology: Different tooling, different scenarios and different problems being solved. Despite the seeming disparities, there are some key themes that are consistent throughout most pipeline implementations. For instance:

Network teams either want to develop pipelines to better manage specific changes to the network or they want to use pipelines to better manage the testing and deployment of automation and orchestration assets like workflows or scripts. Where initially those teams may have turned to network infrastructure-as-code (IaC), a significant (and growing) number are investing in automation asset pipelines.

The trust and resources being dedicated to these pipelines is just one of a growing number of indicators that automation and orchestration solutions are maturing and moving beyond initial implementations to large-scale, business-as-usual technologies.

How Pipelines and Network Automation Complement Each Other

A close examination of recent pipeline developments provided more clarity about the respective strengths of CI/CD tooling and network automation tools for network change management and life cycle—GitLab, GitHub, Jenkins and CircleCI to name just a few. Most network teams would prefer to stick with a “right tool for the right job” philosophy, wherein the best tool for any given task can easily be integrated into an overall solution.

In this context, the “right tool” philosophy looks at pipelines and at network automation/orchestration tools to identify what they do best. Then, a strategy is developed that maximizes the benefit of both technologies. Using this approach, teams can avoid the pitfalls of trying to solve everything with one or the other, leading to a suboptimal process.

Some good current rules of thumb are based on two simple points:
- Pipelines focus on creating consistency and repeatability by managing activities related to code versioning, testing coordination, code merging, code validation and approval. The primary benefit of pipelines is the ability to automate the change process.
- Network automation focuses on generating the network payload and applying that payload to the network elements. This may also include automation of engineering and business logic steps that typically must be performed in a specific order. The best uses of network automation within the change process require attention to business logic, order of operations and network engineering rules.
By integrating both technologies into a common pipeline, network teams can get the best of both toolsets.

The example below illustrates an integrated model where network automation tools provide the engineering logic for the generation, testing and deployment of configuration changes, while the CI/CD tools (represented by the orange arrows) carry the activities through a repeated, automated process, ensuring that the changes go through a consistent path with proper version management, approvals and testing. Multiple teams have successfully implemented this model and they are beginning to iterate their solution to incorporate their unique experiences and improvements.

The widespread expectation from the industry is that, in addition to the integration of CI/CD and automation into a common pipeline, we will begin to see other indicators of maturation of orchestration and automation. Some of these are areas that are already taking shape, such as the shift in skillsets and competencies outside of traditional CLI-centric engineering toward DevOps and CI/CD skills, or the growing importance of quantifying the business impact of automation and orchestration.
July 14, 2022
Bringing NetOps Up to Speed With DevOps
Automation can be an unwieldy topic for network operations and DevOps teams who have enough challenges working together cohesively. Networks have been traditionally viewed as the long pole in the tent for emerging DevOps practices.

But for DevOps practitioners and business leaders alike, there’s good news: NetOps are more prepared than ever to embrace automation and programmability—increasing the likelihood of on-time service delivery, with far less friction between IT teams.

According to Enterprise Management Associates (EMA), a strong majority (77%) of organizations deem network automation a high priority, and 58% have already implemented or are planning to implement, approved solutions. This is an important and promising shift.

Traditionally, network leaders have favored an operating model that emphasizes stability and resiliency over agility. Understandably so, as any network interruption can have a cascading impact on the business’ operations and profitability.

While ensuring five 9s of availability remains a top priority for NetOps, the growing need to support DevOps, containerized infrastructure for applications and cutting-edge digital services—among other initiatives—continues to highlight some of the glaring inefficiencies of traditional network management tools that are too error-prone and bound by scalability issues.

With the right automation framework, NetOps can play an equal part in enabling DevOps practices by becoming more adaptable to evolving business and application requirements in the cloud and locally.

Centralizing Management With Cloud-Based Service Delivery

For decades, improving agility has been a dyed-in-the-wool practice for network teams. Organizations worldwide have now realized enormous benefits by implementing new software-defined management tactics that allow the network to become more centralized, programmatic and dynamic in nature. By disaggregating software from hardware, networking teams now can manage many more services and devices with less effort and resources.

Among NetOps’ new arsenal of tools and best practices, centralized management has become one of the key ingredients driving toward greater efficiencies. Utilizing a cloud-based approach for provisioning, deploying and managing network services closely mirrors the way IT has spun up cloud storage and compute over the last decade. Now, DevOps teams no longer have to wait days or weeks for the NetOps team to provision the right networking resources so that they can begin designing, testing and deploying new applications.

Along with the added efficiencies, NetOps teams have greater and more uniform control over device configurations and policies, where security vulnerabilities lie in wait if these do not establish a zero-trust environment. Orchestrating these elements centrally, rather than device by device, places far less burden on individual network operators for protecting high-value corporate assets. Rock-solid network access control also reduces the risk of security or performance degradation that commonly arise from using an ad hoc patchwork of scripts, templates and CLI commands.

Advancing Network Automation Frameworks

Because creating a cloud-native network infrastructure does not all happen with the press of a button, network operations won’t become fully automated overnight. Other factors come into play that inhibit broader adoption of automation frameworks, including budget constraints, difficult technology implementations that lack integration with legacy systems and skill gaps for existing staff.

It’s essential that NetOps teams embrace a network automation framework that aligns to current and future operating models. Through proactive collaboration, NetOps folks can extend the value of automation platforms already used by their DevOps or application counterparts.

Fortunately for NetOps teams, myriad networking vendors today readily offer pre-built, certified solutions for DevOps platforms, making it easier to get started on a cloud-native journey by automating activities such as device onboarding and configuration changes. This way, network administrators can leverage existing vendor partnerships, in-house knowledge and technology that is already proven within the larger IT environment.

Additionally, network engineers shouldn’t need—and won’t have the extra time—to become top-notch developers to take advantage of programmability during their cloud-native journey. Developing basic programming skills is advantageous, but network management systems that offer Python scripting, a consistent set of APIs and webhooks can perform the “heavy lifting” when it comes to enabling extensibility with third-party IT platforms.

Today, this level of extensibility includes being able to integrate with third-party IT service management tools. A common use case that can realize significant time savings and greater network and application availability is to auto-trigger and assign an incident ticket when a performance SLA is breached. It’s very easy to collect and use machine learning-based tools to correlate all of the relevant diagnostics data in order for a network operator to immediately begin a root cause analysis and quickly resolve the issue.

On the Horizon: AI-Driven Operations

Another core aspect of DevOps is continuous improvement—both in terms of elevating service levels, as well as enhancing user experiences. But faster rates of change resulting from continuous improvement processes can negatively impact application performance, and the network is often the first to be blamed.

But there is a huge silver lining within this cloud created by faster rates of change, which also highlights another growth area for NetOps teams to learn and master. That is, the use of AIOps.

Today’s networks are bound by human scale. When things break, it takes a human to realize it broke, research the issue and then apply the fix. While traditional management and monitoring tools can alleviate some of the burden, alert fatigue often becomes a huge problem—in other words, too many false alarms, false positives or, worse yet, drawing attention away from more critical issues.

AIOps tools and technologies mitigate these issues and limitations by combining artificial intelligence, machine learning, and advanced network analytics to automate many aspects of traditional monitoring and troubleshooting. Once deployed, IT organizations have the ability to predict and preempt failures,\ or quickly recover from issues when they do occur. This includes being able to:
- Leverage ML-trained data sets to continuously monitor network performance and detect anomalies.
- Eliminate guesswork by prioritizing fixes or improvement efforts and validating that those efforts have the desired business impact.
- Continuously optimize network performance based on benchmarks.
Put another way, AIOps systems are worth their weight in gold by helping networks move toward a self-healing, self-securing and self-optimizing model. Even in the interim, during the early phases of deployment, such AI-assisted operations can help IT identify and resolve network- or user-impacting issues faster and easier. The results are that dramatic.

Next Steps: Bring a DevOps Mentality to NetOps

Contrary to popular belief, the goals of NetOps are not diametrically opposed to DevOps. Both groups want to deliver more value and help the business innovate.

Equipping NetOps teams to build on existing DevOps-based automation frameworks, paired with AI-driven insights and modern management tools, will go a long way in ensuring frictionless delivery of new services.

Now is the time for network operations teams to get to know your DevOps team, learn how they operate and come together to create new methods for advancing the organization’s IT mission. The investment in time and effort should be a creative win-win for both groups and produce a big payoff in IT’s strategic position and importance to the organization.
July 10, 2020
Bridging the Network Automation Skills Gap

With the introduction of network programmability and APIs for everything from management systems to devices themselves, the modern network landscape has evolved to being managed like software. This shift from hardware-focused networks to software-centric functions has had a profound effect on network management techniques and skillsets required to keep pace with the changing ecosystem.

A recent EMA survey notes that skills gaps associated with network automation are an issue for 96% of enterprises. Even respondents with significant network automation initiatives in place felt their solutions stretched the abilities of their network teams. Network automation can expose skills gaps where network engineers might lack experience with new tools and software development. At the same time, application development teams, who likely would be included in building an in-house network or automation software solution, will also lack networking skills. Hybrid skills with both development and networking expertise are rare and available only at a premium.

This journey to network automation can be difficult, especially given the required technical skills that are not in abundance within many organizations. Simplifying network automation and empowering a broader set of participation through capabilities to easily create and manage automation workflows presents a key opportunity for organizations to not only mature efforts around network automation, but to also execute against broader business digital transformation objectives.

Four Keys to Democratizing Network Automation

The key to democratizing network automation and bridging the skills gap is to provide NetOps teams with the ability to easily design and build network automation without having to re-tool, build custom code or learn specialized software skills. There are four key elements required to effectively democratize network automation in the enterprise.

Easy Onramp for Non-Developer

Automation is typically accomplished through writing code in a particular programming language. For a person without any programming skills (or even minimal skills), they must first learn how to code before they can even attempt their first automation task. This is very difficult in terms of time and interest. Additionally, many network engineers see the amount of work it would take to learn to write code, and become disinterested. There has to be a new set of tools and methodology to empower these people to automate their work without the long road of learning to become programmers. It’s important to invite as many people to the automation table as possible including IT, security, cloud and NetOps teams, to streamline business processes.

Capture Network Tribal Knowledge

Within the organization, there is already existing embedded tribal knowledge that is incredibly valuable. By harnessing and transferring that knowledge into an automation workflow without a lot of programming, it provides an accelerated path to network automation. The goal is to leverage traditional network automation and management practices (Scripts, CLI) for re-use across the organization.

Network engineers with years and years of experience have a valuable set of processes and procedures they’ve built up inside their heads, and this wealth of information is a valuable asset. Unfortunately, this information is rarely documented and is normally passed on by activity and word of mouth. This is the type of tribal knowledge that needs to be captured, preferably through automation workflows that can be used in a task, documented in the workflow and modified over time to improve it or change it as the infrastructure changes.

Integrate DevOps Principles

Augment network automation with CI/CD pipelines and API services. These are well known solid principles that are already used in other domains outside of networking and pay big dividends when applied to the automation of complex networks. When we look along the horizon of the network automation journey, these same CI/CD practices will be adopted by future networking teams and allow them the same advantages of reliability and velocity within the network.

Enable Cloud Architect Participation

It is important to have networking people that can understand the networking constructs of the cloud, as well as cloud teams that understand how it integrates and operates within the network. Enabling cloud architect participation in network automation will allow for the real-time feedback and control of cloud-based applications and infrastructure critical to modern networks.

Cloud networking is increasing in complexity and will soon require the skills and experience of network engineers to help solve the problems of cloud networking. Traditionally, cloud and networking teams have been separated because their technology domains have been siloed and there was very little interaction required between the teams. As technologies like SD-WAN have been adopted, the line of responsibility between cloud and network teams has blurred. Cloud and networking teams will need an automation platform that can be used by both to create workflows and automate between both domains. Business technology decisions are forcing these two groups together, and they must have a way to use their own tools, but also have a platform to collaborate automation between them.

As applications and services are becoming more complex, distributed and require connectivity and policy enforcement across a diversity of domains (such as cloud, SD-WAN, data center, wireless and network applications), the management of these network concepts requires us to re-think how we have traditionally managed networks. As network automation plays an increasingly larger role in enterprise networks, a byproduct of the complexity is a skills gap that threatens to slow or stymie enterprise initiatives.

By making the onramp to network automation easier for non-developers through capturing tribal knowledge, integrating DevOps principles and enabling cloud architect participation, enterprises better ensure a successful migration to network automation across multiple domains.

— Rich Martin

April 2, 2020
How DevOps Training and Certification Benefits the Team
Cisco Systems recently announced sweeping changes to its training and certification programs to increase the real-world value of both. Cisco’s move demonstrates a trend by organizations that provide training and certification to place more value on the practical aspects of educating technology professionals—which, in turn, should have a major impact on those looking to build DevOps teams.

“Advances in network technologies have enabled new applications and business models—converged IP networks enabled voice over IP, 3G networks enabled mobile applications and 4G networks enabled mobile video,” said Susie Wee, senior vice president and CTO of DevNet at Cisco Systems. “DevOps has become a key factor for bringing those new networked applications to life. Training and certifications for network professionals and software developers have become a must. The goal here is to bring enhanced software skills to networking, enabling application developers to write applications that leverage the full capabilities of the network.”

Cisco’s realization that real-world knowledge brings demonstrable value to businesses has culminated in new DevNet Certifications, which are designed to demonstrate an individual’s ability to bring software skills to evolving network technologies. “The only way to fully leverage the capabilities of the network is to build DevOps teams with both networking and software skills that can address the intricacies of both software-defined networking and physical networking solutions,” Wee added.

Cisco is not the only company pursuing a change in training and certification processes. Industry group CompTIA recently revamped its certification programs. For example, the latest version of the CompTIA Linux+ exam has been redesigned to validate the competencies and skills required of early career IT professionals supporting Linux systems.

“For certifications to have real value today, people must immerse themselves in the technology,” noted James Stanger, CompTIA chief technology evangelist. “The new Linux + certification uses performance-based testing that presents candidates with real-world scenarios, meaning that achieving certification demonstrates the ability to function in the real world.”

CompTIA may very well be one of the leaders in the movement to bring real-world relevance to the certification process. The company has been using performance-based testing and has been embracing certification pathways to build teams. “For DevOps teams, critical skills, including project management, both agile and waterfall, are important. Developers will need to know the operating systems they are developing in, so they’ll need to understand Windows and Linux,” Stanger said. “The idea is that for DevOps, you need to get deep into the guts of operating systems and understand them well. You can specialize in one area, but you’ll also need to understand databases, relational databases and object-oriented databases.”

Cisco also understands the importance of educating DevOps professionals. “Our new Cisco Certifications, with new DevNet Certifications, are designed for training and certifying network professionals and software developers, so they can bring software skills to networking,” Wee noted. “This new education process teaches application developers how to write applications that leverage the full capabilities of the network, and it helps build IT teams with network skills and expertise and software skills and expertise so they can leverage the full capabilities of the network, helping to support the DevOps process.”

Cisco is also investing in the idea of building a community in the form of the company’s DevNet offering. DevNet was created some five years ago to help network engineers and application developers take full advantage of the revolutionary changes shaping the industry. The company is now focusing its efforts on what it sees as its customers’ biggest problem—network automation—and has created the DevNet Automation Exchange, a community-based developer center where networkers and developers can learn to “walk, run and fly” with network automation using Cisco’s products and the full capabilities of the new network.

According to Wee, those three processes can be broken down into:
- Walk: Get visibility and insights into their networks in a read-only way. This enables use cases such as gathering telemetry and insights from running networks and performing audits to ensure the ongoing security of the network.
- Run: Activate policy and intent across different network domains. This enables use cases such as providing self-service network operations that comply with security policies and operational guidelines.
- Fly: Proactively manage applications, users and devices with DevOps workflows. This enables use cases such as deploying applications in continuous integration and continuous delivery (CI/CD) pipelines while configuring the network in accordance with new application policies.
Wee noted Automation Exchange has been seeded with more than 50 code repositories, many contributed by Cisco’s engineering teams and SEs. Now, she said, the company is inviting DevNet community members to submit code that solves their use cases.

Updated certification and training resources from the likes of Cisco and CompTIA will go a long way to helping build DevOps teams that can hit the ground running.

— Frank Ohlhorst
July 8, 2019
Autonomous Networks: The Future We Need Now

Automation has become a necessity for managing systems in today’s age of dynamic and agile cloud-based environments. Operators are looking to optimize with efficiency and minimalism—to do more with less—as the goal. This same line of thinking should be applied to managing the network or system infrastructure.

While network operators face the same challenges as what is seen in compute environments, they appear to have the advantage of fewer pieces to manage. However, as the tendency of building more densely utilized CLOS fabrics, each network node has 10 to 20 times the number of servers under it. Even though there is a lower ratio of network devices to servers to manage, this multiplier means each network node has a higher potential impact—which means we can’t accept the notion that fewer devices result in a lower interest in automation, due to the high risk of outages. Automating these systems should be prioritized, spearheading the movement toward being completely autonomous, lowering the risks of day-to-day operations.

What’s the Holdup?

In the past decade, the transition to mobile has required the shift of applications to be flexible enough to move resources between sites and locations on demand. Led by hyperscale operators, the strategy causes application monoliths to separate into microservices, driving virtualization and containerization to allow for maximized portability. East-west bandwidth requirements expand as applications become more dispensed. This makes the network increase throughput and reliable—inciting the easy choice to adopt CLOS networks. These operators use off-the-shelf components, utilizing reliable (and cheaper) merchant silicon at the core. However, this pressures legacy network management methods by effectively increasing the number of devices from two to four distribution/core devices to 48-plus CLOS nodes. Hyperscalers addressed this issue with homegrown network operating systems (NOS), constructed with network automation to make it easier to manage at scale.

But when small to midsized organizations try to imitate the hyperscale operators, they become restrained by legacy NOS architectures. We’ve been hindered by an establishment of manually driven networks, writing scripts on top of the command line interface (CLI). In the best-case scenario, we have tools such as Ansible or Puppet, which can bring automation and some orchestration to the network. In the worst case, network managers resort to just hiring more engineers. As a result, there remains a substantial gap between network and compute infrastructure.

To display this gap between the industry’s current state and where we should be, let’s look at this graphic which shows the levels of service automation maturity:

Most modern networks operate between Level 0 and Level 2, meaning operators are barely scraping by with a combination of zero-touch provisioning and screen-scraping scripts (and the increase of engineers). They are reacting to changes that occur and aren’t actually able to verify that automation worked, outside of logging in and manually reviewing the result—which absolutely does not scale, due to frequent errors and eventual, and immense, inconsistencies between design and actual implementation.

Should Operators Look for the Solution?

Cloud-native solutions provide significantly better experience, both in maturity and agility. They have sufficiently progressed at using software to manage other software, operating at Level 3 or higher on the above scale. We can leverage this for the network, but it has to start discarding obsolete architecture and modernizing the starting point—the network OS.

Microservices, containerization and orchestration (with Kubernetes) are the foundation of cloud-native architectures and are directly addressing the challenge of automation—such as how network and application services automate with the same level of maturity by looking at service fabric mesh in the public cloud. By utilizing Kubernetes and employing modern containerized microservices architectures into the NOS, we unite NetOps and DevOps. The network is essentially just a complex distributed application and can be automated like other similar applications. We need to widely adopt autonomous networking and accept that a modern NOS architecture is needed to implement a method of building networks that match the same level of automation maturity we see with cloud-native applications.

— Adam Casella

May 7, 2019
Query Your Network Like a Database
Network process automation is an often-overlooked component of DevOps. Incorporating DevOps philosophies toward networking, or DevNetOps, can lead to even greater process improvements and more flexible deployments. Fortunately, it’s becoming even easier to automate network tasks with evolving standards and open data models that are allowing more programmatic access and insight to the network infrastructure. The most dramatic shift, however, is now being able to write queries about your network devices and topology as though you were querying a database!

When rapidly deploying application updates, network flexibility is key. Policy changes or configuration updates are often required. Network operators can best serve DevOps processes when they can programmatically check the status and/or repair key issues from these changes. Here are some example queries that are commonly useful across the entire network infrastructure:
- Do all distribution layer access links in my network have redundant paths?
- Are all Border Gateway Protocol (BGP) sessions currently established with configured peers?
- What are the nearest neighbors of a down device?
- Are any device interfaces intended to be operational currently down anywhere in my network?
Note that we are moving away from looking at individual devices, one at a time, to querying about network issues and status across all devices in one simple query. The challenge has been to create a structured data model of network details that was normalized across all vendors and device types that single programmatic query could leverage. The goal is to be able to query about the status of an interface port on a Juniper switch as well as a Check Point firewall, for example. Converting the detailed text files that configure each device into such a normalized data model is becoming easier with established standards, such as OpenConfig. OpenConfig has defined a common syntax and schema for network-oriented applications across vendors and applications.

Accessing such a network schema is now extremely efficient with the widespread adoption of the GraphQL query language. Network data models are going to be graphical data structures by design, and GraphQL is ideal for building flexible data queries in minutes to such schemas. GraphQL was developed by Facebook and made an open source project in 2015. It has now been adopted by hundreds of development-minded organizations as a more efficient API model, regardless of how the underlying schema is structured.

GraphQL is already being used widely in DevOps organizations aside from network-oriented applications. GraphQL is extremely intuitive and queries look like a JSON object without the data included. Queries simply return a JSON object with the specifically requested data. From there, it is a small matter of a few lines of code to embed GraphQL queries into a scripting language (such as Python) to analyze, process or format the results. If you aren’t using GraphQL in your DevOps coding, you should be!

The full process looks like the diagram below (Figure 1), where text files can be parsed and normalized into an open graphical database schema that represents the current network state. The GraphQL queries can be constructed, along with a Python script (Figure 2) to format results, as shown in the third column. The following example returns both the desired admin state of every interface port in the network, along with the current operational state (up or down). The Python script compares the two states and alerts on any discrepancies.

Figure 1

In the example query results from Figure 1, consider that the alerts are reported from both an F5 load balancer (atl-bl01) and a Juniper router (sjc-ce01). Writing code that could parse such disparate device types and text files—without a normalized database schema, based on something such as OpenConfig—would be extremely complex. However, a simple GraphQL query can apply across the entire network. The main portion of the Python code that was used to embed the above query, compare the results and display the table, is shown below:

Figure 2

Network operations are becoming an increasingly important part of DevOps processes. Automating network tasks is greatly simplified when you can view your network infrastructure as a database. Embedding queries that confirm network policies and health checks into Python scripts can be trivial when leveraging open data models and query languages such as GraphQL. This will enable a new convergence of agile development and network operations for the first time.

— David Erickson
March 14, 2019