devopsy.kinsta.cloud

Tag: DevNetOps

  • Focus on High-Level Policy Requirements to Accelerate Network Processes for DevOps

    Focus on High-Level Policy Requirements to Accelerate Network Processes for DevOps

    Network operations traditionally has been the missing piece in DevOps processes. It’s not difficult to see why. Network management has lagged behind server and workload automation. Updating network policies and configurations typically requires a great deal of analysis, testing and time, undermining the goal of DevOps agility. But once we shift how network devices are administered and how changes are validated, there are emerging ways to incorporate network updates and policies into DevNetOps processes through automation.

    Shift the Focus to Policies, Not Configurations

    DevOps has taken off, thanks in part to the ability of virtual networking and software defined networks (SDN) to automate the placement of workloads, establish connectivity and deliver Layer 4 to 7 services in a few minutes. But virtually all network policy updates, particularly those addressed in the underlay, were constrained by having to reconfigure individual network devices through traditional management platforms or command line interfaces (CLI). New application deployments may require new network paths, access to cloud resources, punching holes in firewalls, etc. There was no way to automate these policy-related tasks through existing network management platforms, and DevOps processes were limited to simple changes that could be made in the overlay network.

    Figure 1: The CLI management model and box-by-box management, in general, do not facilitate DevOps processes, which require higher level policy specifications and requirements. DevOps requires the automation of such policy analysis and verification.

    What’s wrong with updating individual network devices through CLI? Network administration is hampered by the inherent nature of IP networks individual devices never have a holistic end-to-end view of network policies and behavior. They are usually configured to know what to do with traffic locally, when to drop it and when to forward it to which nearest neighbor. This is not how policy requirements are formed (see Figure 1).

    If a new DevOps deployment needs to ensure a rack in Data Center 1 can reach a rack in Data Center 2 through a specific secure gateway, then to a cloud-hosted database and return traffic appropriately, then we need a much broader, path-based view than any individual device or CLI can give us. If we are going to automate any of this analysis or verification, we are going to need a solution that maps our intended policy requirements to network implementations across many devices.

    Intent-Based Networking Aligns With Policies and Intent

    This is where intent-based networking (IBN) can help. IBN can help bridge our application policy requirements with the network design and configuration. It can allow us to verify if a particular policy requirement is supported in the network currently or help with a path-oriented search to quickly identify which devices need to be updated to deliver the particular policy. Changes, such as updating a firewall rule, ultimately may still be made to a single device through a CLI, but we have accelerated the analysis if the policy is currently supported, what changes need to be made and how such a change may adversely affect other existing policies and requirements.

    For example, it may be an enormous benefit to application development teams prior to deploying a new workload, to be able to determine if connectivity from our Rack 1 servers to a remote database is available for new traffic requirements. Typically, this might involve a request to the network team, which would have to analyze the policy requirements or arrange for a test scenario. A trouble ticket request such as this would take more than just a few minutes, even if the network team could jump right on it.

    Application Team Self-Service

    Imagine instead the application team could self-service their request by querying an IBN system that could immediately verify the implementation of the policy. Perhaps the application team could use a simple internal messaging platform such as Slack to specify their policy requirement and get an immediate answer without further analysis by the network team. They wouldn’t even need to learn the GUI or capabilities of the underlying IBN platform with a Slack messaging front-end.

    An IBN solution gives teams the power to avoid thinking in terms of individual device rules and opens the door to end-to-end automated policy analysis for the first time. Application teams could never be expected to understand firewall configurations or be able to self-service their queries about those capabilities, but they can think in terms of high-level application policy requirements and end-to-end path behavior, which IBN platforms can deliver (see Figure 2).

    Figure 2: A sample policy query is constructed in the top bar, as the IBN platform shows a number of available paths between data centers that map to the policy requirements.

    Automating Network Verification

    To further accelerate key network policy changes required for DevOps processes, imagine updates are required to individual devices in the underlay network. Not only can those changes be identified nearly immediately now through IBN, but also the verification that the changes were made correctly and didn’t adversely affect any other applications or policy requirements can be completely automated. These IT tasks of analysis and verification were usually the most manual and time-intensive in the network operations workflow. But now that much of the workflow can be automated, DevOps can benefit greatly.

    The verification process would consist of a long list of existing policy requirements that were already in place and implemented. After a proposed change to network configurations, all of those policies could be analyzed quickly to determine if the new update adversely affected any prior requirements, preventing rollbacks or potential security or compliance breaches.

    The question naturally arises, What is really enabling this shift in network operations from a box-by-box focus to an end-to-end, path-oriented policy focus? IBN platforms have generally replicated much of the analytical intelligence of network engineers to determine overall behavior from analyzing individual device configurations and state information. Much of this intelligence is based on mathematical models of the running network in software and much less so now on analyzing live traffic under limited scenarios. This analysis of the mathematical model in IBN platforms now can provide definitive verification of policy requirements for the first time, and even identify configuration errors long before they can cause loss of service or an outage.

    The key to accelerating network analysis and updates in support of DevOps processes is to shift focus up from individual box-by-box device management and CLI interfaces to focus on end-to-end network behaviors that align with application policy requirements. Application team self-service for many policy-related network requests and automation of network verification tasks are just a few of the benefits for DevOps teams that IBN can deliver today and into the future.

    — Chiara Regale

  • DevOps and Networking: Working to Achieve Nirvana

    DevOps and Networking: Working to Achieve Nirvana

    Enterprises don’t own the cloud, just the experience. If there’s an outage or even a hiccup in the performance or network service, it’s on the network and application teams (not the cloud provider) to fix it. But often with problems in the cloud comes finger-pointing, as network and application teams look to determine liability and responsibility when it comes down to crisis situations.

    There is no question that applications are growing more complex, as is the infrastructure to power those applications. According to the 2019 “Stack Overflow Developers Survey,” comparing year to year the amount of time it takes for software engineers to become effective on teams is increasing. This trend could be, in part, due to the amount of tooling and rise of an operate-what-you-build model (e.g., the dawn of the full lifecycle developer). As our applications become more distributed—and to cope with scaling and availability demands—skill sets that traditionally were infrastructure and development-based have begun to blur.

    Networking Shifting Left

    Networking-centric capabilities such as service meshes (e.g., Istio and Linkerd) and container networking interfaces (e.g., Calico and WeaveNet) are gaining a lot of traction and visibility in the cloud-native application development world. Though SDN infrastructure has been shifting left toward development pipelines, application development teams are learning networking concepts. However, the core networking infrastructure is still maintained by the networking teams.

    This paradigm of software-defined infrastructure is creating a learning curve for multiple teams. With the dawn of NetOps, software development lifecycle (SDLC) principles are being introduced to the networking teams. Traditionally, ticket- or service-driven networking teams can participate in iterations and they are now building the next generation of cloud-agnostic networking capabilities. With NetOps, there is more interaction and innovation. But change is not without resistance; the dynamic nature of modern infrastructure and the short-lived nature of pre-emptible/ephemeral infrastructure do not bode well for static networking rules.

    Networking Team Pressures Today

    With infrastructure headed to the trifecta of clustering, replication and load balancing, networking teams have to deal with many more dynamic addresses than before. Static ports and addresses that used to be loaded into networking tickets are now being replaced with wide ranges and service addresses powering the networking stack of our cloud-native and containerized workloads. Adding to the complexity, the networking team is leveraging public cloud infrastructure. The infrastructure that the networking teams are responsible for is leaving the data center and distributed web infrastructure technologies, such as distributed domain name systems (DNS) and content delivery networks (CDNs), are common for the networking teams to be involved with. Like the application development teams, the amount of new technology hitting the networking teams is increasing.

    One of the great advantages of using public cloud infrastructure is the ability to quickly expand or contract available resources. With this ability, we can spin up and down test environments to try out changes and even the infrastructure level that was not even possible. With this, we have the ability to integrate more. The application development team, which has been iterating for some time, now has common ground with the networking team to try out iterations.

    All in a CIDR

    To illustrate the needed collaboration between networking and application teams, let’s look at an example:

    In one of the biggest incidents of my career, my company was migrating pieces of our platform to a public cloud vendor. Because the organization was early on in this transformation, a lot of minutiae was yet to be flushed out. We were packaging classless inter-domain routing (CIDR) calculations into our first-time VPC configurations—a task the development team was responsible for at the onset of our cloud migration. It was my job to enter this for my service. Instead of a /8 CIDR, I implemented a /16 CIDR. A seemingly small miscalculation on several configs turned into an incident, blocking a majority of traffic hitting the service. The mistake was quickly corrected, but the incident response framework was well on its way to being executed.

    So, who was responsible? I was the one who created the configuration and pushed to production, so this was my fault. But in retrospect, the best solution would have been to involve the networking team immediately in these types of changes. Because the application development teams were vanguarding into the public cloud, however, we cut out the networking team. This was an organizational problem that needed to be addressed. If we were going to deploy into a data center successfully, the networking teams needed to look over my requests and catch my CIDR error.

    Silver Lining, Right?

    During this CIDR incident, I was trying to look at the silver lining: Since we can’t access our application, that means the instances/app must have a lot of extra capacity, right? Untimely, yes, but the analogy did make sense with the silver lining. We were getting billed by a public cloud vendor and because there was a drop in usage, our bill technically should have been lower. But it still impacted the customer. The money saved in our incident was outweighed by external measures (revenue and brand damage). A cloud outage and a subsequent dip in digital experience can have a massive impact on revenue, averaging $38,855 per hour of outage. During high impact or busier times, this number can be much more.

    DevNetOps Nirvana

    The adage of bringing everyone to the table early proves to be valuable. Each team in an organization can have a strategic impact and improve architecture in multiple parts of the stack. Different teams have different sets of skills and can have mutual respect for the vertical skills of each of the teams. In the end, a transparent and positive relationship between network and application teams will help with network remediation and outage prevention. And, the closer the teams can work together and the more they can communicate, the more their investments and efforts can contribute to the business bottom line and quicker remediation long term.

    — Ravi Lachhman

  • Query Your Network Like a Database

    Query Your Network Like a Database

    Network process automation is an often-overlooked component of DevOps. Incorporating DevOps philosophies toward networking, or DevNetOps, can lead to even greater process improvements and more flexible deployments. Fortunately, it’s becoming even easier to automate network tasks with evolving standards and open data models that are allowing more programmatic access and insight to the network infrastructure. The most dramatic shift, however, is now being able to write queries about your network devices and topology as though you were querying a database!

    When rapidly deploying application updates, network flexibility is key. Policy changes or configuration updates are often required. Network operators can best serve DevOps processes when they can programmatically check the status and/or repair key issues from these changes. Here are some example queries that are commonly useful across the entire network infrastructure:

    • Do all distribution layer access links in my network have redundant paths?
    • Are all Border Gateway Protocol (BGP) sessions currently established with configured peers?
    • What are the nearest neighbors of a down device?
    • Are any device interfaces intended to be operational currently down anywhere in my network?

    Note that we are moving away from looking at individual devices, one at a time, to querying about network issues and status across all devices in one simple query. The challenge has been to create a structured data model of network details that was normalized across all vendors and device types that single programmatic query could leverage. The goal is to be able to query about the status of an interface port on a Juniper switch as well as a Check Point firewall, for example. Converting the detailed text files that configure each device into such a normalized data model is becoming easier with established standards, such as OpenConfig. OpenConfig has defined a common syntax and schema for network-oriented applications across vendors and applications.

    Accessing such a network schema is now extremely efficient with the widespread adoption of the GraphQL query language. Network data models are going to be graphical data structures by design, and GraphQL is ideal for building flexible data queries in minutes to such schemas. GraphQL was developed by Facebook and made an open source project in 2015. It has now been adopted by hundreds of development-minded organizations as a more efficient API model, regardless of how the underlying schema is structured.

    GraphQL is already being used widely in DevOps organizations aside from network-oriented applications. GraphQL is extremely intuitive and queries look like a JSON object without the data included. Queries simply return a JSON object with the specifically requested data. From there, it is a small matter of a few lines of code to embed GraphQL queries into a scripting language (such as Python) to analyze, process or format the results. If you aren’t using GraphQL in your DevOps coding, you should be!

    The full process looks like the diagram below (Figure 1), where text files can be parsed and normalized into an open graphical database schema that represents the current network state. The GraphQL queries can be constructed, along with a Python script (Figure 2) to format results, as shown in the third column. The following example returns both the desired admin state of every interface port in the network, along with the current operational state (up or down). The Python script compares the two states and alerts on any discrepancies.

    Figure 1

    In the example query results from Figure 1, consider that the alerts are reported from both an F5 load balancer (atl-bl01) and a Juniper router (sjc-ce01). Writing code that could parse such disparate device types and text files—without a normalized database schema, based on something such as OpenConfig—would be extremely complex. However, a simple GraphQL query can apply across the entire network. The main portion of the Python code that was used to embed the above query, compare the results and display the table, is shown below:

    Figure 2

    Network operations are becoming an increasingly important part of DevOps processes. Automating network tasks is greatly simplified when you can view your network infrastructure as a database. Embedding queries that confirm network policies and health checks into Python scripts can be trivial when leveraging open data models and query languages such as GraphQL. This will enable a new convergence of agile development and network operations for the first time.

    — David Erickson

  • DevSecOps and DevNetOps: New Heroes in the DevOps Saga

    DevSecOps and DevNetOps: New Heroes in the DevOps Saga

    The evolution of DevOps is by no means done, but it’s safe to say that there is enough agreement and acceptance to declare it a hero. DevOps has helped glorify IT to the point where it’s no longer preventing business, nor a provider nor a partner of the business.

    Often IT is the business, or its vanguard for competitive disruption and differentiation.

    Splintering the success of this portmanteau hero, we now hear more and more of two trusty sidekicks: DevSecOps and DevNetOps. Lesser known in their adolescence, these tots are still frequently misunderstood, are still forming their identities, and still need a lot of development if they’re to enter the IT hall of fame like their forerunner.

    Just as the terms look, DevSecOps and DevNetOps are often assumed to be about wrapping DevOps principles around security and networking: operators hope to assuage technical debt and drudgery by automating in proficiency and resiliency. For networking, I’ve covered how there is a lot more to that than coding, but to be sure, these sidekicks certainly espouse operators learning how to do develop while DevOps was equally, if not more, about developers learning to operate.

    The Shift Left: SecDevOps and NetDevOps

    As if it wasn’t hard enough to tell what DevSecOps and DevNetOps want to be when they grow up, we’ve gone and given them alter egos: SecDevOps (aka “rugged” DevOps) and NetDevOps. Think about them exactly as the words look—it’s about the shift to the left. Left of what?

    Traditional DevOps practices focus on business-specific applications development. The development timeline is known as a concept to cash, and with all the superpowers of DevOps we try to reduce our enemy—the lead time and repeatable processes between code and cash.

    Security and building infrastructure, like networks, were considered supporting tasks, not revenue-generating nor competitive advantages. Thus, security and networking were far to the right on the timeline with concerns that deal with operational scale, performance and protection.

    Today’s shift left propels security and infrastructure considerations earlier down the timeline, into coding, architecture and pre-production systems. It’s a palpable penny-drop amid daily news of security breaches and infrastructure outages causing technology-defined establishments to bleed money and brand equity.

    Fill the bucket with cash, but don’t forget to forestall the leaks!

    DevOps and Infrastructure: Challenge and Opportunity

    Automation sparks have flown over the proverbial wall into the camp of I&O pros. Operators trading physical for virtual, macro for micro, converged for composed, and configuration for code is proof that the fire has caught security and networking. Controlling the burn now, is key, so that healthier skills and structures arise in place of the I&O dogma and duff. Fortunately, this is precisely the destiny for our newfound heroes, DevSecOps and DevNetOps.

    However, doing DevSecOps and DevNetOps, embracing security and networks as code, we mustn’t be so credulous as to forget the formidable DevOps practices and patterns that need transforming along the ultimate automation journey. Testability, immutability, upgradability, traceability, auditability, reliability and other you-name-it-abilities are not straightforward to achieve.

    Discounting “aaS” technology consumed as a service, a fundamental challenge to innovating SecOps and NetOps, compared to application ops, is that applications are crafted and built; security and networking solutions are mostly still bought and assembled.

    Security and network infrastructure as code is something that needs to be co-created with the vendors. Other than in the cloud, it will take a while before security and networking systems are driven API-first, and redesigned to offer simulation, finer disaggregation into microservices, and orchestration with scale and resilience.

    While this will land first in software-defined infrastructure, there is still a way to go to manage most software-defined security and networking systems with continuous practices of artifact integration, testing and deployment. Hardware and embedded software will be even more challenging.

    Finding Strength in Challenge

    So on one hand, DevOps is evolving with security and networking shifting left. On the other hand, traditional security and networking ops are transforming with DevOps principles.

    Is the ultimate innovation to squeeze out those traditional operations altogether? Does NetDevOps + DevNetOps = DevOps?

    There is a parallel train of thought and debate, with success on both sides. Purist teams cut out operations with the “you build it, you run it” attitude. Other companies such as Google have dedicated operations specialist teams of SREs. While the SRE reporting structure is isolated, SRE jobs are very integrated with that of development teams. It’s easy to imagine the purist approach, subsuming security and networking into DevOps practices, but only if we assume the presence of cloud infrastructure and services as a platform. Even then, there is still substantiation for the SRE.

    Layers below, however, somebody still needs to build the foundations of the cloud IaaS and data center hardware. As they say, “Even serverless computing is not actually serverless.”

    Underpinning the clouds are data centers. And then there’s transport, IoT, mobile or other secure networks to and between clouds. In these areas, it’s obvious there is a niche for our two trusty sidekicks, DevSecOps and DevNetOps, to shake up ops culture and principles. These two heroes can rescue software-defined and physical infrastructure from the clutches of so many anti-pattern evils, like maintenance windows and change controls (ahem, it’s called a “commit”).

    We may not require rapid experimentation in our infrastructure, but we would warmly welcome automated deployments and updates, failure and attack testing drills, and intent-driven continuous response. They will boost resiliency and optimization for the business and peace of mind for the builders.

    Teams operating security, networks and especially clouds, need to honor and elevate DevSecOps and DevNetOps, so that on the journey now afoot, our teams and our new heroes may realize their potential.

    About the Author / James Kelly

    James Kelly is a lead expert architect on cloud and software-defined networking at Juniper Networks. He also serves as a private yoga instructor, and previously, he founded a capital management firm in Ottawa, Canada. Connect with him on LinkedIn and Twitter.