devopsy.kinsta.cloud

Tag: networking

Kong Employs Machine Learning to Extend Services Control

At the Kong Summit 2019 conference, Kong today announced it has acquired Insomnia, an open source testing tool for application programming interfaces (APIs). In addition, it has added support for REST, Kafka Streams, gRPC, GraphQL and plugins written in Go to its service control platform.

Kong also added more capabilities based on machine learning algorithms in the form of a visual service map a Kong Immunity module that detects anomalies across services spanning multiple platforms.

Additionally, the company is delivering Kong Studio, a suite of tools to design, build and maintain APIs for REST and GraphQL endpoints based on the API testing and debugging tool Insomnia, which then can be published to Kong Gateway, proxy software for deploying and managing distributed APIs. The 2.0 release of Kong Gateway separates the data and control plane for the platform and provides the ability to write plugins in Go via Kong’s Plugin Development Kit.

Kong CTO Marco Palladino said collectively, this approach makes it easier to address a raft of potential API issues at design time rather than after an API has been pushed into a production environment.

Finally, Kong also updated Kuma, an open source control plane for a service mesh based on Envoy proxy software developed under the auspices of the Cloud Native Computing Foundation (CNCF).

Kong has been making the case for a service control plane dubbed Kong Brain, which spans both cloud-native and legacy monolithic environments and makes extensive use of machine learning algorithms. Palladino said as more services proliferate across existing monolithic and cloud-native application environments, there is no way IT organizations will be able to keep pace without relying more on machine learning algorithms to manage events and processes occurring within milliseconds of one another across the much more complex IT environment.

In fact, Palladino noted that rather than expecting one environment to supplant the other, IT organizations should expect to see services proliferate across both monolithic and cloud-native application environments. Managing all those different classes of services will require a service control platform capable of spanning multiple environments, he said.

Over time, machine learning algorithms will enable service control platforms to become smarter as IT environments transition to managing services end to end versus continuing to focus on individual applications. Naturally, there’s a high correlation between organizations that have made that transition and those that have embraced DevOps to build, deploy and manage those services, said Palladino, noting the most successful organizations employ DevOps to deploy and update APIs as if they were a standalone application rather than a set of add-on capabilities that tend to be forgotten over time.

It remains to be seen how the confluence of APIs and DevOps ultimately will change how IT is managed. The only thing that is for certain is the days when applications were managed in isolation are finally coming to a much-needed close.

— Mike Vizard

October 2, 2019
Networking: The DevOps Community Deserves Better

Developers of cloud-native applications are at the heart of a process of radical transformation shaking up the business world. Using DevOps and SRE methodologies, modern development teams are bringing a level of agility to the game that is all about freeing enterprises from the shackles of the old, pre-cloud days and letting them take advantage of new opportunities at light speed.

But the developer community has a problem. The process of developing and delivering cloud-native applications in a DevOps paradigm demands the right connectivity to back it up, and traditional networks aren’t doing the job. Cloud-native developers need cloud-native networking—network solutions that work in the same frictionless and agile way they do—solutions that put the application first and replace traditional manual network configuration with connectivity-as-code.

To play their part, developers need application-friendly connectivity that is designed for applications composed of microservices and APIs which span many edges, clouds and service meshes. A new software only, cloud-native approach to connectivity is required to address this and other challenges such as security, performance and operational support at scale.

There are four things developers shouldn’t take for granted in a network solution: agility, security, simplicity and performance.

Agility

What application developers need is a cloud-native, agile, on-demand alternative to traditional networking—something that delivers the speed to market that they need, and safeguards their profitability. With a model such as application-specific networking in network-as-a-service models, application development teams can create on-demand connectivity across edge, cloud and service mesh environments to support their apps wherever their apps travel.

Security

App developers require zero-trust software defined perimeter security that meets and exceeds even the most stringent compliance standards, and can be controlled as code by their DevOps tools and applications.

Simplicity

New networking approaches that puts the application in contol can abstract the developer and DevOps teams from the complexity of differences between underlying networks, clouds or service meshes.

Cost-Effective Performance

Application-controlled network-as-a-service models can deliver operational cost reduction over traditional MPLS and VPN solutions, while boosting throughput and performance compared with legacy alternatives.

Conclusion

Developers deserve better networking. It’s time for them to say goodbye to the old way of doing things and move on from restrictive legacy solutions with their lack of scalability, absence of agility, poor security and unpredictable overheads.

— Galeal Zino

July 17, 2019
VMware to Acquire Avi Networks for NetOps Capability

VMware announced it intends to acquire Avi Networks for an undisclosed price as part of an ongoing effort to close the gap between network operations (NetOps) and DevOps.

Once this deal closes, sometime between now and August, VMware plans to add a software-based load balancer, along with a web application firewall (WAF) and a service mesh to its network virtualization software portfolio.

Tom Gillis, senior vice president and general manager for the networking and security business unit at VMware, said the goal is ultimately to make it as simple to deploy applications in an on-premises IT environment as it is on a public cloud by enabling IT organizations to automate the provisioning of network resources exposed via the VMware NSX network virtualization overlay. Avi Networks will extend those capabilities by adding a high-performance programmable load balancer, which eliminates reliance on legacy load balancers based on proprietary hardware platforms that are both expensive and rigid.

Gillis said VMware will then infuse the WAF technology developed by Avi Networks into its network overlay platform to further secure east-west traffic in the data center.

Finally, VMware intends to marry the service mesh technology from Avi Networks that supports a high-performance ingress controller with its ongoing efforts to integrate the open source Istio service mesh with NSX.

Once all that work is completed, Gillis said it should be possible to deploy a workload on any platform with a single click. The challenge organizations face today is that provisioning and securing the network resources required to deploy an application outside of a public cloud computing environment still takes multiple days to sometimes weeks. VMware has already signaled its intention to make NSX network overlays available across multiple clouds and on-premises IT environments.

Avi Networks CEO Amit Pandey noted that both companies are on a similar trajectory to leverage x86 processors to eliminate reliance on proprietary networking hardware. That shift to software-based networks and security services is driving the biggest transformation in the data center in the last 25 years.

In terms of leveraging application programming interfaces to automate the provisioning of networks, it’s clear networking technologies have been lagging behind advances made in both compute and storage infrastructure. The challenge and the opportunity vendors such as VMware now see is the ability to more holistically manage IT environments spanning from the network to the application. As part of that strategy, VMware has been making a string of acquisitions to extend its core virtual machine franchise.

It’s still too early to say whether all those investments will pay off for VMware. Reliance on VMware technologies in public clouds is relatively slight compared to the dominance VMware has today in on-premises IT environments. VMware is banking that its ability to extend the reach of its portfolio to foster the development of hybrid cloud computing, which is strongly anchored around on-premise IT environments that still run the majority of the workloads in the enterprise, will prevail.

— Mike Vizard

June 13, 2019
DevOps and Networking: Working to Achieve Nirvana

Enterprises don’t own the cloud, just the experience. If there’s an outage or even a hiccup in the performance or network service, it’s on the network and application teams (not the cloud provider) to fix it. But often with problems in the cloud comes finger-pointing, as network and application teams look to determine liability and responsibility when it comes down to crisis situations.

There is no question that applications are growing more complex, as is the infrastructure to power those applications. According to the 2019 “Stack Overflow Developers Survey,” comparing year to year the amount of time it takes for software engineers to become effective on teams is increasing. This trend could be, in part, due to the amount of tooling and rise of an operate-what-you-build model (e.g., the dawn of the full lifecycle developer). As our applications become more distributed—and to cope with scaling and availability demands—skill sets that traditionally were infrastructure and development-based have begun to blur.

Networking Shifting Left

Networking-centric capabilities such as service meshes (e.g., Istio and Linkerd) and container networking interfaces (e.g., Calico and WeaveNet) are gaining a lot of traction and visibility in the cloud-native application development world. Though SDN infrastructure has been shifting left toward development pipelines, application development teams are learning networking concepts. However, the core networking infrastructure is still maintained by the networking teams.

This paradigm of software-defined infrastructure is creating a learning curve for multiple teams. With the dawn of NetOps, software development lifecycle (SDLC) principles are being introduced to the networking teams. Traditionally, ticket- or service-driven networking teams can participate in iterations and they are now building the next generation of cloud-agnostic networking capabilities. With NetOps, there is more interaction and innovation. But change is not without resistance; the dynamic nature of modern infrastructure and the short-lived nature of pre-emptible/ephemeral infrastructure do not bode well for static networking rules.

Networking Team Pressures Today

With infrastructure headed to the trifecta of clustering, replication and load balancing, networking teams have to deal with many more dynamic addresses than before. Static ports and addresses that used to be loaded into networking tickets are now being replaced with wide ranges and service addresses powering the networking stack of our cloud-native and containerized workloads. Adding to the complexity, the networking team is leveraging public cloud infrastructure. The infrastructure that the networking teams are responsible for is leaving the data center and distributed web infrastructure technologies, such as distributed domain name systems (DNS) and content delivery networks (CDNs), are common for the networking teams to be involved with. Like the application development teams, the amount of new technology hitting the networking teams is increasing.

One of the great advantages of using public cloud infrastructure is the ability to quickly expand or contract available resources. With this ability, we can spin up and down test environments to try out changes and even the infrastructure level that was not even possible. With this, we have the ability to integrate more. The application development team, which has been iterating for some time, now has common ground with the networking team to try out iterations.

All in a CIDR

To illustrate the needed collaboration between networking and application teams, let’s look at an example:

In one of the biggest incidents of my career, my company was migrating pieces of our platform to a public cloud vendor. Because the organization was early on in this transformation, a lot of minutiae was yet to be flushed out. We were packaging classless inter-domain routing (CIDR) calculations into our first-time VPC configurations—a task the development team was responsible for at the onset of our cloud migration. It was my job to enter this for my service. Instead of a /8 CIDR, I implemented a /16 CIDR. A seemingly small miscalculation on several configs turned into an incident, blocking a majority of traffic hitting the service. The mistake was quickly corrected, but the incident response framework was well on its way to being executed.

So, who was responsible? I was the one who created the configuration and pushed to production, so this was my fault. But in retrospect, the best solution would have been to involve the networking team immediately in these types of changes. Because the application development teams were vanguarding into the public cloud, however, we cut out the networking team. This was an organizational problem that needed to be addressed. If we were going to deploy into a data center successfully, the networking teams needed to look over my requests and catch my CIDR error.

Silver Lining, Right?

During this CIDR incident, I was trying to look at the silver lining: Since we can’t access our application, that means the instances/app must have a lot of extra capacity, right? Untimely, yes, but the analogy did make sense with the silver lining. We were getting billed by a public cloud vendor and because there was a drop in usage, our bill technically should have been lower. But it still impacted the customer. The money saved in our incident was outweighed by external measures (revenue and brand damage). A cloud outage and a subsequent dip in digital experience can have a massive impact on revenue, averaging $38,855 per hour of outage. During high impact or busier times, this number can be much more.

DevNetOps Nirvana

The adage of bringing everyone to the table early proves to be valuable. Each team in an organization can have a strategic impact and improve architecture in multiple parts of the stack. Different teams have different sets of skills and can have mutual respect for the vertical skills of each of the teams. In the end, a transparent and positive relationship between network and application teams will help with network remediation and outage prevention. And, the closer the teams can work together and the more they can communicate, the more their investments and efforts can contribute to the business bottom line and quicker remediation long term.

— Ravi Lachhman

May 22, 2019
VMware Brings NSX Network Virtualization into DevOps Realm

At the Mobile World Congress (MWC) 2019 conference, VMware moved to bring network virtualization into the realm of DevOps by adding support for a declarative programming model and JSON data formats to version 2.4 of NSX-T.

In addition, VMware is making it possible to use the open source Ansible automation framework to deploy NSX-T.

Finally, VMware is adding support for IPv6 as well as the ability to now extend microsegmentation all the way to Layer 7 of the application stack.

Jonathan Morin, product marketing manager for NSX at VMware, said this release of NSX-T not only will become the dominant version of NSX for VMware in the enterprise, it also represents a concerted effort to make network virtualization directly accessible to both developers and IT operations teams alike. JSON support is critical from that perspective because it makes it easier to programmatically replicate the networking requirements and security controls attached to any class of applications.

Longer-term, JSON support will also make it easier for a wide variety of management tools to consume metrics generated by NSX-T.

IT organizations must determine to what degree they want to fold network operations into DevOps. Some organizations will simply prefer to provide developers with access to self-service virtual networks, while others will move to expose the declarative application programming interfaces (APIs) in NSX-T 2.4 directly to developers, said Morin.

VMware expects that, going forward, a hybrid method for deploying and managing NSX will emerge. Developers will want to programmatically invoke network resources as they build and deploy applications. But network administrators will need to be able to access tools to manage those virtual networks over time. To facilitate that process, the latest version of NSX-T has been updated to add support for an HTML5-based user interface. The goal is to not just distribute NSX-T, but also make it accessible to everyone, said Morin.

Rather than requiring organizations to upgrade switches and routers to enable programmability, VMware has been making a case for employing network virtualization to create a programmable overlay. That approach serves to make the underlying physical network agile enough to be employed with a DevOps framework without requiring organizations to upgrade the entire physical network.

In many ways, networking represents the last major frontier to which DevOps processes need to expand. It’s possible for developers to spin up a virtual machine in the cloud or in an on-premises environment in seconds. But it can still take weeks to manually provision the networking resources in an on-premises environment that a virtual machine requires. VMware has been advocating for the deployment of instances of NSX in both on-premises IT environments and in public clouds to provide a consistent network overlay that can be programmatically invoked and automated.

It remains to be seen to what degree network operations will be pulled into DevOps processes going forward. But the one thing that is clear is that IT organizations have the option to decide to what level they might want to achieve that goal.

— Mike Vizard

February 27, 2019
Go Big: Scalability in the Age of DevOps

Achieving hyperscale is a work of technical art for any business. As companies grow and customers begin to rely more heavily on your service, ensuring reliability during peak business times is mission-critical. An enormous amount of work goes on behind the scenes to ensure scalability, and oftentimes it becomes necessary to redesign or rethink processes and systems to improve them as they face increased pressure.

When customer demand reaches its apex, it’s the ultimate test for any organization’s systems and processes; therefore, it’s important to prepare a comprehensive strategy that focuses on scalability, reliability and hypergrowth in a targeted, calculated and data driven methodology.

The first step on your path to hyperscale is forecasting and gathering data analytics around business and growth patterns for a full year to make smarter forecasts. Set aside time each day throughout the year to go over the allocation of your data and analytics, measuring them for accuracy to see how they stack up to your current predictions for that specific day. This may seem taxing and cumbersome when engineers have other priorities to work on, but devoting time to this in the early months of the year will help you be that much more accurate when it comes to predicting the needs of your customers and how you should go about scaling to meet those needs.

Especially during the holiday season, it’s important to have a minute-by-minute projection of how much traffic you expect and need. That’s where forecasting and an advanced data science model come into play. A common mistake made when it comes to scalability is not getting granular enough about the real pressure points for your systems. Having accurate projections and forecasts in place that feed into your data science model means you can better test your systems and production limits while gaining greater insight into meeting your scaling points. If you find fragilities ahead of time through this, you can assign your engineering team to fix them well in advance.

Another way to truly bolster your scalability efforts is by introducing an operability development model. This entails constantly deploying some controlled chaos into your systems to see how they really stack up and function under pressure, with the goal of up-leveling the capacity of both your operations and system growth. The operability development model allows you to really test the grit of your systems by squeezing the most out of them, prepares you to respond quickly and methodically should the worst case scenario occur and sets you down the path toward continued performance growth and scale.

From here, you’ll want to institute a preventative maintenance model. Think of it like getting an oil change for your car. Instead of waiting until the very moment you need your software to perform exceptionally, start taking a look at the little things that could or are causing problems ahead of time and fix them on a monthly or quarterly basis. A preventative maintenance model helps you inspect, adapt and solve these little nuisances before they become serious problems, granting you the cleanest operating environment possible.

Scalability is king, but availability and reliability are arguably just as critical. Improving availability means reducing latency and getting as close as you can to where your customers utilize your services. This can be achieved by deploying locations around the world to make connections much faster and services more independent. To maintain reliability, you should have processes in place that can quickly respond and offer support to customers 24/7 to maximize consistency and performance.

Another strategy to consider—if it makes sense for your business or if you’re evaluating cloud providers as a growth option—is to double down on your efforts by re-architecting your platform and systems as cloud native from the ground up to support greater levels of scalability and cost efficiency. By going cloud-native, you can achieve independently scalable services so that you don’t have to scale all of your services together. This means your engineers and developers don’t have to spend way too much time figuring out where issues and fragilities occur.

A platform architected as cloud-native allows you to leverage microservices and containerization, creating an environment with more reliable, self-healing, high-tenancy and auto-scaling features that take the most advantage of your hardware and services. An added benefit of a cloud native platform is that you can use your data to find the most opportune times to both scale in advance and descale based on performance and customer expectations. The efficiencies that come from re-architecting as cloud native enable engineers to spend more time on automating software and the delivery pipeline, which is a huge boon for scalability.

At the end of the day, a well-thought-out preparation strategy for mass scalability makes the whole process much more exciting and obtainable. It also greatly reduces lead-up anxiety by getting a head start on planning so that you have more time to sit back and enjoy the fruits of your labor. Lastly, a yearlong scalability strategy enables you to experiment and try new things without the fear of failure. Testing new approaches ahead of time to see what does or doesn’t work will only help you learn and make your scalability efforts that much more effective for the future as your company continues down the path of hypergrowth and hyperscale.

— Jamie Tischart

January 18, 2019
Cisco Looks to Build DevOps Community

Cisco Systems is making a concerted effort to create an era of détente between networking professionals and the rest of the IT operations teams that have embraced DevOps practices.

Susie Wee, senior vice president for the DevNet community at Cisco, said the goal is to provide network operations (NetOps) teams the skills required to programmatically expose a range of self-service capabilities to developers.

At the same time, Cisco is encouraging developers to deploy applications directly on routers and switches that now incorporate general-purpose processors. Those processors would run applications in a way that significantly reduces network latency, while the custom ASICs that Cisco develops would continue to handle all the networking functions.

Wee said that dichotomy is now driving Cisco to classify two classes of developers within its DevNet community. The first is aimed at traditional application developers and the second community is more focused on managing infrastructure as code.

Cisco is now extending that latter initiative into the realm of software-defined wide area networks (SD-WANs). The company has added integrated firewall, intrusion prevention and URL-filtering technologies to its SD-WAN platform. In addition, there are code samples, videos, learning labs and sandboxes available on Cisco DevNet that are intended to make easier to learn how to programmatically manage Cisco SD-WANs using open application programming interfaces (APIs).

Cisco claims to now have 530,000 developers participating in DevNet. One the one hand, the company is trying to entice NetOps teams that have been reluctant to give up legacy command line interfaces to modernize their networking environments. On the other hand, Cisco is trying to attract more application developers who increasingly find themselves encountering latency issues as they move to deploy applications in highly distributed computing environments, such as an Internet of Things (IoT) application.

Over time, Cisco and other providers of network operating systems also will be moving away from monolithic architecture to embrace microservices, which would make it possible to programmatically invoke networking services on a more granular level.

Of course, there may be some developers who would prefer to programmatically take control over networking along with the rest of the IT infrastructure. But Cisco is clearly betting most organizations will want to make it easier for their NetOps and DevOps teams to work more closely together. Today, most enterprise IT organizations can spin up a virtual machine in a matter of minutes; provisioning the network connections to that virtual machine, however, is still measured in days and weeks.

Other networking infrastructure companies besides Cisco have realized the need to make networking more programmable. But in terms of the resources the company is making available to solve that problem, it would appear to ready to make a significantly larger commitment.

— Mike Vizard

November 20, 2018
You Have a Box: The Impact of Containers on DevOps

To continue the discussion from my last blog post, you have a box. Increasingly, you only have to ask, “What do we want to put in it?”

It is interesting to see the growth and change in IT over time. The advent of containers is the focus of this particular blog, but there are many other topics that would be fun to cover in the same manner.

The fact is that containers have become our blank puzzle pieces. We take it, make it what we need and drop it in. They relieve us (for the most part) of the need to worry about deployment issues and allow us to focus on getting the part of the app they represent right.

But those deployment issues didn’t go away, they’re simply obfuscated. Networking, security, storage and even capacity still have to be concerns of IT, just not concerns of a given app or its developers. That creates a separated world of issues, sometimes a split world of issues—such as networking inside/outside the container environment—but these are largely things we’ve had to deal with since the advent of virtualization, just in a larger scope.

The true promise of multi-cloud comes with containers, also. Each cloud vendor wants to lock users into their container architecture, of course, but nothing stops a shop from running containers and container management on cloud instances of their own, maintaining the highest level of portability for the cost of some extra management headaches. In the end, where it is running is even less important in the container world, and that allows what it does to gain more importance.

Management of containers has a bit more overhead; after all, hardware or cloud management needs are the same, and the configuration/management of containers is layered over those needs as an additive. But that’s a worthwhile cost to bear if the result is that tomorrow the container could be thrown to a cloud instance, and the next day for whatever reason—capex versus opex, data accessibility, whatever—could be brought to a local server. Maximum flexibility makes the cost of maintaining another layer of management.

And much of that layer of management can (and usually is, in my experience) be folded into DevOps practices, precisely because a container is 100 percent software. That means the overhead is not nearly as large as it seems. And what isn’t well-covered by DevOps is simply an extension of virtualization.

So with containers, you have a box. The management of the box is handled by DevOps. The only question that really matters is, “What are you going to put in the box?” And that’s pretty cool: Less to worry about during development means more time to make whatever goes in the box rock.

And the innovation around those boxes is just getting geared up, I think. We already see things such as shops/projects with a container life expectancy of less than a minute that are inconceivable in other environments. Not that they’re all good ideas, but the more people doing things in a different manner, the more opportunities for really great discoveries to come along.

And you will be there. Filling boxes with cool new apps. Pretty sweet.

— Don Macvittie

November 1, 2018
The Truth Behind DevOps on the Cloud

DevOps on the cloud may seem simple enough, but in reality it can be very complex

The ultimate goal of a DevOps administrator is to make sure that everything is running properly and seamlessly—so much so that management is convinced that the DevOps administrator is playing solitaire all day. To that end, a successful DevOps team should have their systems automated such that the level of friction they impose on the organization is minimal.

Leveraging a cloud environment makes it look easy—almost too easy, making it look as though all DevOps needs to do is press a button. IT can be deceived into thinking that the level of complexity in DevOps on the cloud is trivial, when in reality it is not.

Running DevOps on the cloud is like renting an apartment in someone else’s building: Even though you don’t own the infrastructure, you’ll have to deal with any issues that may arise. Similarly to a neighbor who’s too noisy, you may experience different levels of performance from the same servers with the same capacity at different times of the day because someone else is running on the same hardware.

There also are security threats that don’t come into play when running on your own hardware. If the same physical machine that runs your most important computations is also being used by Mr. Nefarious, he can use all sorts of tricks and hacks to gain access to valuable data. The Meltdown Inspector attacks is one example—it impacted all cloud providers with severe fleetwide performance impact, sometimes exceeding 10 percent, and required expensive mitigation steps.

Making Sense of Your Gold Mine of Data

The wealth of information provided to system administrators is awe-inspiring, terrifying and frustrating all at once. There is so much information available that, in theory, we should be able to troubleshoot and fix any and all issues. Unfortunately, making sense of the vast amount of information available and nailing down the critical pieces of information necessary can be extremely complex.

The flat network topology typically expected of an on-premises data center is drastically more complex on the cloud. Beyond just VPN and DMZ, there are multiple isolated networks, containers running VMs and running on segmented networks going through NAT with multiple levels of routing and dispatch at play. Trying to troubleshoot in such an environment can be an exercise in frustration, because you don’t own the entire network.

Certain tools and abilities you might take for granted in your own data center are not available, such as Tcpdump, a low-level tool to monitor network packets. For example, in VPN packet loss scenario, your cloud VPN server is shared with other clients, so you won’t be able to capture packets to troubleshoot the connection issues.

Also in managed services—in particular databases—the debug endpoints and tools that often are used in other environments to troubleshoot are not available, to avoid exposing data from other users. For example, you cannot get the full logs from the server to figure out what it is doing. That can hide critical goings that are too precious to ignore in production.

Evaluating Cloud Providers

Since it’s inevitable that problems will occur, one of the important considerations for selecting a cloud provider is the level and quality of support they offer. It’s important to go beyond simple SLAs and evaluate the actual quality and effectiveness of the support you can receive.

Take, for instance, a situation where a machine would freeze, effectively at random, for long periods of time. All monitoring and metrics have told a simple story. The machine was up, had plenty of resources and capacity to spare, but simply would not respond every now and then for any connection attempt whatsoever. Troubleshooting involved stripping the problem to its core and being able to reproduce this in a minimal fashion, at which point one would contact the cloud provider support to try and understand what is going on. The actual problem turned out to be network port exhaustion in the NAT router, which caused a remote server to reset TCP connections, leading to long timeouts that froze the system. This particular problem took days to resolve and required a major change in the deployment topology.

Running DevOps on the cloud is not an easy task, and the cloud itself is not a silver bullet. Rather, it’s a way to democratize hardware and operation resources and work effectively at scale. If you can manage the new challenges, you can enjoy a new level of efficiency.

— Oren Eini

October 23, 2018
DevOps and Containers: A Network Divided

The growth of DevOps and containers has created some interesting split responsibility scenarios. Not that this is necessarily a bad thing, but it is something that bears watching and fixing when it becomes problematic.

Today’s topic is networking. Plenty of bits have been spewed about the situation, but it persists and is even getting more complex as architectures diversify.

Specifically, today’s topic is about the dual responsibility built into today’s networking. Generally speaking, network operations staff are still managing the complex hardware and software environments that deliver packets to your systems. What they do and how they do it is changing—but at a much slower pace than areas fully pulled into DevOps. There are many reasons for this, and I won’t delve into them here, but the truth is that managing a production network and dealing with hardware vendors bring unique responsibilities and constraints that aren’t there in a software world.

That’s experience talking, from a developer who learned network administration. Software is easier and more adaptable, and that is directly reflected in the impact of DevOps.

But we have drifted into a world in which that infrastructure is still mission-critical and still has those same constraints, while the software layer of networking—driven primarily by containers—has grown up in parallel with a different set of people often responsible for it. And that software network can be mighty complex. I shudder to think about what the iptables look like on some—maybe hundreds—of your machines.

Cloud has much the same effect, but since cloud instances often are publicly exposed, they tend to have networking people involved to make sure routing is correct and verifiable, and to help InfoSec people determine that the instance is as locked down as possible (and we can all name companies that wish this were true).

Still, the complexity of cloud spawned companies including CloudCoreo (now a part of VMware), which automate a bunch of network verification and security checks just because there are so many points that require touching. So cloud isn’t the nirvana of shared responsibility either. Yet.

As DevOps and container adoption has grown, it is natural that responsibility for container/container management networking would have spilled over into the development or application administration side. After all, once a subnet is configured for a cluster and software is managing the subnet, which is integrated with the rest of container management, it feels right to have the Dev/deploy team manage it.

And it might be. Like all things DevOps, this is a continual improvement exercise. Having someone who knows exactly what is going on between pods/clusters/networks is important, though. Breaks can cause outages that cost money. Knowledge is the key to fixing breaks and container networking is generally super-complex. Even if you are providing standard IP addresses on a corporate subnet to each container, container management networking is still complex and can still break.

So if you haven’t already, get the networking team involved in inter-/intracontainer communications. Make certain you know what is going where and what should go where.

You won’t regret it, when things inevitably break that cannot be repaired by dropping the instance and spawning a new one to replace it.

— Don Macvittie

October 18, 2018