devopsy.kinsta.cloud

Author: Lori MacVittie

The Curious Connection Between Cloud Repatriation and SRE Ops

I have a fondness for philosophy. I’m about three classes short of a degree, and every few years I tell myself one day I’ll finish it. Thus, I am very familiar with what is known in statistics—and logic—as a post hoc fallacy, from which we get the saying “correlation is not causation.” This is the logical error of assuming that if event Y followed event X, event Y must have been caused by event X. The most famous callout of this fallacy came from Bobby Henderson, who illustrated the absurdity of assuming causation from correlation with his chart demonstrating that global warming was caused by the declining number of pirates in the world. Yeah, that doesn’t make sense, but neither do a lot of charts from which people draw causality. Just because two data points are mapped against each other does not mean one caused the other. In many cases, it doesn’t even make logical sense to correlate the two. After all, pirates and global warming? No one actually takes that seriously. But it’s an important point to make as we dive into the question of the relationship between site reliability engineering (SRE) operations and cloud repatriation.

To be clear, I’m not suggesting that adopting SRE practices causes cloud repatriation. But I am suggesting that there is a close and meaningful relationship between the two. The fact that Google—a cloud provider—created SRE as a practice is not a mistake. The model, mindset and skillsets associated with SRE are integral to successfully operating cloud infrastructure and services.

Cloud Repatriation Is Real

Public cloud repatriation itself is a somewhat taboo topic in certain circles. Consider the controversy raised by Andreessen Horowitz when it published “The Cost of Cloud, a Trillion Dollar Paradox” and suggested companies were repatriating from the cloud and realizing significant cost savings as a result. Some would have you believe it’s not happening, but there’s enough data and anecdotal evidence to indicate that yes, it is.

F5’s 2021 state of application strategy report asked the market about public cloud repatriation. A mere 13% had repatriated apps and another 14% were planning to. One year later, that combined total rose 40 percentage points to 37% and 30%, respectively. This is not an anomaly, as there are multiple credible analyst firms reporting similar results. Interestingly, the rate of repatriation is not globally universal. APCJ and LATAM are both far less likely to repatriate than EMEA and NA.

I maintain that companies are repatriating apps from the public cloud—and the question isn’t ‘Are they?’ but rather ‘How many workloads are they pulling out—and where are they going?’ That’s a question for the State of Application Strategy 2023 research.

For now, let’s consider a possible enabler of repatriation—SRE operations. Because even if the increasing cost of cloud is a driver of the desire to repatriate if you don’t have the skills to operate as efficiently elsewhere—and thus benefit from lower cost—then why would you repatriate?

And we posit that it is SRE operational practices and skills that enable companies to repatriate and maintain the efficiency and cost savings needed to justify the decision—whether they’re moving those workloads to another public cloud, on-premises or to the edge.

Digging Into the Data

On the surface, there is a strong correlation between the adoption and application of SRE practices with cloud repatriation that seems to indicate that organizations with the ability to operate in a cloud-like manner, i.e., they’ve adopted SRE practices, effectively pick up their toys (apps) and go home (on-premises or elsewhere) because they can.

Cut another way, only 4% of organizations that have not adopted SRE practices have repatriated apps from the public cloud. A whopping 73% of those who have adopted SRE practices have also repatriated apps.

Of course, adopting practices does not necessarily mean applying practices. So, we looked at how organizations are actually operating applications, systems, and infrastructure. Specifically, we looked at the percentage of their operations that use SRE practices. Perhaps unsurprisingly, that generated similar results.

Of those who operate 0% of their apps, systems, and infrastructure using SRE practices, 81% are not repatriating. Conversely, of those who use SRE practices for 76%–99% of apps, systems and infrastructure operations, 54% have repatriated. The point at which repatriation appears to begin picking up steam is when organizations surpass using SRE practices to operate more than one-quarter (25%) of their apps, systems and infrastructure.

Remember I noted that APCJ and LATAM were far less likely to repatriate? They’re also far less likely to be leveraging SRE practices to operate their apps, systems, and infrastructure. In fact, over one-quarter (26%) of respondents in LATAM and APCJ (29%) were operating ZERO percent of apps, systems and infrastructure using SRE practices. In EMEA? That’s only 5%. And in NA, even lower at 2%.

Meaningful Relationship or Curious Coincidence?

There appears to be an inarguable correlation between organizations embracing SRE as an operational practice and public cloud repatriation rates. But is it a meaningful relationship or merely a curious coincidence?

I’m going to argue that it’s a meaningful relationship.

The practices and skillsets associated with SRE are wholly suited to operating a cloudy environment—at scale. As I said before, it’s no mistake that it was Google who created SRE and has literally written the book on it. The value of cloud is in its operational model, which can dramatically lower the cost per transaction, whether measured by HTTP exchanges or customer sessions. That enables a cost-efficient scale of applications and digital services.

The use of automation and practices that tend to focus on meaningful incidents rather than non-disruptive occurrences provides the cost-efficient scale of the people (and thus their expertise) who are tasked with maintaining a high level of availability and performance.

The adoption and use of SRE practices enable organizations to efficiently scale operations whether in the public cloud or on-premises or at the edge. And what the data tells us is that organizations appear to be using the capability to do just that.

September 13, 2022
The Definition of Faster in the Age of App Capital

Businesses want speed. One of the impacts of digital transformation—and the associated pressure to succeed in an app economy—is the desire to move at speed. Nearly half (48%) of organizations are driven by digital transformation to deploy faster according to our State of Application Services 2019.

But it’s not all about deployment. It’s also about development and responsiveness to threats and changes in demand.

Organizations want to develop and deliver apps faster. They want to adapt to unpredictable changes in business conditions faster. They want to respond to attacks faster.

So, how are they going to get it?

Faster Development

Deploying more frequently implies you’re first developing faster. That often means adoption of speed-oriented Agile methodologies. GitLab’s 2019 Global Developer Report: DevSecOps found that the majority (54%) are choosing Scrum, while 37% go for Kanban.

Componentization

Methodologies aren’t enough if application architectures aren’t well-suited to development styles. Smaller, focused teams sprinting toward frequent releases isn’t really a good fit for developing new features or fixes for traditional and monolithic applications. Componentization through the adoption of microservices and more distributed forms of architecture fit more naturally with modern architectural and operational principles. It’s no surprise that, on average, more than 80% of a modern application is made up of third-party components, heavily weighted toward open source.

APIs

Adoption of APIs, as well, is no surprise amongst those seeking faster development through componentized application architectures. APIs decouple implementation from interface and allow teams to make modifications to processing without impacting the use of APIs from other components or applications. This is a common pattern, with an impressive 64% of organizations creating APIs today for use in either internal or external use cases. Nearly 50% of those organizations rely on APIs to get ideas to market faster according to Jitterbit’s State of API Integration 2018.

Automation (CI/CD)

The build pipeline, too, has to keep up if frequently deployment goals are to be met. That means CI/CD tools that seamlessly move code from commit to test to release. The most popular CI and build tools in GitLab’s survey were, unsurprisingly, GitLab (61%) followed by Jenkins (36%) and Travis CI (12%). Notably, our own research finds Jenkins in use by 16% for network automation as well—a promising result if organizations are seeking to extend DevOps beyond delivery into deployment.

Faster Deployment

Developing faster doesn’t get ideas to market faster. That requires deployment. While born-in-the-cloud technology companies have mastered the delivery-deployment divide with ease, many established enterprises find this transition challenging. Existing organization structures as well as the continued need to support traditional and monolithic applications introduces conflicts that can be challenging to integrate with modern operational demands. But there is no doubt that organizations must overcome these challenges for those applications requiring faster and more frequent deployments.

Continuous Deployment

Don’t be fooled; established organizations are embracing automation and applying it to the deployment pipeline. The issue is often that traditional IT structures introduce inconsistent automation and self-service efforts. We see this play out in our own research, where strongly siloed team structures continue to impact automation efforts across IT.

Team structures matter and organizations whose compass is pointed at automated pipelines will need to address the cultural aspects of continuous deployment if they want to consistently deploy faster.

Public Cloud

Traditionally, the inability to achieve continuous deployment has pushed developers and app owners to turn to the public cloud, because it eliminates the obstacles in the slower, enterprise deployment path. The disparity in opinion on frequency of deployments is, in part, a contributor to the issue. Our 2018 NetOps/DevOps survey found that while 55% of DevOps and 52% of cloud architects felt their organization did not deploy frequently enough, only 30% of NetOps and the same percentage of operations agreed.

But it’s not the only driver. Digital transformation is certainly a factor; 33% of respondents in our State of Application Services indicated they are intentionally moving to deliver apps from the public cloud as a result of embracing digital transformation initiatives. The ability to easily integrate application services and subsequently automate operation is a huge boon to organizations seeking a faster route to frequent deployment.

Containers

In the cloud—but also on-premises—continuous deployment often requires the ability to adopt per-app pipelines and application services that support the model. Increasingly, organizations are turning to containers because of their ability to support rapid updates and operate seamlessly in highly volatile environments. We see containers in demand not only to support modern application architectures such as cloud native—the top container use-case at 33% of respondents to Diamanti’s 2019 Container Adoption Benchmark—but also for infrastructure. The demand for on-premises application services in containers has consistently grown year over year in our own research, rising from just 4% in 2017 to 15% in 2019.

Faster Responses

Not everything is about delivery and deployment. Organizations also need speed when it comes to security and operations. In today’s world, where more than half of all app interactions are conducted by bots, it’s important for organizations to respond quickly with a reject message to avoid falling prey to exploits or infection.

Real-Time Threat Analytics

One of the ways organizations are seeking to speed up their ability to respond to attacks is through the adoption of real-time threat analytics. With security continuing to be a top priority and consistent challenge, it was no surprise to see this category ascend into the top five strategy trends and technology in 2019 with 41% of respondents tagging it.

The availability of intelligent security application services will accelerate as machine learning and automation continue to apply their considerable capabilities to the problem of identifying troublesome traffic faster.

Orchestration Included

Machine learning isn’t the only way in which infrastructure and application services are getting smarter. As the need for faster responses to capacity and processing grow along with users, infrastructure and application services are evolving to include orchestration as a core capability rather than an add-on. Application service platforms with integrated orchestration layers will mean services that scale along with demand—on-demand. While such capabilities exist today—indeed, they are part and parcel of cloud and containers—the capabilities to automatically scale up and down based on application and user defined requirements is not native to most systems today. But they will be.

Faster Identification of Bad Actors

Finally, organizations need to combat bots for both business and security purposes. Whether organizations need to stop scraping—a real threat to business—or prevent bots from probing for vulnerabilities, identifying bad bot actors faster is critical today. Checking a box (I’m not a robot) isn’t enough today as bots are growing more intelligent and are able to foil such primitive techniques.

Organizations are turning to Bot Defense services capable of applying more modern, successful techniques to quickly identify and block bad actor bots. We’ve seen the use of bot defense services grow quarter over quarter in response to this need, and expect this trend to continue.

Faster Development, Faster Deployment, Faster Responses

When it comes to business’ need for speed today, we wind up looking at processes. Development processes are automated to gain speed with CI/CD. Deployment processes are automated to gain speed with continuous deployment and cloud. Scaling and security processes are being automated by systems and the introduction of orchestration as native to application service platforms.

Faster, it turns out, relies on an organization’s ability to automate processes across development, deployment and security.

— Lori MacVittie

February 14, 2020
Sharding for Scale: Architecture Matters

In the days of monolithic applications, scale was simple. As every instance of the app was the same, it really didn’t matter which one processed a request, as long as it was processed. On the back end, a single, ginormous data store held everything. This model ran into serious issues as apps migrated to the web and APIs introduced even more users. The ability of the database to scale efficiently without impacting performance became a serious issue.

A microservices-based architecture, on the other hand, is far from simple. Not only are there more moving pieces to track, but our understanding of scalability has matured and morphed under the staggering weight of the increased use of apps and APIs over the past decade. Some of those moving pieces are intimately tied to the decentralization of data, which is no longer monolithic, either, and so the microservices responsible for interacting with that data are impacted by these new data models.

Sharding was introduced before microservices existed. The premise was simple and based in part on the foundations of load balancing: Distribute the load. Data stores were split up and given responsibility for only a subset of data. This made them more efficient and faster, which in turn benefited everyone.

As microservices have become more prevalent to improve development velocity and agility, they also have introduced new variables into the sharding equation. It’s no longer a matter of just sharding the data; it’s now a matter of how to distribute the responsibility for sharding throughout the architecture.

The way in which you design your microservices—your decomposition strategy, if you will—to handle sharding has a serious impact on how ultimately you scale the services. It’s no longer just a matter of tossing a load balancer in front and praying for the best.

Sharding Decisions

The Scale Cube, as defined by the most excellent book, “The Art of Scalability,” provides some guidance on architecting for scale. Of particular note are Y- and Z-axis scaling.

In Y-axis scaling, applications are decomposed into individual services, each responsible for a single or closely related set of functions. This allows each to scale independently. It is well-suited for microservices, obviously.

With Z-axis scaling, the focus is on scaling by cloning. That is, multiple copies of the same service are horizontally scaled to increase capacity. The kicker is that each of those copies is responsible only for a subset of requests or data. It is most often used to scale databases (sharding) but it also can be beneficial for scaling based on location or another identifiable user-based attribute. Two of the most pressing challenges with a Z-axis-based architecture is granularity of the shard definitions—too fine, and you waste a lot of resources. Too broad, and you risk running into performance and scale issues if you succeed and grow.

Best practices today are to employ some sort of data sharding. So, the question is not whether you will shard your data; it is, what piece of your architecture will be responsible for managing it?

There are primarily two options: shard at the microservice layer or at the data layer.

Sharding at the Service Layer

Sharding at the microservice layer takes advantage of a Y-axis scale pattern. It assumes there are services designated for handling a subset of requests. This pattern requires an upstream decision point capable of inspecting requests and determining how to route them. This means data must be sharded on some identifiable variable carried within either the HTTP headers (preferable) or within the payload (data) itself.

Architecturally, there are a number of ways of scaling this pattern. The simplest is to use a single, smart ingress “router” to both route and scale at the same time. The more complex way is to use an ingress router to route requests to load balancers that scale each service individually.

It is important to note that containerized environments—popular for deploying microservices—operate primarily on the more complex pattern. A single ingress controller acts as the decision point, with either native or plugged-in load balancers distributing requests across the nodes that comprise the service.

Sharding at the Data Layer

Sharding at the data layer is easier on the overall architecture, but couples microservice code to your sharding strategy more tightly. This is because the services take on the responsibility of routing and must implement the sharding strategy. Any microservice can accept any request. The microservice then determines which data store is appropriate, given the relevant variables.

This pattern relieves complexity on the ingress as there’s no real need to do anything but distribute the load across available microservices. It does, however, increase network complexity on the back end between the service and data tiers, as every service must be able to communicate with every data store.

Depending on your implementation, this can require more networking assistance to ensure optimal performance and routing.

This pattern also puts more stress on the data stores, as they are required to communicate with every instance of the microservice. Depending on your language of choice, this can have unintended consequences on capacity and performance as the database connection pool becomes the primary bottleneck in scaling.

Shard Smartly

There is no single “best” answer for how you should shard data when adopting a microservices architecture. There are pros and cons to both, and each method has consequences. Understand the impacts—not just on the microservice architecture but on the overall architecture, including scale. As scale has become a must-have rather than a nice-to-have capability, it’s important to understand the impact of application architectural choices on that function.

Shard on. Scale smart.

— Lori MacVittie

October 15, 2017
Automation: Critical Missing Pieces in the CD Puzzle

As enterprises accelerate the frequency with which they develop and release new code into production to meet the ever-mounting demands of their consumers, there’s growing pressure on networking and security operations teams to keep pace with their DevOps counterparts. And it’s not hard to see why. According to Forrester, nearly four out of five business leaders believe it is IT’s responsibility to ensure the network can support the company’s digital transformation plans.

And yet, 67 percent of businesses believe the network is the bottleneck—and with good reason. The current networks that NetOps and SecOps teams manage are highly complex, distributed entities that have historically been configured and provisioned manually—and often, in silos. Yet, as DevOps accelerates ahead at lightning pace, underpinned by the continuous integration, delivery and deployment (CI-CD) approach, NetOps and SecOps teams are scrambling for new ways to gain speed to more quickly launch, scale and secure applications and code updates in production.

The Need for Speed

Enterprises are increasingly leveraging DevOps principles to execute daily deployments into their multicloud test and production environments. For example, Etsy deploys 80 times per day, while Netflix and Amazon deploy up to thousands of times a day. In this new world order, collaboration, communication and synchronization across development, operations, network and security teams is essential.

This also means that testing of the network and application services infrastructure must happen concurrently with the application development and testing processes—not afterward, which can dramatically slow down the agile CI-CD process. Similarly, security cannot be retrofitted after the development and test phases are complete. A recent study by Puppet Labs found that high-performing teams spend 50 percent less time remediating security issues when security is addressed at every stage of the software development and delivery cycle.

New CI-CD Tools Deliver on the Promise of DevOps

New innovations in virtualization and programmability of network and application service infrastructure are making it possible to test policies and configurations prior to production—even in highly complex environments. Powered by validated templates and APIs, these network orchestration and application services tools help automate task execution for scale, preserve provisioning and configuration policies for consistency and bolster security, while dramatically reducing management time cycles and the risk of human error. They empower NetOps and SecOps teams to leverage DevOps principles and apply them to network and security during the development, deployment and delivery of multi-cloud apps and code updates.

Similarly to how software-defined components such as servers and storage are provisioned, network orchestration tools leverage a software-defined infrastructure (SDI) approach to rapidly provision network services across multiple network devices. These tools support flexible application provisioning across physical and virtual environments, help identify and address issues quickly and improve overall network performance. In the CI-CD model, network orchestration helps NetOps and SecOps teams ensure that all production-critical application and network elements—such as availability, core network services, security services and performance—are accounted for and are rolled out in proper sequence and at scale.

Application services automation focuses on application availability, performance, and security. While automation used to focus solely on simple load balancing, it now also includes a range of solutions that help network teams configure and automate application services during the test phase. These include web app and network firewalling, DDOS protection, application acceleration, WAN optimization, DNS services and TCP optimization, among others. In the dev and test phases of the CI-CD process, automation allows for faster configuration and deployment of network settings and policies. Similarly, in the production phase, automation significantly speeds up and simplifies areas such as policy adjustment and other time-intensive processes by eliminating the need to reconfigure underlying infrastructure. This means that ticket resolution and necessary changes can be completed in a matter of minutes versus weeks. At the time of rollout, application services automation toolsets also help to ensure the infrastructure is ready to perform at peak levels.

Network and application services automation innovations—the critical missing pieces in the continuous deployment puzzle—work together to ensure the entire DevOps team—including network and security—can deliver and deploy applications with the speed, reliability and security necessary to meet today’s business challenges.

About the Co-Author / Ranga Rao

Ranga Rao is Director of Product Management & Solutions Engineering at Cisco Systems, where he leads a team of product managers, architects/TMEs and software engineers to collaborate with partners to define use cases and build software integrations for leading Cisco solutions including Cisco ACI, Nexus 9000 and Tetration Analytics. Ranga earned an M.S. in Computer Science at Stanford after receiving his B.E. from the National Institute of Technology Karnataka. Ranga is also an author of a popular book, Programmability and Automation with Cisco Open NX-OS, published by Cisco.

— Lori MacVittie

August 9, 2017
How DevOps Can Help Improve Security
Automation and standardization along with development-inspired reviews can address the three top risks to IT security.

The term “DevOps” these days tends to evoke images of automation and push-button application deployments whenever app dev wants. It’s anarchy, it’s chaos, and it’s a frightening notion to those for whom stability and security of the core business network is their top priority. After all, folks in the DevOps camp routinely cheer on a “Chaos Monkey,” whose sole purpose is to break things in the production network. On the surface, that hardly seems conducive to stability or security.

So, it might be surprising to discover that the seemingly laissez faire approach to building and delivering apps with DevOps is actually one of the best ways to combat significant risks to IT security.

Not kidding.

First, let’s consider the survey, conducted by SpiceWorks, in which IT pros were asked to rank a set of threats in order of risk to IT security.

According to the report, the respondents ranked the following threats as their organization’s biggest three risks to IT security as human error, lack of process and external threats.

All three of which DevOps can positively impact, without negatively impacting stability or reliability of the core, business network. Let’s examine how, shall we?

Human Error

We’ve all fat-fingered configurations and code before. Usually we catch them, but once in a while they sneak into production and wreak havoc on security. A number of “big names” have been caught in this situation, where a simple typo introduced a security risk. Often these occur because we’re so familiar with what we’re typing that we see what we expect to see, rather than what we actually typed. We’ve entered that command a hundred times, after all; we know what it’s supposed to look like and we aren’t as careful as we (likely) should be because it’s Friday and it’s almost 5 o’clock. Whatever the reason, we’ve done it, and it’s the repetitious nature of the beast that actually ends up biting us in the proverbial derriere.

Codifying those commands into scripts or, even better, into templates means once it’s right, it’s right. The inescapable reality that we will fat-finger something eventually is reduced to virtually nil because we aren’t typing it in over and over and over, each time with less and less attention. The reliance on codification, on templates and APIs and scripts as a means to automate tasks, is not just about speed but also repeatability. And accurate repeatability at that. That’s the kind of assurance you need to prevent “human error” from exposing apps—and the business—to a wide range of risks.

To reduce risk from human error via DevOps you can:

Use templates to standardize common service configurations;

Automate common tasks to avoid simple typographical errors; and

Read twice, execute once.

Lack of Process

This is probably my favorite risk (and not just because it lets me talk up Six Sigma and some really hairy mathematical equations) because it encompasses such a broad category of problems. First, there’s the fact that there’s almost no review of the scripts that folks already use to configure, change, shut down and start up services across the production network. Don’t let anyone tell you they don’t use scripts to eliminate the yak shaving that exists in networking and infrastructure, too. They do. But they aren’t necessarily reviewed, they are certainly aren’t versioned like the code artifacts they are, and they rarely are reused. Everyone has their “own” favorite language and scripts, and they are the Frank Sinatra of IT: They do it their own way. (Yes, I know you’re probably too young to get that reference, but until your musical icons have a better one, that’s the one I’m using because my generation didn’t have a better one, either.)

The other problem is simply there’s no governed process. It’s tribal knowledge. Bob does X and then Alice pushes Y and someone from Team Rocket* pushes button to make it go live. What can happen—and sometimes does—is one step is simply overlooked. It isn’t that there is no process, it’s that there’s no real standardization and no overarching governing process. There’s no one coordinating between Bob and Alice and Team Rocket to ensure the process goes smoothly. The thing is that the app might go live and actually work in production minus a few services, but if those services are related to security in any way (such as locking down a port), then your lackadaisical approach to the deployment process just put the business at risk.

To reduce risk from lack of process:

Define the deployment process, clearly. Understand prerequisites and dependencies and eliminate redundancies or unnecessary steps.

Move toward the use of orchestration as the ultimate executor of the deployment process, employing manual steps only when necessary.

Review and manage any scripts used to assist in the process.

External Threats

At first glance this one seems to be the least likely candidate for being addressed with DevOps. Given that malware and multi-layered DDoS attacks are the most existential threats to business today, that’s understandable. There are entire classes of vulnerabilities that only can be detected manually, by developers or experts reviewing the code. There’s DevOps in that, but it doesn’t really extend to production, where risks becomes reality when it’s exploited.

More extensive testing, and development of web app security policies during development that can then be deployed in production is one way that DevOps can reduce risk. Adopting a DevOps approach to developing those policies—and treating them like code, too—provides a faster, likely more thorough policy that does a better job overall of preventing the existential threats from being all-too-real nightmares.

To reduce the risk of threats becoming reality:

Shift web app security policy development and testing left, into the app dev life cycle.

Treat web app security policies like code. Review and standardize.

Test often, even in production. Automate using technology such as dynamic application security testing (DAST) and when possible, integrate results into dev life cycle for faster remediation that reduces risk earlier.

There is no technology, methodology or approach to IT security that will completely eliminate risk short of a complete power outage. Given that’s not a viable option, the best approach is one that reduces risk to acceptable levels (where acceptable is defined by your business’s tolerance, regulatory requirements and how much your MBO is based on not being hacked this quarter). DevOps and security are a good fit, as the former can help standardize, codify, automate and expand security practices across production and dev/test.

So get out there, and put some DevOps in your security. And while you’re at it, put security in your DevOps.

* Yes, that’s a Pokémon reference. I figured I needed something more modern after tossing Frank Sinatra in there. If you got both, give yourself +5 geek points.
July 21, 2016
Infrastructure as Code is SDN for Ops

As we continue to recognize the importance of DevOps—particularly its automation and orchestration aspects—to achieve the agility necessary for continuous deployment of apps, the question of how the term “Infrastructure as Code” (IaC) applies to traditionally network-bound systems and services is bound to crop up more frequently. After all, in addition to core IP-based networking characteristics required for an app to communicate and serve up content, an entire ecosystem of app services that support, secure and scale it must be considered.

IaC is driven by the application of development principles to network infrastructure. Configuration, scripts, templates, profiles and the like are treated as code, in that they should be reviewed, versioned and provisioned from a centralized repository (one hopes on-premises) rather than managed as discrete network elements.

In a recent article in TechBeacon, Boyd Hemphill, director of evangelism at StackEngine, noted, “The basic principle is that operators (admins, system engineers, etc.) should not log in to a new machine and configure it from documentation.”

Christopher Null, CEO of Null Media and author of the article, then wrote, ” … But IaC is a concept that extends beyond simple infrastructure automation. IaC requires applying DevOps practices to automation scripts to ensure they’re free of errors, are able to be redeployed on multiple servers, can be rolled back in case of problems, and can be engaged by both operations and development teams.”

The idea is that you can describe any given service as a set of artifacts and deploy (and redeploy) them at will. So rather than set up a system and configure it directly, its configuration any associated artifacts (such as profiles or templates) are treated as code and stored in a common repository. A controller—the orchestrator—is responsible for distributing those artifacts when necessary. The controller uses an API (or protocol such as OpenFlow) to manage the configuration of services across a distributed infrastructure. It’s decoupling the service definition from the service and its platform.

Read that again. It decouples the service definition from the service and its platform and assigns responsibility for provisioning and configuration to a controller. Kind of like decoupling the control plane from the actual forwarding and switching actions (the data plane) and assigning operational responsibility to an external controller.

At its simplest, treating infrastructure as code is separating configuration from the platform and enabling a software-driven or software-defined approach to provisioning, configuring and managing infrastructure. It’s applying the principles of software-defined networking (SDN) to operations, rather than to devices and systems.

But as Null makes clear in his article, IaC goes beyond just automation and orchestration, adopting the best practices of development with respect to code reviews, versioning and management of that “code” to ensure consistent, predictable and repeatable deployments.

It’s all driven by software, ultimately, and deployed by software, and defined by software but always under the auspices of the operational experts: the engineers and architects.

Treating infrastructure as code is basically applying the architectural principles of SDN to operations itself. It’s SDN for ops.

April 22, 2016
In DevOps, Standardize to Minimize Architectural Debt

I recently reading a great Q&A with the Etsy CTO John Allspaw over on The New Stack, which contained some nuggets of good advice that increasingly are important as we continue the journey toward DevOps adoption. Of particular note were statements on the introduction of “new” things to solve problems. In a nutshell, Allspaw reinforced the importance of standardization as a means to reduce architectural debt.

If you aren’t familiar with the concept of architectural debt, it’s similar to technical debt, in which choices made today incur longer term investments of time and money when future changes are introduced. If you settle on a caching solution such as Redis, for example, there are technical and architectural implications. You can’t simply switch out Redis for memcached later without significant changes to both code and the infrastructure.

Even your choice of load balancer today implies architectural debt because of the growing inclusion of scalability as part of the application infrastructure itself. The result is integration with automation and orchestration systems that are not easily changed later on if you decide you want a different solution. The style of service registry you choose is an architectural decision that impacts clients, servers and systems across the microservices architecture. Switching from client-side to server-side will later be costly.

That’s architectural debt.

Allspaw noted the need to not reinvent the wheel when possible, and instead encourage the collaborative aspects of DevOps as a means to avoid reinventing the wheel and potentially increasing the architectural debt load on the organization.

This is not a new concept. There’s a reason enterprises in particular proudly proclaim themselves “Java shops” or “Microsoft shops” and primarily require development in one or perhaps two languages, but no more. Maintainability is a component of technical debt, just as middleware decisions are a component of architectural debt.

So as we continue to see wider DevOps adoption, it’s important that we carefully consider the impact of our choices in terms of the automation, orchestration and integration required to implement continuous delivery and deployment. Once you’ve invested time and energy (and probably budget, too) in Puppet or Chef, VMware or Cisco, OpenStack or SaltStack, you’re going to find an unavoidable amount of technical and architectural debt in development as well as production across infrastructure and networking teams.

Each of these systems, solutions and frameworks comes with its own components. It isn’t just a language, it’s a comprehensive system with management and repositories and agents and reporting components. All of these impose certain architectural requirements on organizations, from needing certain software running to prescriptive details on networking that must be enforced, lest the system break down.

We’re already starting to see that fragmentation in enterprises that are engaged in automation and orchestration efforts. When asked the number of automation frameworks (Cisco, VMware, Python, OpenStack, Chef, Puppet, etc…) in use, one-third today reported using three or more.

That’s not necessarily “bad,” as it could be indicative of standardization across silos (networking wants to use Cisco, infrastructure is already on VMware, and Python is being used to cobble it together by ops), but it also could be an early sign of the kind of fragmentation that causes exponential growth in technical and architectural debt.

Certainly there is room for experimentation in these early days, but it behooves us to maintain a watchful eye on the use of frameworks and systems and languages we employ for automation and orchestration to ensure we don’t end up buried in technical and architectural debt.

Standardization, even loosely enforced, can help minimize the impact of that debt before it spirals out of control.

March 14, 2016
In DevOps, Business and Operations Metrics Both Matter

Proponents of adopting a complete DevOps approach to application delivery and deployment often note that DevOps includes a change in culture. And not just a change in lateral change in culture—improvements in the way teams across (and within) silos communicate and collaborate—but also horizontal change that permeates every aspect of the organization, from the top down.

That change means more than sharing information, it means sharing the right information. And in the case of convincing business stakeholders and even executive leadership that DevOps is the right approach to succeeding in this fast-paced, API and app economy, that means communicating its impact as it relates to their primary concerns.

Surveys such as Puppet Labs’ State of DevOps has done an excellent job of providing insight into the operational impact of DevOps. We’ve seen improved mean time to recovery (MTTR) and shorter lead times required. If you’re familiar with these measurements, then said improvements no doubt provide an excellent reason to at least begin adopting DevOps.

But what about executive and business leadership? They’re likely no more impressed by an improved MTTR than you are by a 10 percent reduction in average handle time or improved call resolution rate. To get buy-in from business and executive IT stakeholders, we have to provide proof of positive impact in their metrics. CA Technologies illustrated this divide in its DevOps Jigsaw survey, in which 86 percent of respondents consider “business stakeholder education” important, and yet only 33 percent have completed.

Communicating the benefits of DevOps in a meaningful way to non-technical people is an important part of that educational process. That could mean translating MTTR to dollars saved by reducing downtime, or improvements in app performance (response times) translated into productivity increases or reduced abandonment rates. That means focusing as much on measurement after delivery as it does measurement of delivery. It means crossing the chasm (and I know it’s a big chasm, trust me) between marketing and IT to understand abandonment and conversion rates and how they relate to performance and availability. It means making connections between IT metrics and business impacts and being able to connect the two and educate business leaders on how DevOps will or can make a difference.

It means, at a minimum, showing a direct and positive correlation between DevOps and a more agile IT environment and the metrics that consume the lives (and dashboards) of executives and business stakeholders. You don’t have to necessarily do the math for them (though that would be nice if you could), but being able to connect the dots between operational and business metrics should be considered minimum viable effort. And it’s a good place to start the process of affecting the cultural transformation required to sustain the benefits of a DevOps approach.

February 8, 2016
Survey says network agility is still immature

At least that’s the word from SDx Central’s Cloud Management, Orchestration and DevOps 2015 Report in which 76% of respondents said the network was the most immature component in the cloud management stack.

Now to be fair, the network is one of the last components (I’m looking at you, security) to get on board the “automate or die” train, which has already left the station. That’s partly due to pressures from increased DevOps adoption in general and more specifically thanks to greater enterprise penetration which is continuing to up the amperage in the data center with respect to velocity and ease of deployment of all the network bits, bytes, and pipes needed to support each and every application.

It’s also partly due to the complexity of the network environment and the blast radius of potential errors or misconfigurations as well as the broad set of differences in the object models (and therefore APIs) used to configure, manage, and automate provisioning of network services. The more features and capabilities, the more differences between them.

OpenStack manages this difference by forcing standardization; that is, it defines the interface for LBaaS and if you want to be a part of the stack, you have to conform. Cisco ACI manages the difference by taking a declarative policy-based approach that let’s plug-ins define their own interfaces and expose them to ACI in a standardized way. VMware NSX and related technologies take yet another approach, leveraging its ecosystems’ APIs to integrate and orchestrate the network.

All are relatively new to the game (OpenStack may be the exception but even its network implementation has lagged far behind other components of its system) and thus the perception that the network is the most immature in the cloud management stack would seem to be reflective of reality.

But at least it’s heading in the right direction.

The network was going to have to be included at some point in this grand journey to transform the data center through automation and orchestration. At some point Agile and DevOps methodologies being embraced by dev and ops were going to start paying off in the form of more applications with more frequent deploys. Eventually that was going to overwhelm the network (and its staff) and give them cause to start figuring out how they were going to transform the network from the static and very fixed methods of scale and management to a more modern dynamic and fluid method.

Eventually is now. Whether it’s DevOps expanding into the network or SDN or cloud management stacks, the network has to mature in terms of its orchestration and automation capabilities if organizations are going to continue on their journey toward the agile network.

We’re still trying to move out of the first phase in that transformation; toward a culture inclusive of measuring and standardization with templates (or configurable policies, if you prefer). We’ve got a long way to go toward a sustainable, agile network and just as it’s taken years for Agile to truly envelope most projects within the enterprise, it’s going to take years for the network to get from where we are today to where we suddenly want to be yesterday but accept that we won’t get there till tomorrow.

October 21, 2015
Imperative or Declarative? The question of DevOps in the network.

A significant amount of bandwidth is spent discussing automation and praising the value of APIs within the broader DevOps umbrella. These concepts are particularly important in the context of CI/CD and the evolution of application architectures toward decoupled, service-oriented systems. Automation is used for build and test, to drive the workflow that moves an app from disconnected sets of services to a working release candidate ready to breach the wall of production and be put into the hands of adoring fans impatient consumers.

But in between the consumers and your app stands the production pipeline; a process-filled wasteland through which an app must travel so that the appropriate services required to deliver it can be provisioned, configured and put into place. It is this pipeline that will meet DevOps head on next, whether we call it DevOps or SDN or SDDC. It stands in the way of an increasingly critical “time to market” metric by which all of IT is being measured. Research shows that the production pipeline is a major pain point for IT; taking weeks or even months to traverse. The network is still in the way, and that must be addressed if C-level concerns about meeting increasing pressure to get apps to market are to be met.

Applying DevOps principles to the infrastructure and network makes sense as it’s as much about optimizing processes (workflow) as it is writing scripts and it is the production pipeline process that would benefit. Automation and orchestration, of course, will help drive a more consistent, efficient, and repeatable production pipeline traversal. Identifying where inefficiencies (latency) lies in the process and eliminating it by establishing more efficient (and perhaps automated) handoffs using automation and orchestration systems can go a long way toward shortening the amount of time between hand off to production and delivery to consumers.

APIs are our “go to” technology today for driving automation, orchestration, and integration with the frameworks and systems we rely on to scale operations. Most every infrastructure and network device today has an API and can be controlled via frameworks like Puppet and Chef as well as directly using .

But most are imperative methods of control. It requires processes to be strictly defined through a sequence of API calls. Unlike simpler resources like virtual machines, one does not simply need “start, stop, or pause.” Configuration that was once (and still is, let’s be honest) accomplished via CLI (command line interfaces) using SSH is simply migrating to an API using HTTP instead.

But otherwise, not much has changed. The same commands used on the CLI are now sent via an API. We’re still in an imperative mode and if we need to tweak or adjust the configuration, we need to tweak and adjust the script that drives the configuration.

Compare that now with a different approach: a declarative approach. Using templates to describe what needs to be configured or accomplished (data) as opposed to how it needs to be accomplished (commands) makes a significant difference in not only the process but in the long-term technical debt acquired (or not acquired, as the case may be). APIs change; methods deprecate, versions matter. Treating application services (load balancing, caching, optimization and acceleration, security) as “code” (templates) instead of a series of API calls can dramatically change the velocity with which such services can be provisioned and configured. This has the added effect of reducing the amount of churn in the automation code which should reduce technical debt as well. Such methods are also more portable, enabling easier migration into cloud environments for applications that might be (or will be) moved into or around different environments.

Ultimately, the question is whether or not you want to tie your processes – and the interfaces between workflow steps in that process – to many API calls or a standardized set of API calls that leverage declarative policies (templates) to provision and configure the services that make up the production pipeline.

June 1, 2015