devopsy.kinsta.cloud

Author: Lori MacVittie

The DevOps value stream is not SOP

#devops #lean Don’t confuse doing something faster with greater efficiency.

Much of the focus on DevOps remains at the implementation level – on gaining efficiencies through automation of tasks and orchestration of processes. While this is not a bad thing, such a micro-focus on the moving parts can result in the loss of efficiencies to be gained at the macro-level.

One of the biggest inhibitors of a timely release cycle is not the moving parts, but rather the process in which those moving part are executed. It’s not necessarily the actual push into production that’s problematic, but the length of time it takes to accomplish certain tasks when they cross internal organizational boundaries.

For example, many years ago we began the process of pushing a new application into production – a web application. This required, perhaps obviously, changes to the corporate firewall. The “process” for accomplishing this change – essentially opening a port on the firewall for the intended public IP address – required no less than five different signatures across networking, security and architecture teams. There were only two security roles allowed to approve such a change, and the process was noted as taking up to 15 days to get a signature on a piece of paper and subsequently a rule change.

This was not a task level problem. It was not implementation issues. It wasn’t even difficulty in making the change on the firewall. It was all about the process. Scheduling a release meant taking into consideration not only the readiness of the application itself, but its supporting services – including networking.

This is why DevOps should start with a review of the processes used to release an application into an environment (whether test, QA or production). Mapping out the process is the first step to finding where the bottlenecks might be that are causing delays in moving that application to the next phase of its lifecycle.

Simply automating an existing process may not provide the value expected or touted by DevOps supporters, because if the process is poor, you just end up doing something poorly much faster.

In other words, the value of DevOps is not necessarily in simply automating SOP (Standard Operating Procedure).

The initial Value Stream Map captures the current process as it exists. This is known as the “As Is.” It is the actual process flow and should be created through observation of the current flow and with participation of those who are in the process. Do not make the mistake of pulling a process map from the procedural manual (SOP). The best technique for mapping is to be the unit (the “thing”) coming through the process. For example, in the insurance industry it might be the policy. In the auto industry, it may be the car. In each instance, the Value Stream Map begins with the request for the unit i.e. insurance policy or car. [emphasis mine]

http://www.goleansixsigma.com/how-to-visualize-a-process-with-a-value-stream-map/

For application deployment, the “thing” is … an application, and the process should be mapped based on what’s actually happening. Only when you have the process mapped can you evaluate it for bottlenecks and understand the steps of the process at which each group adds value. Value is defined as any step that moves the thing (the application” toward the end goal. Efficiency is then measured as a percentage; specifically it’s the value added time as a percentage of the total time it takes to complete the process (the cycle time).

Don’t confuse doing something faster with greater efficiency. If every step of the cycle is automated, the actual process efficiency hasn’t changed. The ultimate goal of DevOps should be about improving the efficiency of these processes, not just doing what’s always been done faster.

Some of the opportunities for improvement in efficiency that will surface is certainly interfaces with other organizational groups. Perhaps it’s that a process step might not take as long if you had relevant data or information up front. If you’re wasting time waiting for some piece of information, that’s not value added time. It’s just cycle time. More cycle time reduces the efficiency of the overall process by overloading it with wasted time. Better information gathering up front can help reduce the wait time and immediately provide greater efficiency of the process.

It is these opportunities for improvement that will be particularly helpful in the enterprise arena, where the issue is not so much attempting to hit ten deploys a day, but rather the predictability and efficiency of the processes that enable teams to hit a specified change window as desired.

DevOps is not about the tools, or the automation of manual tasks. It’s about the big picture, and that means gaining greater efficiencies through process re-engineering and orchestration as well as automation. Focusing on the micro aspects of DevOps will only ensure that if you’re doing it wrong, you’re doing it wrong faster. Step back and look at the big picture, and find those process steps that are really causing inefficiencies in your organization.

June 16, 2014
Avoiding Parrot Processes

One of the things devops practitioners are tasked with is the provisioning and configuration of all sorts of infrastructure. Application servers, web servers, load balancers, proxies and database servers are among the lengthy (and no doubt growing) list of “boxes” devops needs to get up and running to support just about any given application today. One of the key value propositions of a devops approach to operations is that it can reduce the time it takes to get applications to market by getting them up and running in production faster. That’s increasingly important as we move into the Era of Things, driven by extreme connectivity of everything, because all those “things” need are going to need to be talking to applications on the back side.

So speed is of the essence, but not at the cost of accuracy. We’re encouraged, then, to automate tasks and orchestrate processes to achieve that speed. The side effect of codification of tasks is higher accuracy (aka fewer mistakes).

So far so good.

But you may notice that I have (and will continue to) clearly delineate between “automate” and “orchestrate”, between “task” and “process”.

The reason for that is that, simply, they are not the same thing. We should strive not to confuse these two because while the two share some benefits, there are other benefits that are achievable only by one or the other – or the combination thereof.

Process optimization, for example, can’t be achieved by simple automation. Automation is concerned with a single task – launching a web server, configuring a web server, stopping a service. Orchestration, however, is concerned with automating, if you will, the execution of a workflow – of a process. A provisioning process may be comprised of multiple tasks and involve multiple systems. An “application” is not just a single server, after all, it’s likely several servers – web, app and database in a traditional three-tier architecture.

The goal of orchestration is not just to automatically execute a service, which brings speed to the deployment process and gets applications into production faster. It also affords an opportunity to streamline – to optimize – those processes for even greater gains in deployment velocity.

One of the simplest ways to optimize a process is to eliminate repetitive steps. This entry on Vagrant explains it fairly well :

“We could just SSH in and install a webserver and be on our way, but then every person who used Vagrant would have to do the same thing. Instead, Vagrant has built-in support for automated provisioning. Using this feature, Vagrant will automatically install software when you vagrant up so that the guest machine can be repeatably created and ready-to-use.” [emphasis mine]

— Vagrant Docs “Getting Started | Provisioning”

Now certainly the step of installing a web server on a new box with vagrant is hardly all that time consuming, it’s just a single line of text. But every line of text must be manually typed, and that introduces the possibility of fat fingering the command, which either consumes time in fixing it or introduces an error into the system. Neither are desirable results.

Ensuring that the process is repeatable and orchestrated by automating all the tasks that comprise that process can optimize it by eliminating redundancy. That results in greater deployment velocity.

So while you’re “doing devops”, remember to evaluate the processes and consider whether or not they’re repetitive, and how you might be able to eliminate that repetition through more modern architectures or toolsets.

May 20, 2014
Doing Devops: URL Management
A fairly common task for operations folks is managing URLs. URL management either uses redirection or rewrites to manage things like vanity URLs or marketing campaign URLs or even just getting users to current URLs from old, deprecated ones. The business purpose behind the need to manage URLs aside, the process for doing so can become quite unwieldy and time consuming.

That’s because traditionally URL redirection and rewriting is a layer 7 proxy or web server function that requires modification of a configuration file. For example, here’s a simple URL rewriting configuration excerpt for Apache:
```
01.RewriteEngine On
```
```
02.RewriteBase /books
```
```
03.# If there's a cookie called thgtp-pre-version set,
```
```
04.# use its value and serve the page
```
```
05.RewriteCond %{HTTP_COOKIE} thgtp-pre-version=([^;])
```
```
06.RewriteRule ^guide-to-node$ %{REQUEST_FILENAME}-%1.html [L]
```
The key here is that it’s in the configuration file. This is just one rewrite rule. Imagine needing hundreds of these rules and then imagine them being changed on a weekly or even daily basis. Not only is this likely not the job you signed up for, it’s inevitable that at some point you’re going to fat finger a rule change and you know what that means. Yup. Apache no start. And you get to figure out where it is.

In some architectures, rewrite/redirect rules are simply made to be part of the web server configuration. Each and every web server has to have its configuration file updated when a single rule changes. More modern approaches take advantage of proxies, which greatly minimizes the number of instances of Apache (or whatever proxy you’re using) that need to be updated.

Regardless, it’s a pretty mundane and boring thing to do, and honestly, the whole process of having marketing (or whoever) kick off a request for a new rewrite/redirect rule and having the ticket make its way to your desk probably takes more time than it does to enter the rule and type restart httpd.

So how about we leverage some programmability in the network to automate this bad boy and, if you want, hand off the tedious management of the URLs to the people who have ownership of them (business and marketing folks)?

Programmable Proxies

I’ve explained before about programmable components, particularly those in the data path, and one of those components can be described as simply a “programmable proxy.” Proxies are not unfamiliar to dev or ops, so I’m going to assume we’re good on that front. The programmable piece comes in when you basically move all the logic that’s encoded in the configuration file (like rewrites and redirects) into a programmatic environment instead. So rather than configure the behavior of the proxy, you’re going to code the behavior of the proxy instead.

So how does that help, you might ask? Well, for starters it makes the phrase “infrastructure as code” ring a heck of a lot more true because, well, it really is code. That means it becomes an artifact like any other piece of code and gets managed like code. That means simpler rollbacks and version control.

Second, it means if you’ve got a robust enough environment you can extend the logic to include data sources like a database (SQL or NoSQL, your choice) that manage a mapping of old URL to new URL. And if it’s in a database, it’s a pretty simple thing to have someone put a pretty HTML form in front of it that enables business folks to manage that list.

The actual execution is straightforward: the proxy receives the request for one URL, say www.example.com/TVad and queries the database, gets back the real URL, say www.example.com/marketing/tv/2014 . It’s not that much different than a configuration-based approach, with the exception that you’re able to dynamically determine the mapping rather than hard-code it in a file.

Now, what the proxy does with it is up to you. You can send a redirect to the client with the “real” URL, or (a better choice if you care at all about performance) rewrite it and send it on to the right server.

A programmable proxy should be a key tool in every devops practitioner’s back pocket. With both network and application fluency, a programmable proxy provides the foundation for a variety of architectural patterns that rely on dynamic evaluation of application-layer requests in real time. The programmable proxy is well-suited for these tasks because it can be inserted into the data path nearly transparently, and doesn’t require special networking (or perhaps more importantly, the cooperation of the networking teams).
May 12, 2014
What Cisco’s OpFlex means for DevOps

Some big news in networking land at Interop this year revolved around Cisco’s latest contribution to open source and the Internet, its proposed “OpFlex” control plane protocol.

The protocol itself is focused on communicating with network elements and specifies encoding in a variety of formats, including JSON, that are developer-friendly. That’s nice, of course, but the big question as someone interested in operations and devOps is why should you care about what appears to be a networking protocol? After all, beauty is only interface deep. What’s in it for you?

To see where DevOps fits into the picture, consider the description of OpFlex as specified in the IETF draft of the OpFlex protocol:

The OpFlex architecture provides a distributed control system based on a declarative policy information model. The policies are defined at a logically centralized policy repository (PR) and enforced within a set of distributed policy elements (PE). The PR communicates with the subordinate PEs using the OpFlex Control protocol. This protocol allows for bidirectional communication of policy, events, statistics, and faults. This document defines the OpFlex Control Protocol. [emphasis mine]

What’s being offered is not just a protocol for bidirectional communication, but an entire architecture for sharing policies and actionable data. What’s more important to those who for whom “networking” is not their native technical tongue is that policies are designed to be declarative and abstract. That means instead of issuing a set of commands, often specific to the device or service, a policy declares (describes) what’s needed and each component participating in the architecture interprets those needs according to their specific function and capabilities.

You might be thinking at this point (I know I certainly was) “What is this policy and who is defining it?” because that seems pretty central to the value proposition of this whole endeavor.

For that, we need to look to the OpenDaylight (ODL) project which has created a new project called the ODL Group Policy plug-in. As explained by Cisco in an OpFlex white paper, “the goal of this project is to provide a policy-based API that can serve, in practice, as a standard information model in OpFlex implementations.”

Jumping over to the ODL Group Policy Plug In wiki, then, the intention of the Group Policy Plugin is to introduce “the notion of groups of endpoints and policy abstractions that govern communication between these groups. The goal is to support a northbound API that accepts abstract policy based on application requirements while offering numerous concrete southbound implementations for programming network element with configuration/programming stanzas rendered from application policies.”

Okay, so that’s a whole lot of .. words. What it boils down to is that instead of a policy describing a set of rules governing routing with the appropriate ACLs, one might simply declare that “Web Server A can talk to Database Server A to perform database transactions” while “Administrative Console B can talk to Database Server A about administrative tasks.” If you were to translate that into specifics you would likely have multiple imperative command lines, each describing VLAN participation and routing as well as entries allowing traffic between specific VLANs or IP addresses on the ports appropriate to the application layer protocols. It would be long and prone to misconfiguration simply due to the number of commands required. Using a declarative (abstract) policy language simplifies not only the policy but also addresses the disparity inherent in competing products across a given market by eliminating it from being problematic.

The problem with traditional proposals to solve the “different definitions and object models” across a given market space (like ADCs or IPSs or switches) is that they’ve all relied on finding the lowest common model, which inevitably ended up with all offerings needing to dumb down (commoditize) their value in order to ensure consistent support for all “XYZs”. By abstracting to the policy – and not the object model – level, the OpFlex approach effectively eliminates this issue and emphasizes defining what is desired by the application, but not how.

Now, the actual policy declaration is unlikely to take the form of natural language (that’d be nice, but we’re not quite ready for that… yet, though Watson might disagree with me on that) but the general premise is to provide a far more, shall we say, friendly policy language.

What this means is the operations folks most likely to be familiar with the application architecture and who understand what it needs and how it needs to work, could be the ones to actually codify that into policies. You’ll note that the intention for OpenDaylight, at least, is to “support a northbound API that accepts abstract policy based on application requirements.” That means someone – hint, hint, that’s you – might be able to integrate into the application deployment lifecycle provisioning of the network services and topological constraints required based on an understanding of the complex web of connections between each piece of an application. Someone who knows from experience that a “three tier architecture” means more than just three components, and can more effectively describe how they should be interacting.

But that’s just the beginning. You’ll note that in the RFC summary above it specifies “bidirectional communication of policy, events, statistics, and faults.” That means actionable data is being exchanged between participating components. Actionable data enables, well, action. Actions that might address a fault that occurs between end-points. Actions that might address performance issues. Actions that might mitigate attacks in real-time.

Ultimately this level of capability will drive new solutions – new workflows and processes – that take advantage of event-based systems and policy-driven networking. OpFlex should enable statements like “When X happens, do Y” to be executed upon, even if it requires coordination of three or four or more network components.

What OpFlex means for the devops practitioner is a way to influence (or, one hopes, specify directly) what services the network provide for the applications you’re tasked with deploying and delivering, and how the network responds to the things that happen on any network that impact the performance, security or stability of those applications.

As networking continues to recognize and address the need to provide application-driven services, including provisioning and real-time adaptation, it’s going to continue to become more programmable and integrated with the systems driving a more automated, orchestrated data center. And that means devops practitioners are going to have more and more options as to how they manage the entire application (and network) service lifecycle.

April 23, 2014
Programmability in the Network: Stop a Bleeding Heart…

It is not often the case that a security vulnerability can get the entire Internet talking. And not just the security community on the Internet, but everyone. End-users and IT alike are looking for answers and trying to mitigate Heartbleed. It has its own web site and logo. It’s that big of a deal.

Many service providers have already patched their systems, but there are a whole lot of sites on the Internet and it’s estimated that a significant number of them are vulnerable.

Netcraft notes that Heartbleed, based on OpenSSL, “affects around 17% of SSL web servers which use certificates issued by trusted certificate authorities.” One of the most trusted sources of data regarding web server software in use today, Netcraft’s “most recent SSL Survey found that the heartbeat extension was enabled on 17.5% of SSL sites, accounting for around half a million certificates issued by trusted certificate authorities.”

But it’s not just server software that may be vulnerable. Any device in the data path that terminates SSL and is based on OpenSSL could be vulnerable. Any “thing” that might use OpenSSL to enable secure communication and accepts connections could be vulnerable. In other words, hearts are bleeding all over the Internet right now, and it could take quite a while for the Internet in general to plug the holes.

The hole itself is fairly simple – it takes advantage of the mishandling of what seems like an innocuous meta-protocol that runs atop TLS and DTLS that consists of two message types: HeartbeatRequest and HeartbeatResponse. It’s used to “ping” the other party in a secure connection to make sure the peer is alive. It can be used to offset idle times that might otherwise cause the connection to time out.

It can also, thanks to a missing bounds check on the message structure, arbitrarily and completely silently grab 64 kilobytes of the server’s memory. That memory might include sensitive things like the private key used to encrypt communications or passwords or pieces of critical documents.

Whatever might be in memory when the vulnerable software goes to grab it is leaving the building faster than Elvis – and without a trace. There’s no logs, no evidence of the theft, nothing. It’s clean and remotely executable by anyone on the Internet. That may explain the attention it’s currently getting.

The answer to any vulnerability is ultimately to patch it. Depending on how many servers or systems are impacted, that could take some organizations days – assuming patches are available immediately. In the meantime, the heart is still bleeding and the only way to prevent more information from leaking is to shut off the hose.

Except it’s not the only way.

Programmability in the data path can mitigate this, now. Right now. Like in the next five minutes.

Let’s review programmability in the network for a brief moment: you have, in the data path, a device. That device (or software) is capable of programmatically intercepting, inspecting, and transforming the data that’s traversing it. That means such a system can intercept a request to initiate SSL/TLS traffic, inspect it and, if it finds a request to include Heartbeat support it can reject the connection.

Interestingly, one of the primary motivators of early SDN architectures was the ability to support new or experimental protocols by inserting them into the data path in the network. While that’s a driver for academics, a more realistic driver for businesses is the ability to interact with protocols in flight and “fix” them – especially when the “fix” involves preventing the exploitation of a vulnerability.

Programmability in the network enables this kind of rapid response to threats. Being able to deploy a solution very quickly to prevent further data leakage provides protection for both the organization and those who interact with its properties (employees, partners and customers). It also offers time for IT to patch and update all impacted systems without remaining vulnerable or taking the drastic step of going completely dark until the situation is resolved.

By enabling a view of “infrastructure as code”, programmability not only enables a more software-oriented approach to deploying services, but also a software-oriented approach to quickly creating and deploying an appropriate response to emerging threats.

A devops approach can facilitate this capability, and push protection into the network as code.

April 10, 2014
Automation versus Orchestration

Yes, Virginia, there is a difference.

One of the things devops practitioners are tasked with is the provisioning and configuration of all sorts of infrastructure. Application servers, web servers, load balancers, proxies and database servers are among the lengthy (and no doubt growing) list of “boxes” devops needs to get up and running to support just about any given application today. One of the key value propositions of a devops approach to operations is that it can reduce the time it takes to get applications to market by getting them up and running in production faster. That’s increasingly important as we move into the Era of Things, driven by extreme connectivity of everything, because all those “things” need are going to need to be talking to applications on the back side.

So speed is of the essence, but not at the cost of accuracy. We’re encouraged, then, to automate tasks and orchestrate processes to achieve that speed. The side effect of codification of tasks is higher accuracy (aka fewer mistakes).

So far so good.

But you may notice that I have (and will continue to) clearly delineate between “automate” and “orchestrate”, between “task” and “process”.

The reason for that is that, simply, they are not the same thing. We should strive not to confuse these two because while the two share some benefits, there are other benefits that are achievable only by one or the other – or the combination thereof.

Process optimization, for example, can’t be achieved by simple automation. Automation is concerned with a single task – launching a web server, configuring a web server, stopping a service. Orchestration, however, is concerned with automating, if you will, the execution of a workflow – of a process. A provisioning process may be comprised of multiple tasks and involve multiple systems. An “application” is not just a single server, after all, it’s likely several servers – web, app and database in a traditional three-tier architecture.

The goal of orchestration is not just to automatically execute a service, which brings speed to the deployment process and gets applications into production faster. It also affords an opportunity to streamline – to optimize – those processes for even greater gains in deployment velocity.

One of the simplest ways to optimize a process is to eliminate repetitive steps. This entry on Vagrant explains it fairly well :

“We could just SSH in and install a webserver and be on our way, but then every person who used Vagrant would have to do the same thing. Instead, Vagrant has built-in support for automated provisioning. Using this feature, Vagrant will automatically install software when you vagrant up so that the guest machine can be repeatably created and ready-to-use.” [emphasis mine]

— Vagrant Docs “Getting Started | Provisioning”

Now certainly the step of installing a web server on a new box with vagrant is hardly all that time consuming, it’s just a single line of text. But every line of text must be manually typed, and that introduces the possibility of fat fingering the command, which either consumes time in fixing it or introduces an error into the system. Neither are desirable results.

Ensuring that the process is repeatable and orchestrated by automating all the tasks that comprise that process an optimize it by eliminating redundancy. That results in greater deployment velocity.

Recognizing that processes should be comprised of composable tasks offers a higher degree of reuse across operations. The task of provisioning and configuration a web server with the appropriate base software means consistency, aka standardization, across all applications. Processes that take advantage of standardization result in greater infrastructure stability, another benefit often cited by organizations that have adopted a devops approach to operations.

The key here is to recognize the difference between automation and orchestration and how they relate to one another. Automation is about codifying tasks and orchestration is about codifying processes. Orchestration takes advantage of automation by reusing these basic building blocks.

There is a difference, and done right, the two play off one another to result in streamlined, more accurate execution of operational processes that speed up successful application deployments.

April 10, 2014
Don’t Let the (April Fools Day) Joke Be on You

As much as seasonal increases in demand for on-line resources are discussed, the one day that’s often overlooked is April Fools Day.

It’s the day the Internet loses its mind and your best bet is take everything not just with a grain of salt, but with the entire shaker.

The use of April Fools’ Day to promote brand awareness and bring some laughter into the world as everyone tries to outdo everyone else in a bid to be named the King of Fools is well known. This is no more a surprise today than is Black Friday, Cyber Monday or Super Bowl Sunday.

So it behooves those participating in the tom foolery to be prepared. That means being ready to handle an increased demand for whatever it is you’re offering up for consumption. This is where devops shines; where automation and orchestration and a demand-driven scalability strategy is your best answer. Because you can’t be sure your offering is going to go viral, but you can be sure it won’t if any piece of the application delivery chain breaks under the pressure.

In other words, this is unacceptable:

What might have gone viral may end up languishing as just another overlooked and forgotten attempt because of a lack of database resources and a failed scaling strategy.

One of the key aspects of devops is that it should approach scalability with a comprehensive view of how each piece of the delivery chain needs to scale – and when. Thresholds and metrics collected from each domain of an application architecture must be correlated and understood in the context of how it impacts capacity so that it can be automatically scaled – or at least addressed – on demand.

Maybe that means devops must advise a cloud-based approach to handling demand for such days. Maybe that means a bit more preparation and integration across the entire application delivery chain to ensure the load balancing service is basing its decisions on the right variables such that resources are maximized and enable greater scale across the application. Maybe that means throwing up a cache to reduce the pressure on web and application servers, or taking advantage of a CDN.

Regardless of the approach to how you’re going to scale, devops practitioners should be driving the process to ensure that on April Fools Day – or any well-known high demand generation day – the joke is not on them.

April 1, 2014
What “Programmable” Means

One of the most often used words these days to describe emerging architectures and technology is (drum roll please) “programmable.”

Now, from my used-to-be-a-developer-and-architect-turned-infrastructure-geek perspective, I know what that means. It means I get to write code. But to what purpose that code is put to is a question that’s rarely answered. So today we’re going to answer it because, well, my used-to-be-a-developer-and-architect-turned-infrastructure-geek persona likes to talk about code.

When you hear the term programmable used in the context of Software-Defined Networks (SDN) they’re talking about the same things that “programmable” means in the context of Devops: programmatic control of both the data and the control path.

The easiest way to visualize the difference between these impressive sounding terms is to recognize that the data path is inside and the control path is outside.

What that means is if you’re interacting with the data path, you are participating in it. You can inspect it, evaluate it, and transform it and, perhaps more importantly, you can make decisions based on data in flight. That’s powerful for the network, because a variety of Devops patterns are based on this very concept and require the network for implementation. Consider, for example, the implementation of an A/B testing pattern. The basic premise of the pattern is that somehow the identity of a user is known and, based on their membership in a defined group (beta testing customers, roles within the organization, volunteers from the community, etc…) they are routed to different versions of the same application to determine how differences in the UI impact behavior.

Notice the term routing, which implies it is a function of the network. The reason programmability is critical is two-fold: one, you must be able to identify the user as belonging to a specific group and two, you must be able to dynamically route based on that information. The former, identification, could ostensibly be accomplished using fixed lists IFF (If and Only If) it is possible to map a user to some variable readily accessible to the network, which traditionally means things like IP and MAC addresses and TCP ports. Most folks will immediately recognize that today this isn’t enough (and actually it hasn’t been enough since the days of the mega-proxy, but I digress). The information that’s needed is much higher in the networking stack at the application layer, aka in the OSI model as Layer 7. Which means you’re probably going to have to dig into HTTP headers and maybe the payload to get the data you want. That requires programmability in the data path, inside.

To really make the pattern scale (and to make it flexible enough to support a phased approach to beta testing) you probably don’t want group membership to be a static, fixed thing. So now you’re going to want to extend your implementation to include making a call to a database or identity store (LDAP, AD, etc…) to get current information. That requires programmability in the data path, inside.

So what programmability in the data path means is the ability to use code to direct the behavior of some “thing” in the network, be it an application proxy or a load balancer or what have you. You can’t easily do this through configuration-based proxies because they don’t allow for the execution of the domain logic necessary. There are a limited number of interactions you can achieve with configuration-based systems because of the logic they’re hard-coded to execute,and most of them don’t include calling out to a data store to look-up arbitrary information.

Data path programmability = Code in Flight. So you now you know where the name of the blog came from. This developer-and-architect-turned-infrastructure-geek finds the intersection between programmability and networking irresistible.

Okay, so that’s data path programmability. What about control path programmability?

Well, that’s the easy part. Control path programmability is on the outside. It’s the APIs that enable automation and orchestration of provisioning and configuration. Control path programmability is how you integrate infrastructure into command control systems like OpenStack, Puppet and Chef. Infrastructure comes with varying degrees of API-enablement, some extremely granular and others less so, depending on the particular function and maturity of the vendor.

So what you can do with the control path depends on the API, but in general it enables things like provisioning new services – think configuring a new virtual server in Apache and all the settings that accompany it – or start up and shut down functions that enable rolling maintenance across a set of devices or servers to avoid downtime.

But all control path programmability is on the outside, enabling integration and management without having to learn a CLI or configuration file syntax.

SDN and Devops use both programmability to reach relatively the same goals: scale of operations, rapid provisioning of services, to decrease the possibility of human error (the fat finger factor) and to introduce flexibility into systems that were previously bound to fixed processing and configuration-based paradigms.

Programmability in the network is going to enable a variety of new patterns, processes and services and as traditional infrastructure becomes more and more programmable through the adoption of architectural approaches like SDN, Devops is going to see more and more opportunity to apply its maturing expertise in leveraging programmability to a much wider audience.

March 24, 2014
PEBKAC Avoidance

We’ve all said it. We’ve all done it. We’ve all shook our heads at it. PEBKAC. Problem exists between keyboard and chair.

User error.

While generally applied to the end-user community – those folks who are considered technical neophytes by IT professionals – it can and should also be applied to those of us who have, at least once (admit it, come on, I know you’ve done it) fat fingered a configuration on a web server, a switch, a router, or some other network or application service.

It’s okay. We’ve all been there – head down on the keyboard, a litany of words we wouldn’t use in front of our mothers streaming from our lips between enumerations of how long we’ve sat at our desk looking for the problem. Mine was a misconfiguration of route metrics in the now long gone Network Computing lab that sent traffic from one side of the lab over a simulated T1 and back over the 100 Mbps link. When you’re trying to simulate a WAN environment and put WAN optimization solutions on either end, this is highly problematic and tends to, well, not work. Not work at all. If it wasn’t for the astute observation of my lab partner, I’d probably still be at my desk hunting for the problem.

The point is, we’ve all had our PEBKAC moments. We know they happen, and avoiding them is a very real goal of devops in general. We want consistent, repeatable and successful deployment processes in part to avoid the time and effort it takes to troubleshoot some minute configuration error in the lengthy and often complicated data path over which application traffic must traverse in our domain.

And it’s no longer just about downtime, which is costly and frustrating for everyone involved. It’s also about security; security of the network, the applications, and data.

Tufin Technologies, in a 2010 survey of 100 registered DEF CON 18 attendees, found that “73 percent of hackers came across a misconfigured network more than three quarters of the time – which, according to 76 percent of the sample, was the easiest IT resource to exploit.” [1]

With more and more layers of complexity added to the network in an attempt to simplify the network (yes, how’s THAT for irony?), there are more and more opportunities to experience the unique pain of PEBKAC every time a new service is provisioned. Consider the math:

A typical data center has over 500 servers, each running a hypervisor. Each hypervisor carries an average of 20 virtual workloads. Each workload requires at least 5 separate network attributes to be configured. That’s a total of 250,000 opportunities to make a mistake. At even an error rate of less than 1 percent, that’s still 250 problems you’ll need to track down.

This is, in part, why Devops exists. To enable the development and use of reliable processes and methods for provisioning services. That means configuration is happening under the covers, unless you’re already living in rainbow unicorn land where services configure themselves automatically. I didn’t think so.

There’s a reason why typists are measured based not just on speed but on the number of errors made while typing. It’s not just about the speed, it’s about the accuracy because eventually, you’re going to miss the mistake you made and it’s going to be sent out in a corporate wide e-mail and … well, you can peruse the Internet at your leisure to see how well those turn out. The same is true for deployments and provisioning. It’s not just about how fast you can get a service up and running, it’s about how fast you can get that service up and running correctly.

Devops should focus as much on accuracy as it does speed. You’ll save more time avoiding misconfiguration and the associated long hours of troubleshooting by simply slowing down a few MPH (or WPM) and ensuring that you’ve got it right the first time.

[1] http://www.continuitycentral.com/news05331.html, http://www.darkreading.com/perimeter/misconfigured-networks-are-easiest-prey/227200159

March 19, 2014
The Fourth Seat at the IT Service Velocity Table

Agile. Devops. SDN.

Development. Operations. Network.

Notice anything missing? How about security? On the one hand, that could be because we’ve finally managed to realize that security is an integral part of every aspect of IT and it’s by default embraced by the other three groups already.

I’ll pause while you stop laughing and catch your breath.

On the other hand, it could be because it’s like every other technological movement and advancement and we’ve simply plowed ahead without considering security until, as is always the case, someone remembers that security is important too. If development, operations and the network teams are going to have an answer to improving the speed with which applications get from the IDE to the end-user’s iPhone, then shouldn’t security have a seat at that table, too?

Absolutely.

It can be argued, in fact, that some areas of security already have a seat at the service velocity table. Consider automated pen testing, virtual patching and continuous vulnerability assessment services. These are often continuous processes that integrate (sometimes) with existing systems or, at a minimum, with some process through which aberrations can be noted and addressed.

It’s important to note that this – and devops – isn’t a plea for automation. Automation and orchestration are tools in a toolbox that may or may not be a fit for a particular tasks or process. Most enterprise organizations, for example, refuse to automate changes to data center firewall rules. Not because it can’t be done – it can – but because the ramifications of a change going wrong is too great to take even the small risk that might be incurred.

But in the interests of speeding up the deployment process (in which security is – or should be – a significant player) there are likely adjustments that can be made to align security with other groups already adopting more agile methodologies.

For example, consider that along with agile development often comes a “nightly build” of the software, designed to ensure that changes don’t break the complete package. It might be beneficial for application security-focused assessments to be inserted into that process, to test early and often the security profile of the software being developed. This helps in two ways: first, it ensures that developers can be made aware of potential vulnerabilities while they’re actively developing the software. That’s important because one of the precepts of agile is move quickly and by the time you’ve found a vulnerability in the released software the developers are already half way through the next spring and not thinking about what they just tossed over the wall.

Second, if there are vulnerabilities that turn up that developers can’t, for some reason, address you have a head start in figuring out how to mitigate them. If that requires a security service of some kind, you’ve got time to get the right policies into place before the application hits the Internet and becomes a target.

There are plenty of ways in which agile methodologies can be adopted by security professionals. One of the precepts of devops is getting outside the silos that exist within the organization and communicating with the other folks seated at the “get the application to the user” table. Being more involved with development and operations and even the network will enable security to integrate more fluidly with the increasingly agile processes that drive the application lifecycle.

March 11, 2014