devopsy.kinsta.cloud

Author: Lori MacVittie

API Discovery and Search with APIs.JSON

If that sounds vaguely like UDDI (which dealt with discovery and search of WSDL then you’re on the right track.

Glue is this week, and preceding the conference was the 2015 API Strategy and Practice Tech Un-Workshops. One of the lightning talks (and they were lightning fast, let me tell you) was all about APIs.JSON – a relatively new collaborative effort between Kin Lane, API Evangelist, and Steven Willmott and Nicolas Grenie of 3scale networks,

The effort – which you can dig into at apisjson.org – is an attempt to provide a more formal and standardized definition of APIs designed for “public deployment and consumption by automated software agents (robots)”. In a nutshell, automated agents (and people, too, if you like reading machine-consumable metadata) can access a meta description of the APIs available for an organization at a known (standardized) URI such as http://company.com/apis.json.

It’s not quite a WSDL for JSON, as the metadata is not describing the actual interfaces, but more like UDDI entry that might have once (and maybe still is) hosted in a registry (repository). The current incarnation provides basic information about the APIs and includes basic properties like a name and descriptions and URLs and can also contain more (some might say usable) information such as links to documentation or an actual, real-life readable API definition such as Swagger or API Blueprint formats as well as WADL and WSDL.

The effort is showcased in an open API search engine (http://apis.io) that allows individuals to seek out APIs (over 900 are listed today) based on a variety of criteria, all of which of course is derived from the metadata within an APIs.JSON file.

What does this all have to do with DevOps? Well, as Steve Willmott astutely pointed out in his lightning talk, APIs.JSON is not just about describing APIs in a consumable format, but it’s about being able to have machines discover and bootstrap APIs. That means potentially being able to trigger and signal changes about an API to kickstart a new build and deployment process.

This is a great example of the possibilities for collaboration between Dev and Ops when everything is treated like code and automation moves from simple tasks to workflows – to orchestration.

And I probably don’t have to mention (but I will, because , me) that the same trigger and signaling process could (and probably will) hook into broader network infrastructure (the components that exhibit high application affinity like load balancing and caching and app security) to drive more automated change management.

APIs.JSON is not the first (and won’t be the last) effort of this kind because the mobile and app economy of the future is being built on APIs and thus , as Steve Willmott stated in his talk, “the Web of APIs is the future of the web.”

May 20, 2015
Enterprise DevOps: Standardize for Security

Etsy is a shining example of the success that can be achieved with continuous delivery (CD). By 2014 Etsy had doubled its deployment rate, deploying more than 50 times a day. Today the site deploys API changes in 18 seconds and launches a new website every 150 seconds.

That’s speed, to be sure. Speed that very few enterprises need or want to achieve. But that doesn’t mean that Etsy and CD don’t have valuable lessons for the enterprise, because one of the reasons CD and Etsy succeeds is because of the underlying principle of standardization.

Whether it’s standardization of tools or of processes – or both – it is standardization that eventually drives such speed. The use of the same tools and the same processes over and over hones them until they’re an optimized, lean delivery machine. That standardization has other, benefits, too that are as beneficial – maybe moreso – to the enterprise than the speed it eventually brings.

Standardization of delivery processes in the enterprise, especially when extended to such concerns as security and the network, result in direct and indirect benefits. Directly, standardization allows security and even network infrastructure to ensure that corporate policies are implemented. If all infrastructure – application and network alike – should only be hosting certain services (like HTTP and SSH) and not others, standardization can help enforce that. Template-based (or policy-based, if you prefer) automation along with an “infrastructure as code” approach can assist in ensuring that basic security and network policies are put into place for every application deployed. Customization can (and should) be allowed and encouraged as extensions to the core templates (or policies) deployed during the app deployment process.

Indirectly, standardization on a common toolset and processes will reduce the stress on network and security operations that comes from trying to implement common policies across a conga line of solutions. There’s no getting around the reality that many enterprises employ sometimes twenty or more “middle boxes” that provide everything from security to monitoring to performance and scalability services. Each one must be provisioned and configured during the app deployment process. This is not a trivial problem. 26% of organizations cited “security policies that are too complex and can’t be enforced with the current network security processes and controls” as a significant challenge in a 2014 ESG survey. Establishing a standardized template that encapsulates common (required) policy implementations can go along way toward addressing the challenges network security professionals face when trying to secure applications across a growing portfolio of both applications and network services.

This level of standardization is not easily achieved. There is no easy button, and that’s one of the reasons DevOps can be a desirable approach in the enterprise to addressing these kinds of problems. It isn’t just about achieving phenomenal speed, but rather achieving consistency through the standardization of processes and methods used to deploy security policies and controls that can improve the stability and the security of the entire application and network environment.

DevOps is, in relation to network and security operations, more about CPR for infrastructure services than it is speed. Consistency through standardization. Predictability through automation. Repeatability through templates and by treating infrastructure as code.

DevOps is an approach to operations and inter-operations that can have just as many benefits in the enterprise as it does for web-scale and startups born in the cloud. The same standardization that benefits Etsy by giving it extreme speed can benefit enterprises by endowing it with extreme security.

April 23, 2015
What IT expects from SDN is DevOps

There was much excitement in the air when the announcement that Docker had acquired SocketPlane. Not the least of which was emanating from my corner of the world, where DevOps for network (the whole network) is something you’ve noticed I pretty much live and breathe.

The announcement was couched as an SDN (Software-Defined Networking) play by containerization-leading Docker. Which might be confusing at first because, well, Docker (and to be more general, containers) are generally associated with DevOps, not SDN.

Ah, but therein lies the rub, doesn’t it? While the original precepts of SDN were most definitely not DevOpsy in nature, it (like every other trendy term in tech) has evolved to focus as much (if not more) on what are certainly DevOps-like principles and goals than on technical specifications and standards.

Consider, for example, this recent Avaya sponsored, Dynamic Markets report (gated) on “SDN Expectations.”

The first data point to examine is the ranking of “problems” experienced by respondents. The number one issue (38%) was downtime caused by human error. The second (tying with a network-failover issue at 37%) was complexity of configuring services and applications across the network.

If these aren’t typically problems that are said to be resolved by DevOps – particularly with attention paid to collaboration and automation – I’ll eat my hat. And yes, I often wear hats, it’s a quirk I have.

And just as interesting was this table illustrating not just problem areas, but how many respondents had the problem and how many expected SDN to solve them:

If you ignore the network management tuple, the top two problems are definitely often associated with DevOps as issues that can be resolved with collaboration and automation and some measurement thrown in to boot.

SDN has evolved, at this point, to focusing more on network automation than it does on automated networks. The difference is actually profound as it relates to DevOps, as the former is really about management, about configuration and provisioning and automation of processes designed to extend CD across the entire application deployment lifecycle into the production environment. It’s about codifying and encapsulating configurations in templates and centralizing state of the network in order to engender better agility in both moving forward and falling back when things go awry.

SDN today really is (or should be) a subset of DevOps, focusing on network infrastructure operations rather than app infrastructure operations. But the same principles and goals are very much aligned between DevOps and SDN, between the network and the application infrastructure. That’s a critical convergence starting to appear that’s necessary, as when you dig into IT and its problems delivering apps on time these days, it’s mostly the network that’s still in the way.

The network is identified often as a key IT pain point and usually the pain is related to the provisioning of services and applications across the network. High costs of networking is also cited, and that is largely due to the complexity of the network and the amount of operational sweat and blood that goes into manually provisioning, configuring and testing the various pieces of network infrastructure required to deploy the services business needs to deliver applications.

SDN may continue to be referred to as SDN, if only to emphasis its focus on network infrastructure, but the reality is that it’s DevOps. DevOps for Networks.

Which brings us right back to why a DevOps-related, containerization company like Docker would swallow up an “SDN”-focused start-up like SocketPlane.

Because what IT expects from SDN is really DevOps. And if DevOps is ultimately about culturally change and breaking down silos, the folks who focus on the network can’t be (and shouldn’t be) isolated from the folks managing the application infrastructure.

Docker’s move brings software-defined networks together with software-defined operations under (what should be) an IT encompassing concept: DevOps.

April 2, 2015
The right answer was “process”
The question is “why is there no one-size-fits-all automation and orchestration solution”?

Automation, a component critical for continuous deployment particularly as it extends beyond app operations and into the broader environment to encompass network and security operations, too, continues to be a custom build kind of thing. There’s no single automation tool, framework or pattern that can be generally applied to a significant number of environments. Mi casa es no su casa, after all. That’s because environments don’t actually look like power point slides; your environment is not the simplistically diagrammed firewall –> load balancer –> web server typically found in decks extolling the virtue of one solution over another. They are far more complex, comprising not just multiple tiers of interdependent and interconnected apps and services, but multiple chains of infrastructure services spanning the gamut of security, availability, mobility, performance and access control. The specific flow through and between these services is unique. Indeed, the intimate relationship between network latency and app performance is well known, and thus the navigation through the critical yet complex site of services required to deliver an application could easily be considered a competitive advantage if it’s able to keep performance under the 5 second toleration limit on response time apparently now the standard for consumer satisfaction.

So it’s no surprise that automation and orchestration, responsible for automatically setting up the triggers and routes required for application traffic to correctly traverse the right set of services for optimal performance, would not be a simple, turn-key solution. One size does fit all. It must, like its business process focused counterpart – BPM – be customizable and easily changeable.

BPM? Are you seriously?

I’m dead seriously. BPM – business process management – has been around far longer than automation and orchestration of IT systems and its focus is, as its name implies, on process. Automation is included, but it is the orchestration of business processes that often change in the face of variable business conditions that makes it an invaluable tool for optimizing business and a source of competitive advantage. “The ability to change quickly is essential. Businesses change their key processes 4-7 times per year. The driver for change can be internal or external. New opportunities can arise. New partners or customers need you to support a different way of doing business. Federal or international regulations can require you to change your processes.” (Making the Case for BPM, Jim Rudden)

Consider for a moment this data from a 2012 Bearing Point survey on the goals and impact on culture:

Top BPM goals are increased efficiency (93%), increased transparency (90%) and the realization of standardization potentials (85%). A number of BPM goals have an impact on corporate culture, e.g. increased transparency (rank 2), increased customer orientation (rank 5), disclosure of weaknesses/ bottlenecks (rank 6) and the reduction of mistakes/ error rates (rank 9).

– Bearing Point Business Process Management Survey 2012

If many of these goals and impacts sound familiar to you, they should. Because what DevOps ultimately needs to focus on is exactly the same as the focus of BPM – process. In an application world, these processes require applications and services. IT must be able to support these process changes in the application and service infrastructure.

BPM (Business Process Management) has known this for years, and it is the orchestration of the process – the workflow – from task to task, from approver to doer that generates the efficiency gains achieved with that technology.

Indeed, adopters of DevOps who focus on the “big” picture would do well to consider the similarities to BPM which touts as benefits such things as:

Process quality improvements

Continuous process improvement

Reduces costs

Improves business agility

It would be difficult if presented with that list of benefits with no attribution to determine whether it was referring to DevOps or BPM. That’s because the operationalization of business through process orchestration is not all that different from the operationalization of IT through process orchestration.

And processes are not, nor should they be, turn key and one-size fits all. They should be tailored to address the needs of a specific operational or business goal, i.e. deploy an application or secure an order from a customer, based on the systems in use to achieve those goals. BPM shines where it is most able to address bottlenecks and obstacles caused by handoffs between systems and teams. DevOps will shine where it is most able to address bottlenecks and obstacles caused by handoffs between systems and teams. It isn’t about racing through a given task with automation. Speed isn’t efficiency and, if you do the math, a focus on automation without similar focus on process can lead to even greater inefficiency.

And that’s why there is no one size fits all automation and orchestration tooling; because the goal isn’t automating tasks, it’s orchestrating process. Coordinating workflows. Ferreting out inefficiencies.

Orchestration is the glue that binds automated tasks together. For example, recently a missed route update caused some unexpected down time during what was considered a routine* decommissioning of gear. A missed route update. In other words, someone forgot. That task might have been automated, it might have just needed to be kicked off. But it wasn’t.

That’s the kind of thing that would likely not happen if the process was governed by some form of orchestration; if there was a clear list of what needed to be updated or changed based on the removal of that particular component. If orchestration was involved to provide visibility and oversight of the operational processes needed to perform such a routine task. This does not mean the tasks should have been automated; in fact automation is not strictly required to achieve the visibility and oversight necessary because processes are about workflows and hand offs, not specific tasks completed during the steps that comprise a process.

Which leads us right back to Business Process Management. Take a deeper dive into the aforementioned Bearing Point survey and you’ll find more similarities with DevOps:

Crucial elements for sustainable BPM are clearly defined roles (95%), top management support (92%) and a clear mandate (92%)

Typical implementation barriers such as the lack of resources (53%), sponsorship (32%) and experience (25%) need to be identified early and addressed in a structured way

BPM as a continuous (=sustainable) approach is already adopted by 59% of companies, coordinated and implemented either centrally or locally

Involvement of stakeholders (90%), process oriented thinking (90%), organizational institutionalization (82%) and a well-communicated concept (75%) are elements of communication and change management which must accompany the establishment of sustainable BPM

That’s because both are ultimately about process, with automation being only one of the tools in the tool chest that support implementation of a continuous approach to enabling agility in the process layer. Indeed, about 38% of the companies in the survey use software to support their BPM initiatives while another 37% do not. A spreadsheet with the right Six Sigma calculations can be as powerful as any automation framework, toolset or API in identifying the dead space between tasks that introduce inefficiencies and delay application deployments.

BPM has, like DevOps, brought tangible, measurable results to the table. Bearing Point found that almost three quarters of the surveyed companies (72%) were able to achieve measurable benefits through BPM such as process lead time (72%), process cost (68%) and error rate improvements (67%).

So rather than focusing too much on automation, adopters and proponents of DevOps would do well to up-level the conversation and start looking at process orchestration and improvement as the goal. The successes BPM has had on the business side of the equation tell us that doing so will provide a better path to achieving success with DevOps in the operational side of the equation.

* I’d argue that in today’s environment of hyper-sensitivity to availability that nothing that touches a service or gear in the critical path of an application is “routine” but, hey, it is what it is.
March 12, 2015
How Deploy Frequency Impacts Infrastructure Stability
I recently took some time out from reading the Internet to read a very specific piece of research I found on the Internet from IDC – its “DevOps and the Cost of Downtime” to be exact. Sponsored by AppDynamics, this fascinating read included a variety of statistics that are helpful in understanding the impact of (or a lack of) DevOps on organizations given certain financial and competitive impacts.

I won’t bore you by repeating what you can so easily read yourself, but I have pulled out a few key stats that are worthy of further investigation – that of the failure of stability on the business and the predicted increase in deployment frequency.

The average hourly cost of an infrastructure failure is $100,000.

The average hourly cost of a critical application failure is $500,000 – $1M.

The average number of deployments per month is expected to double in 2 years.

Interestingly, that prediction on deployment frequency is a good thing and carries with it the possibility of making sure the costs associated with the other two rarely (if ever) happen.

We’ve already gotten to a point where “speed” with respect to DevOps is effectively equivalent to “time to market”. But there’s another measure of “speed” that’s just as critical, particularly when we apply it to maintaining stability in the infrastructure. That measure is MTTR (Mean Time to Resolution). This measure is very important when you consider that 60% of production failures are caused by human error or lack of automation according to research from ZeroTurnaround. A goal of DevOps, then, should be to reduce that number as much as possible and introduce more stability into the environment. Frequency of deployment can actually be of assistance in realizing that goal.

So how does frequency of deployments help do that?

First let us consider the number of possible points of failure that might cause downtime – both in the application and core network infrastructure. Then let us consider that each of those possible points of failure also have multiple potential causes of failure. A single service in the logical critical data path within the core service infrastructure, for example, can easily cause “downtime” that, in the case of IDC, would be characterized as “infrastructure failure” and run a tab of $100K per hour.

If that application is a critical business application, the tab runs much higher per hour.

Obviously the speed with which we restore stability (service) in such an event is a factor in the overall impact to the business.

This is where DevOps comes in and where operationalization can play a significant role in mitigating the impact of downtime by aiding in faster time to resolution.

Consider that one of the most difficult tasks in the network is not provisioning or configuration, but troubleshooting. To be fair, troubleshooting in app dev, too, is one of the more time-consuming and frustrating tasks, consuming 75% of a developer’s time (according to research from InitialState). That’s close to the 65% of network engineers who are dedicated to “break and fix” tasks, who spend their time troubleshooting network issues up and down and all along the network stack. Needless to say, if you just deployed ten or fifteen different configuration changes splayed across the core and per-app service infrastructure and suddenly something is “wrong”, it’s going to take some time to figure out exactly which configuration change caused the problem in the first place.

The relationship between number of changes and frequency of deployment is no secret; if you’re deploying changes more frequently then it’s likely there are fewer of them. With an average sprint cycle of 3-4 months, the amount of change is logically not going to exceed that of a cycle that takes 12-18 months. The same holds true for the infrastructure; more frequent deployments should mean fewer changes).

Logic (and math) tells us that the fewer lines of code and configuration that have changed, the less time it should take to identify the culprit should something go wrong, thereby hastening the time to resolution. Assuming that an agile, frequent deploy rate results in fewer changes to the environment it stands to reason that this approach should also reduce both the number of failures and the time to resolve any that do crop up. Frequency is a significant factor in reducing failures by increasing its overall stability.

This is backed up anecdotally by Puppet Labs findings in its “State of DevOps” in which organizations adopting a DevOps approach reported 50% fewer failures and a 12x improvement in time to resolution. That’s stability at work and it’s driven in part by the frequency of deployments and its positive impact on MTTR.

So let’s apply this approach (and logic) to the rest of the services along the critical data path in the data center infrastructure, i.e. apply it to application services like caching, web application security, load balancing, identity management and application access control. Automation to reduce misconfiguration caused by human error and smaller, more frequent changes that enable a faster time to identity the inevitable issues that will be introduced. No methodology or approach is going to completely eliminate errors, after all. There’s a limit to how perfect we can get and it’s pretty much a Zeno’s Paradox kind of problem. While we keep solving current issues, more config and code is being deployed that is statistically likely to contain yet another error, and so and so forth.

But the goal is not necessarily to eliminate all errors (though that’d be nice, it’s simply not practical and it’s the primary reason why our goals for uptime are called “5 9s” and not a “1 with two zeros”) it’s to reduce the number of errors and their impact on downtime. And that is something we can achieve without running afoul of Zeno and his paradox.

Smaller, more frequent deployments can result in a more stable environment in which downtime is reduced to a level consistent with acceptable risk.

DevOps, whether applied directly to infrastructure or not, will play a role in improving its stability which in turn has a direct and very real impact on the business by eliminating (some) downtime thanks to more frequent deployments.
February 26, 2015
Measure Twice. Automate Once.

“Shipping” code may start with the completion of development but it doesn’t end until after the operational delivery process ends. Just like any fulfillment process, shipping is actually just the kick off of an often arduous journey that ultimately delivers the app to the end user.

Ship It

Back in the old days (when they still let me write code) one of our favorite phrases was “ship it” to indicate completeness of a task. That’s because the ultimate goal was, of course, to “ship” the code off to the users so they could start doing whatever it was they were going to do with it*.

But the reality was, of course, even when we were really ready to “ship it” to users that it didn’t actually get to users for varying amounts of time. Like shipping anything today, it depends on the shipment method. In terms of today’s enterprise IT, “shipping method” is really a metaphor for the operational approach to deploying all the infrastructure – the “middle boxes” – with the services required to actually deliver a secure, fast and available application to its ultimate users.

We know from various studies that this takes time. From days to weeks to even months. And we know we don’t like that and we want to improve it. Which is where DevOps comes in, or NetOps or SDN or SDDC or SDO or whatever you want to call it.

Speeding up the deployment process is certainly one of the goals of operationalizing the network, but without a baseline how do we know if we’ve gotten faster in the first place?

That’s a piece of the DevOps puzzle we’ve been ignoring – the need to measure the deployment process. You might notice that “measure” is bolded and in red, that a visual clue meant to reinforce the importance of the measurement aspect of DevOps. If you aren’t measuring how long it takes to deliver that app from dev to its destination – production use by an end-user – then you have no idea which pieces of the process are holding you back. You can’t effectively determine where automation might be applied to improve the time it takes to get to market. You don’t know exactly where the obstacles are that need to be addressed in the first place.

While we can make some assumptions based on long experience and research and point to handoffs across the IT silos, the reality is that there probably isn’t just one handoff between ops and the network and there may be multiple intra-IT group handoffs. For example, the network ops that handles provisioning network services might not be the same team that handles load balancing and performance services. But they’re both within the “network” silo. There may be multiple handoffs back and forth as various services also need networking services on top of the ones required by the app itself.

Image Credit: Gaping Void

Without some kind of measurement of how long each of the various steps is taking it’s impossible to determine exactly where the bottlenecks in the app deployment process are in your organization. I can not only determine where that Minecraft Lego set I ordered for my son’s birthday is right now, I can see how long it’s taken to get from each point in the process. I know it goes from point A to point B in X days, and from point B to point C in Y days, and I know if there’s a winter storm that it’s going to cause a slowdown in that second step (which is called a movement in the logistics vernacular, in case you were wondering. I knew you were, you’re welcome!)

Measure Twice. Automate Once.

The hidden value in setting out to measure the deployment process is in pulling together an understanding of the process itself. It’s in finding redundancies and discovering sub-processes that are running in serial but could be running in parallel to improve them. It’s not necessarily about automating the whole thing up front, it’s about understanding what the process is and making it more efficient – whether through elimination, reorganization or automation (or all three).

You don’t simply want to be automating things willy nilly without any idea whether it’s going to do any good. As my dad taught me about working with wood, you measure twice and cut once. With DevOps you want to measure twice (at least) and automate once. Otherwise you may be wasting cycles on one-off tasks or tasks so specific to one application that automation has no real value because it’s not the same across applications.

The key is to start with defining what you’re going to measure, do the measurement and then analyze it to determine what you need to improve.

Because automating just for automation’s sake isn’t any more fruitful in the end than adopting technology for technology’s sake.

* I say “whatever” because over the course of my career I’ve worked on tax software, GIS software, communications and transportation and logistics systems. There’s a lot of variance there in what end users might be doing with “the app”.

February 2, 2015
DevOps needs infrastructure multi-tenancy

DevOps will, under whatever moniker you want to give it – NetOps, DevOps for Networks, operationalization – continue to make inroads into the network infrastructure because ultimately it’s part of the application deployment lifecycle. While DevOps folks call it “application delivery” the reality is that an application is not ready to deliver to a user (internal or external) until all its requisite services have been provisioned and configured.

Yes, infrastructure and network devices are increasingly API-enabled and supportive of a variety of tools and frameworks most often associated with DevOps – Puppet, Chef, OpenStack, VMware – and those more commonly associated with just the network – Arista, Cisco ACI, and OpenDaylight.

But it takes more than APIs. APIs do not inherently bestow upon devices the ability to support multi-tenancy. That is, the isolation of services unique to a team (or application) required to effectively deploy application-supporting infrastructure services within what is traditionally a shared environment.

According to ThoughtWorks Technology Radar (jan 2015) partitioning infrastructure along team bounds is becoming an imperative to removing roadblocks in the application deployment lifecycle:

Many of our customers have made DevOps a reality in their organization with delivery teams that build, deploy, and support their own applications and services. Unfortunately, a regular roadblock on that journey is allowing teams to have superuser privileges in production environments. In most organizations, the production environment is shared, and therefore risky to provide access widely. It is effective when we can partition infrastructure along team bounds,so that those teams can have safe isolated access to do their work, without risking impact to other systems. Where cloud environments are used, this is much easier to implement, aligning account structures to team boundaries. [emphasis mine]

This has always been an issue where shared services are relied upon. A change to support one application (or team) can be disruptive or cause issues for another application or team. Thus, most infrastructure and network teams staunchly refuse to let anyone in the application or operations side of the house play with their toys.

They are the network ninjas; guarding their configurations with their very lives.

It’s this kind of conflict that needs resolution if we – as a unified IT organization – reach goals of improving the time to market and frequency of deployment. It doesn’t matter if the app team can deliver an app in 3 weeks if it takes 3 months to deliver it to the customer because of overloaded network and infrastructure staff.

There are two good solutions to this problem*.

1. The governance over more application-affine infrastructure services – load balancing, caching, proxies, etc… – is necessarily moving closer to the application and under the control of app and operations teams. This is supported by concepts like Network Service Virtualization, proposed in Lippis Report 217 as well as the increasing adoption of open source proxy-based (virtual and software) services. This model assumes continued shared network and infrastructure services as appropriate while migrating per-application services to the more agile, operational architecture.

2. The introduction of multi-tenancy in infrastructure architecture provides a similar approach while still maintaining shared resources (hardware). A single, high capacity system is able to host multiple, virtual instances of its services, each dedicated to either teams or applications, as is the organization’s need. This approach spreads the cost of the hardware resources across the entire organization while allowing a more per-application approach to provisioning, configuration and ultimately, cost. Isolation addresses the concerns of shared resources and configurations that can cause consternation among network and infrastructure teams because the underlying shared system is managed by them while role-based control over application services is offered to teams or designated responsible individuals.

Both approaches are valid and in many cases both architectural solutions will be used to resolve the issues caused by traditionally shared infrastructure.

The key takeaway here is that there are answers to the need for partitioned infrastructure that are supportive of DevOps-driven workflows and processes and can certainly provide a path toward ensuring a more seamless, time-sensitive end-to-end application deployment process.

* Oh, I’m sure there’s more, but right now these appear to be the best two options.

January 22, 2015
Awkward questions for those getting started with automation
This blog asking awkward questions for folks jumping on the microservices bandwagon was enlightening from the perspective that many of the same questions are applicable to operations as they begin to tackle operationalizing all the things in an application lifecycle.

For example: “What is your deployable unit?”

Great question, isn’t it? What is your deployable unit of automation? Is it per service? Per device? Per cluster of devices? Per container?

Wait, but there’s more from this great list of awkward questions:

Is it acceptable for another team to take your code and spin up another instance of your microservice?

Can team A use team B’s microservice or are they only used within rather than between teams?

Do you have consumer contacts for your microservices or is it the consumer’s responsibility to keep up with the changes to your API?

Is each microservice a snowflake or are there common conventions?

How are these conventions enforced?

How are these conventions documented?

What’s involved in supporting these conventions?

Are there common libraries that help with supporting these conventions?

All these questions are applicable to devops, in particular, as you go about building out automated scripts, systems and orchestrating processes that operationalize the data center and make it a lean, green deployment machine.

You are going to have some kind of code, whether it’s a script or a recipe or a template of some kind, and questions regarding the usage of that code is important. Can others use it? Change it? Claim it as their own? Are they useful in a general, “here’s a basic load balancing provisioning and configuration script” or are they specific, as in “this is a piece of code that provisions and configures the networking required to ensure all 18 tiers of this very specific application can communicate”?

And as you’re developing those scripts (that code) are you documenting it? Are there <gasp> comments? Do you use a standard convention for code formatting (cause let me tell you, where you place your brackets is a matter of great technical religious significance)?

Most of those jumping into automation head first aren’t thinking of the long term ramifications of the choices they are making right now. Proof of concepts turn into pilots that turn into systems upon which all of IT relies. Rarely are you given the opportunity to go back and put any kind of standard conventions or practices into place. That’s what builds technical debt in software and, no matter what you think you’re building with those scripts, you’re building software.

Adopting the technical and methodological practices of agile development is a good thing, but let’s not ignore that along the way from waterfall to agile developers have learned a lot of lessons the hard way. That means we should, as practitioners of software-related activities within operations (and that means any one of the four operations groups in IT), pay attention to those lessons and try to avoid the pitfalls that developers have already encountered and, one hopes, overcome.

To do that, don’t just ask the awkward questions; answer them.
January 7, 2015
You Might Already be Doing DevOps in the Software-Defined Data Center
I was out reading the Internet and came across a report from EMA (Enterprise Management Associates) regarding the state of the Software-Defined Data Center (SDDC) in 2014. After perusing the data and highlights, I was surprised (and yet should not have been) to find so much DevOps-related content and concepts within. Neither should I have been surprised to find significant mention of software-defined networking (SDN). It makes sense, in a puzzle-fitting-together kind of way, that SDDC comprises software-defined operations and software-defined networks (and probably software-defined storage but it’s too early in the day for me to contemplate that).

In any case, a significant percentage of organizations take advantage (today) of software-defined operations and networking, based on the report:

Which of the following technologies, structures or processes does your organization take advantage of?

4. Cross-functional processes to orchestrate provisioning and management of storage, network and server resources (57%)
5. Solution for server administrators to provision their own storage volumes (56%)
6. Continuous/centralized capacity management for physical, virtual and cloud resources (55%)
9. Policy-based automation of infrastructure for routine adjustments without human intervention (50%)
10. Software Defined Networking (virtual overlay network) (48%)
11. Software Defined Networking (separated control plane from delivery plane) (46%)
13. APIs for developers to provision their own app environments (servers, network and storage) (42%)
14. Configuration management solutions (Puppet, Opscode Chef, SaltStack, ServiceMesh, etc.) (41%)

More telling, perhaps, of how critical DevOps is to enabling the SDDC is the emphasis put on not just automation but on process. When asked what respondents believed were the “most important aspects of the Software Defined Data Center”, number 2 and 3 focused on deployment and automation of applications and infrastructure:

Best-practice, repeatable configurations of software and infrastructure for workload deployment (46%)

Orchestration and automation to easily deploy applications across silos (44%)

In case you weren’t entirely convinced, yet, “operational analytics” (we might call it “measurement” or “metrics” here in DevOps land) came in number four with a respectable 42% of respondents.

There’s more in the report, some less relevant but for the most part, reading this SDDC report left me with a decidedly DevOps after taste. Which was kind of a minty chocolate coffee taste, in case you were wondering.

The concepts typically associated with DevOps are not necessarily peculiar to DevOps or SDN. They’re generally part of a larger trend to operationalize all the things: the network, the storage, the compute, the security. The business goals of SDDC are strikingly similar to those promoted by DevOps and SDN – reducing risk, lowering opex, improving stability and time to market. The collaboration through creation of interfaces designed to promote repeatable, consistent (and measurable) handoffs between groups is present and looms as large in SDDC as it does in DevOps.

The point is that DevOps may, in fact, be in the adoption process right now under the guise of another name in an organization near you – or around you. If you hear someone saying software-defined something, check it out. You may be surprised to find out that someone has snuck in DevOps through another door, under a different name.
December 10, 2014
Infrastructure Automation and API Taxes

That automation is one of the critical components of DevOps is apparent in numerous surveys and studies that focus on the benefits organizations have realized from implementing the approach across operational groups. Automation itself starts with (or should start with) a comprehensive API. That means infrastructure – network and application focused – needs to expose via an API the myriad configuration options it requires as well as optional features it offers.

It is through these APIs, after all, that integration between devices and systems and across operational groups occurs. The much vaunted SDN or cloud “ecosystem” is really a euphemism for pre-packaged and validated integration. Integration that occurs via an API.

These same APIs are available to everyone, of course, and lay a foundation to operationalize app deployments by automating the various provisioning and configuration tasks required to move an app from dev to production.

APIs allow for fine-grained automation and orchestration, with each and every knob, button and slider option available to be flipped, switched or turned. The more features and functions possible, the more buttons and knobs and sliders there are to manipulate, figuratively speaking.

Each and every one of those calls to do so needs to be carefully managed. Exceptions happen, errors occur, and any script or tool ought to carefully validate that each and every one of those calls has been successful. Which is going to take a lot of code. Code that itself will introduce the possibility of errors and require maintenance and reviews and all that goes along with good development practices. Code that will also impact the time required to execute. Given that most APIs are RESTful and require invocation via standard HTTP exchanges, each and every API call incurs time to send, time to execute, and time to receive a response. The overhead is basically an API tax; for every API call you invoke, you’re paying with time… and more time.

Martin Fowler touches on this in his post “Microservices and the First Law of Distributed Objects“:

The consequence of this difference is that your guidelines for APIs are different. In process calls can be fine-grained, if you want 100 product prices and availabilities, you can happily make 100 calls to your product price function and another 100 for the availabilities. But if that function is a remote call, you’re usually better off to batch all that into a single call that asks for all 100 prices and availabilities in one go.

Thus for sufficiently complex systems another option is needed: templates. Templates do not replace APIs, but rather provide an additional and far more compact method of accomplishing the same tasks automatically, but without the overhead imposed by highly granular sets of API calls. Templates encapsulate a configuration; codifying the settings for a variety of knobs, buttons and sliders into a single configuration artifact. In a fine-grained API-base integration, each of those settings must be maintained somewhere external to the automation script or system. Each API call will include one (or maybe two) of those variables as read from some file or system that maintains the application-specific set of data. This introduces many potential points of failure. Whether file or database, variables must be retrieved before being associated with an API call and then transmitted to the receiving device. Errors may occur on read, on format, or on transmission.

In contrast, if all the variables exist as a single, validated configuration template artifact, it can then be sent, as a single entity, via an API call to the device or service in question. The number of potential points of failure is effectively reduced to read/retrieve and on transmission. This greatly simplifies the process as well as reduces the time it takes to actually execute the configuration of the target device or service.

In addition to simplifying integration by reducing the provisioning and configuration tasks from potentially hundreds of API calls down to a small set of calls, this has the benefit of fitting much easier into an infrastructure as code strategy, where common configurations are treated as artifacts and stored in repositories much like its application counterparts.

There is an inflection point at which it becomes advantageous to use a template-based approach over individual API calls. That point occurs before the scripts driving the API-based automation become as complex as an application and thus bring with it associated issues like error rates, time spent troubleshooting and maintainability.

A template-based approach simplifies upstream workloads and integration as well as discrete automation scripts by reducing the set of variables required to just a few – usually those dependent on network-specific values such as IP addresses and network segmentation membership (tags or ids representing VLAN, VXLAN, NVGRE, etc…).

This further ensures a consistency (and in turn results in greater stability of the underlying infrastructure) in app deployments.

APIs are a good thing. They’re a key enabler of software-defined architectures like SDDC, cloud and SDN. But API-enabling infrastructure doesn’t necessarily mean only on a checkbox and radio-button basis. That can be valuable but it can also lead to integration efforts that are just as complex (or more so) than their manual counterparts. A template or policy-based (application-driven) approach coupled with an API through which to deliver and execute such constructs results in a much cleaner, more consistent and stable means of integrating provisioning processes into the greater software-defined architecture.

November 20, 2014