devopsy.kinsta.cloud

Category: Enterprise DevOps

What Cisco’s OpFlex means for DevOps

Some big news in networking land at Interop this year revolved around Cisco’s latest contribution to open source and the Internet, its proposed “OpFlex” control plane protocol.

The protocol itself is focused on communicating with network elements and specifies encoding in a variety of formats, including JSON, that are developer-friendly. That’s nice, of course, but the big question as someone interested in operations and devOps is why should you care about what appears to be a networking protocol? After all, beauty is only interface deep. What’s in it for you?

To see where DevOps fits into the picture, consider the description of OpFlex as specified in the IETF draft of the OpFlex protocol:

The OpFlex architecture provides a distributed control system based on a declarative policy information model. The policies are defined at a logically centralized policy repository (PR) and enforced within a set of distributed policy elements (PE). The PR communicates with the subordinate PEs using the OpFlex Control protocol. This protocol allows for bidirectional communication of policy, events, statistics, and faults. This document defines the OpFlex Control Protocol. [emphasis mine]

What’s being offered is not just a protocol for bidirectional communication, but an entire architecture for sharing policies and actionable data. What’s more important to those who for whom “networking” is not their native technical tongue is that policies are designed to be declarative and abstract. That means instead of issuing a set of commands, often specific to the device or service, a policy declares (describes) what’s needed and each component participating in the architecture interprets those needs according to their specific function and capabilities.

You might be thinking at this point (I know I certainly was) “What is this policy and who is defining it?” because that seems pretty central to the value proposition of this whole endeavor.

For that, we need to look to the OpenDaylight (ODL) project which has created a new project called the ODL Group Policy plug-in. As explained by Cisco in an OpFlex white paper, “the goal of this project is to provide a policy-based API that can serve, in practice, as a standard information model in OpFlex implementations.”

Jumping over to the ODL Group Policy Plug In wiki, then, the intention of the Group Policy Plugin is to introduce “the notion of groups of endpoints and policy abstractions that govern communication between these groups. The goal is to support a northbound API that accepts abstract policy based on application requirements while offering numerous concrete southbound implementations for programming network element with configuration/programming stanzas rendered from application policies.”

Okay, so that’s a whole lot of .. words. What it boils down to is that instead of a policy describing a set of rules governing routing with the appropriate ACLs, one might simply declare that “Web Server A can talk to Database Server A to perform database transactions” while “Administrative Console B can talk to Database Server A about administrative tasks.” If you were to translate that into specifics you would likely have multiple imperative command lines, each describing VLAN participation and routing as well as entries allowing traffic between specific VLANs or IP addresses on the ports appropriate to the application layer protocols. It would be long and prone to misconfiguration simply due to the number of commands required. Using a declarative (abstract) policy language simplifies not only the policy but also addresses the disparity inherent in competing products across a given market by eliminating it from being problematic.

The problem with traditional proposals to solve the “different definitions and object models” across a given market space (like ADCs or IPSs or switches) is that they’ve all relied on finding the lowest common model, which inevitably ended up with all offerings needing to dumb down (commoditize) their value in order to ensure consistent support for all “XYZs”. By abstracting to the policy – and not the object model – level, the OpFlex approach effectively eliminates this issue and emphasizes defining what is desired by the application, but not how.

Now, the actual policy declaration is unlikely to take the form of natural language (that’d be nice, but we’re not quite ready for that… yet, though Watson might disagree with me on that) but the general premise is to provide a far more, shall we say, friendly policy language.

What this means is the operations folks most likely to be familiar with the application architecture and who understand what it needs and how it needs to work, could be the ones to actually codify that into policies. You’ll note that the intention for OpenDaylight, at least, is to “support a northbound API that accepts abstract policy based on application requirements.” That means someone – hint, hint, that’s you – might be able to integrate into the application deployment lifecycle provisioning of the network services and topological constraints required based on an understanding of the complex web of connections between each piece of an application. Someone who knows from experience that a “three tier architecture” means more than just three components, and can more effectively describe how they should be interacting.

But that’s just the beginning. You’ll note that in the RFC summary above it specifies “bidirectional communication of policy, events, statistics, and faults.” That means actionable data is being exchanged between participating components. Actionable data enables, well, action. Actions that might address a fault that occurs between end-points. Actions that might address performance issues. Actions that might mitigate attacks in real-time.

Ultimately this level of capability will drive new solutions – new workflows and processes – that take advantage of event-based systems and policy-driven networking. OpFlex should enable statements like “When X happens, do Y” to be executed upon, even if it requires coordination of three or four or more network components.

What OpFlex means for the devops practitioner is a way to influence (or, one hopes, specify directly) what services the network provide for the applications you’re tasked with deploying and delivering, and how the network responds to the things that happen on any network that impact the performance, security or stability of those applications.

As networking continues to recognize and address the need to provide application-driven services, including provisioning and real-time adaptation, it’s going to continue to become more programmable and integrated with the systems driving a more automated, orchestrated data center. And that means devops practitioners are going to have more and more options as to how they manage the entire application (and network) service lifecycle.

April 23, 2014
Advanced Automation – Getting Your Systems to Work for You

In my previous post I discussed how to take your DevOps to the next level by taking it beyond infrastructure automation, to the automation of your deployments and code pushes, through patches and updates.

And then I promised to make it interesting…so here is the next stage – actually using the extracted data to get your systems to work for you.

Monitoring Like IT’S YOUR BUSINESS

Let’s start by discussing what it means to monitor your application, and what kind of data you can extract from it.

At the most basic level, you want to know whether your application is available for your users. It sounds very basic and simple, but setting it up properly is actually not an easy task. It is intended to give you an answer to the most important question to your business: can my users use the application? Think of it as a big red or green light on your dashboard. Although the answer may seem obvious and simple, but it is not at all. There are all kinds of parameters to take into account when answering this question, for example, what if the system is apparently up and running but the response time is very slow, or what if only parts of the system are actually running. At the end of day, you want to have that kind of an alert “traffic light” that tells you whether your system is functioning – yes/no, up/down.

System, Application, and Business Metrics

The green-red traffic light is an aggregation of multiple levels of monitoring. The most basic one is system related. It contains indicators like process availability, CPU and memory utilization, etc.

The next level of monitoring is application related and is composed of application level KPIs (key performance indicators). KPIs can be anything from the average response time for a user for a certain request, to the number of concurrent database connections. In other words, anything that’s specific to the application and its architecture.

And then above these, comes the highest level of metrics, which are business metrics. Ultimately these are what’s really important. If your business metrics are ok, then you can assume everything is running fine. Sample business metrics include the rate of failure to register to your website, how many users did a certain operation in the website or how many users executed a specific transaction. If we want to be really specific, then take Google hangout for example, a business metric of theirs would be how many people in a certain timeframe joined or started a hangout.

Logs to the Rescue

Logs are often used here as means to collect your metrics and alert about erroneous conditions. Logs usually contain a lot of useful information, but you need to be able to extract it and make sense of it. This means gathering all of the logs emitted from all of your servers, parsing them and looking for specific patterns that will help you generate the KPIs over time. For example, you look for a log message that says “User X signed in”, and by counting these messages you can produce a business level metric for how many users register over time. If you add some more data to the log message, such as the user’s location, or operating system, or anything else for that matter, you can slice and dice data and get deeper insights into how your application is behaving. When logs are emitted in a structured and consistent way (e.g. in a JSON format), it becomes very easy to analyze them and produce the relevant KPIs. There are many toolsthat can help with that.

From Simple Automation to Orchestration

That’s where orchestration comes into the picture. At the most basic level, orchestration is a higher form of automation, which helps you setup all the pieces that are related to your application, starting from the infrastructure (VMs, networks, block storage volumes, security groups, etc.), to the platforms your app runs on (database, web server, etc.), and all the way up to the application modules and code. This entire setup is often referred to as a topology. The role of an orchestration framework is to materialize a certain topology. More advanced orchestrators go beyond materializing the topology, and change it to meet the current workloads and needs of the application.

As you probably gathered by now, monitoring and log gathering are an essential part of any running app, so they should be an integral part of the orchestrated application topology. Since the orchestration process is topology aware, it can wire and configure monitoring for your application components very easily, which is one of its greatest benefits. Going through these process without a global view of the topology can often be time consuming and error prone. Moreover, as the topology changes , you need to reconfigure and rewire your monitoring tools, and a good orchestrator will that for you as well.

Next Up: Reactive and Proactive Orchestration, The Devops Holy Grail

Hopefully by now you see the value of orchestration as a higher form of automation. In the next post I’ll dive deeper into the post-setup phase, and discuss how orchestration tools can react to events and monitoring data and adjust the application’s runtime topology to best fit the current workloads.

April 23, 2014
DevOps adoption – Enterprises (part 2 of 2)

A large enterprise has institutionalized an incredible number of processes. Their fundamental approach to IT has been built over years if not decades of experience. Their leaders may have grown up with these methodologies and approaches.

In this article I wanted to continue the thread I started in my last article about how different organizations adopt DevOps differently. The focus of this article will be on how enterprises adopt DevOps. I will also contrast it with how startups adopt DevOps, which was the focus of the first part of this article.

To recap the first part of the article, we discussed how startups are adopting DevOps. We stated that adoption occurs organically because of the nature of startups. Because they only start with a few people and they have little to no process, DevOps starts through the addition of tools. The founding developers or CTO will find tools to solve the practical problem of automating different aspects of their development, QA, or operations functions.

With so few resources, automation is critical. As a startup grows they add more people to the team and process becomes more critical. So, they extend Agile into other areas of their product development process. They become more intentional with developing a DevOps oriented team.

As the company continues to grow, DevOps is embedded into the culture. Hiring decisions are made based on whether a candidate understands how to be a part of a team practicing DevOps. The rest of the company starts to beat to the drum of rapid iteration, cross-functional integration, and increased automation. DevOps adoption in startups follows the tools, process, and then culture sequence.

Enterprises follow a different path, and so usually take a contrasting trail to DevOps adoption for a variety of reasons. A large enterprise has institutionalized an incredible number of processes. Their fundamental approach to IT has been built over years if not decades of experience. Their leaders may have grown up with these methodologies and approaches. Further, their entire business is beating to a certain IT drumbeat. Changing the beat throws everybody in the organization off.

There are also other major issues to DevOps adoption within an enterprise. Institutional resistance and inertia is a significant one. Change is always difficult to people. Some groups handle change better than others, but in any organization – especially a large enterprise – change needs to be orchestrated and worked effectively. Change in large organizations takes time and perhaps nothing takes longer than bringing people on board with a new approach. In addition to the personnel aspect, the organization has built strong, existing processes. Those have been over many years. Changing complex processes doesn’t happen overnight. Those processes have been built with internal and external customers in mind. Those customers expect their “solutions” delivered in a specific manner and timeframe. Accelerating that though DevOps may be great, but not if they aren’t on board and can accept the increased pace.

DevOps impacts the entire system of an organization. The system has been tuned to a certain speed and level of quality. Increasing the front end of the process will not necessarily increase overall throughput unless the rest of the organization can increase their speed in concert. In a startup, that system is generally small and contained. Most of the individuals, systems, and processes are contained and when the pace increases, the rest of the organization can react quickly or better yet the entire organization has been built with those expectations in place. Not so for a large enterprise.

The startup approach of adopting DevOps through adopting tools, changing processes, and embedding into the culture, will be easily rejected by a large enterprise. Even if the IT organization can deliver much faster and does, the rest of the organization won’t know what to do with the completely different approach. Perhaps the rest of the organization will even continue operating at their pace while IT operates at a different one. While not catastrophic, it is not helpful either. As larger organizations think about the best way to adopt DevOps understanding this perspective aids in adoption. Enterprises have a much harder uphill climb when it comes to adoption, however, the rewards can also be monumental.

An enterprise’s journey into DevOps – if it is to be successful across the entire organization – starts at the cultural level. And, as we all know, cultural change in a large organization is perhaps the hardest step. Clearly the positive here is that the organization adopting DevOps will have an opportunity to perform far more effectively. Cultural change starts with the key stakeholders at the senior level creating a commitment to adopt this new methodology. Then that commitment needs to be evangelized through the organization. A top down edict doesn’t work just as a grassroots movement without senior level support doesn’t. It should be reasonably straight forward to bring those two groups together to build a shared commitment to moving to DevOps (or not). Note that large organizations can adopt and test DevOps in one of their divisions or smaller groups to understand how different the approach will be from their previous business methodology.

Once a shared vision is reached, some of the detail work can begin. As discussed earlier, a different pace of product development needs to be matched by a different pace of distribution, sales, and marketing too. Updating and changing processes is the step that makes this happen. How will development processes change? And, how do those changes impact existing customers both internal and external? Can the same approach to quality be taken or do new processes in development require new processes in QA? Security? Operations? All of these downstream impacts will need to be matched, reviewed, and changed. Changing processes that are upwards of decades old is also fraught with risk. Many of those steps exist for a reason. Altering them may have consequences that aren’t known to all. This is yet another reason why DevOps adoption doesn’t happen overnight at large enterprises.

Finally, as processes are updated and the cultural shift is largely underway, the IT organization can turn its attention to finding better tools. These tools need to match the processes built and, of course, the cultural style of the organization. This is why large organizations would be better served by deeply understanding their newfound processes before diving into tool selection or even purchases. Only after they understand their needs is it worthwhile to embark on deep tool selection.

As we said in the last article as well, we’ve shown this to be a serial process, and, of course, it is never so cut and dry, especially, in a large organization. There will be overlap, there will be things that happen out of sequence, there even may be areas that jump far ahead, but largely, the adoption approach takes on the character of culture first, process second, and tools third. DevOps can be transformational for a large organization with some even going so far as to say that a DevOps culture is a predictor of stock performance. Understanding how DevOps takes shape at a large organization will help those in other large organizations as well as the vendors and consultants supporting their efforts.

April 21, 2014
DevOps adoption can succeed where ITIL failed, if we let it
When I look at the future of DevOps adoption and acceleration I contrast it with what I feel was the failure of ITIL to really move beyond large enterprise IT shops. I am of the firm belief that DevOps has the legs to succeed and soar where ITIL sort of crashed and burned. OK if not crashed and burned, at least sputtered out. To understand why I feel this way, let me share a little personal history with you.

ITIL demands you to come to IT

I remember my first ITIL project. I was working at a large retailer and I was asked to help our IT department understand what this ITIL thing was and to help drive adoption of some of ITIL’s basic concepts. First on the list was to get a service catalog up and running. In the space of just a few weeks, I realized the main barriers to adoption and helped our IT executive team understand that getting IT consumers to understand, much less embrace an IT-centric taxonomy such as ITIL was not going to be an easy task.

This scenario points directly at one of the big reasons ITIL failed to achieve critical mass (i.e., adoption of the model outside of large enterprise IT) and also reveals why DevOps has a much better chance of success of achieving critical mass (as long as we ourselves don’t mess it up).

Mind you I’m not saying that ITIL is dead or that it has nothing to teach IT shops. I am saying that ITIL alone has not really changed much of anything inside of IT shops, let alone improved the reputation of IT with it’s consumers and customers.

When you contrast ITIL and DevOps several things really set the two approaches apart:
- ITIL makes it easy to blame the business for not talking IT’s language while DevOps continuously seeks to meet people on whatever terms they are capable of communicating in. DevOps inspires, while ITIL deflates
- DevOps is centered around improving the lives of IT workers everywhere while also improving bottom lines at the companies they work for. ITIL is centered around improving stability in the most efficient manner possible. I don’t know about you, but I just don’t see a room full of people, much less an entire organization getting excited or inspired by the concept of efficient stability. The only people likely to get sustainably excited by “efficient stability” are the accountants who can now start to push cost (i.e., people) out the door.
- ITIL at its core is reductive (or at least allows for reductive methods and approaches to grow unchecked). For those uninitiated in philosophy or how language influences our ideas and behavior, reductionism is when people essentially believe that a complex system is nothing more than the sum of its parts. This reductive approach at the core of the ITIL movement is what reinforced the division between Dev and Ops – because the concept of team chemistry has no place in ITIL given that the organization is nothing more than a collection of faceless workers who play out a prescribed role.
Reductive goals and processes, essentially reduce people and their efforts to numbers. The act of using quantitative targets as goals in large initiatives is a dangerous practice in general and grows more dangerous in larger groups and organizations – case in point the recent GM ignition failures that resulted in 13 deaths in an effort to save .57 cents per car affected by a defect.

Before you get excited, I’m not proposing that ITIL will kill people. I am stating that allowing an organization to be led by reductive philosophy allows a whole bunch of unintended consequences to come into play and often leads to holding dollars as singularly more important than anything else; sometimes even human life.

Whether or not you agree with my point that efficiency driven quantitative metrics are not inspiring, it’s kind of hard to argue with 40 years of scientifically proven fact that when “a measure becomes a target, it ceases to be a good measure”.

DevOps shops “live to serve”, while ITIL shops “serve to live”

DevOps contains a concept of craft that extends beyond efficiency. When a problem arrives in a DevOps shop (or at least one that really embraces the cultural ethos), the staff looks at the real resolution of that defect as a matter of personal pride. This pride is not a statement of “pride in self” however, but a statement in “pride in craft”. To hammer this contrast home, does ITIL by itself even have a deep cultural bias at all, much less an ethos?

This perspective is what allows DevOps shops to act in a manner that transcends the checkbox approaches that pervade formalist driven shops that believe in “adherence to process” above all rather than the “results the process is intended to achieve”. If you think I’m wrong, go ask your business clients how they feel about the DMV-like experiences they get when trying to get a problem solved.

This perspective is also why DevOps forms strategic pillars in automation and performance. Performance and automation efforts are not only about the manufacturing of time. Automation isn’t only a method of driving towards error free processes, or even a liberator of individuals from mundane tedium. Driving highly performant code and infrastructure isn’t only about conservation of compute resources.

Well implemented automation routines and highly performant applications are also a deeply creative expression of both development and operations practitioners. The same sense of artistry and elegance that was once reserved for just the software developer has been opened up to all IT practitioners. With the arrival of DevOps, being a great IT professional is not out of reach of any IT practitioner no matter their discipline.

We just can’t afford to mess this one up. In order not to, here is some practical advice:

As you go forth in creating or playing your part in your organization’s DevOps journey, look out for these pitfalls and pivot back to the center.
1. Reductionism – Don’t let numbers be the mission. This doesn’t mean don’t have numbers in your goals. It only means to make sure that your number oriented goals have a basis in making things better for people. When you hear someone say “We need to get this deployment routine under 4 hours!”, make sure that someone reminds everyone that the reason to get it under 4 hours is to allow employees to have a reasonably well balanced life outside of work. Off hours deployments that last four hours, when things go right, should not be a target that anyone is proud of. Getting people home to their families and allowing people to be able to keep a healthy sleep schedule is goal worth shooting for.
2. Formalism – Worshipping the shovel rather than the hole it digs is an easy trap to fall into for people whose jobs revolve around buiding and using tools. Remember that processes and tools are mechanisms intended to guide novices and intermediates through decisions. When you see someone turning a process or a tool into a sword and shield for working with others, make sure you remind them that results are better than processes.
People Over Process

Neither processes nor tools solve problems; People do. Processes and tools don’t create business opportunities by themselves; People do. Isn’t that why we chose this profession in the first place? To be useful in solving problems and to create opportunities to make things better while allowing people (ourselves included) to have a good standard of living.

Now go DevOps practitioners! Go kick some ass and make the internet awesome!
April 18, 2014
Nagios is not a monitoring strategy

When I visit clients to talk about DevOps, I usually ask them what their monitoring strategy is. Too often, the answer I hear is “We use Nagios”. I think Nagios is a great tool, but it sure is not a strategy. Nagios does a good job of monitoring infrastructure. It will alert you when you are running out of disk, CPU, or memory. I call this reactive monitoring. In other words, Nagios is telling you that your resources are getting maxed out and you are about to have issues. Proactive monitoring focuses more on the behavior of the applications and attempts to detect when metrics are starting to stray away from their normal baseline numbers. Proactive monitoring alerts you that the system is starting to experience symptoms that can lead to a degradation of performance or capacity issues which is more preferable than Nagios telling you are about to be screwed. With reactive monitoring, it is not uncommon that customers start complaining about the same time that the Nagios alerts start going off. The goal of proactive monitoring is to head off issues so that customers don’t even notice.

The next question I ask is “What things are you monitoring?” A typical answer usually revolves around various infrastructure assets and databases. That’s a good start but there is much more to consider. But first, let’s talk about why proactive monitoring is so critical. In the pre-cloud days we used to ship software to our customers where they would install the software, perform capacity planning tasks, manage the infrastructure, and operate the day-to-day activities. Once we shipped the code we were done. In today’s world, we are no longer shipping product. Instead we are delivering services that are always on. The customer no longer owns and operates the infrastructure and the software. Instead they pay for a service and expect that service to run reliably all the time. To meet those expectations, we need a more robust monitoring strategy. We need to monitor more than just the infrastructure.

A good monitoring strategy starts by identifying all of the actors who needs access to data and all of the categories of data that needs to be tracked. Some metrics are monitored in real-time while others are mined from log data. Every good monitoring strategy is accompanied with a sound logging solution. In order to perform analytics to predict trends within the data, one must collect various data points ranging from customer usage activity, security controls, deployment activities, and much more. The following presentation goes into much more detail about the different areas that should be monitored and why different actors need these data points to perform their jobs.

[slideshare id=33320431&doc=monitoringstrategies-0408014v1-140409081614-phpapp01]

The bottom line is, before building in the cloud, it pays to invest some time into a sound monitoring strategy. I have seen too often where teams don’t think through how to support these highly distributed, always on SaaS solutions and end up delivering software that does not meet the reliability and quality expectations of customers. Monitoring provides feedback to developers, product owners, operators, and even customers so that systems can continuously be improved. Nagios is great, but there is no single monitoring solution that can implemented to effectively operate today’s always on services.

April 15, 2014
Not all collaboration types are equal

Talking about collaboration in general terms is easy. I’ve done it and so has everyone else. It’s time now to dig deeper into the specifics of collaboration, meaning the different types – asynchronous vs. synchronous, vertical vs. horizontal – and how different companies fit into the magic collaboration quadrant. And yes, the quadrant is indeed magical.

Let’s start with the two types of collaboration that are time-relevant, asynchronous and synchronous. Asynchronous collaboration takes place outside of real-time. The best example of a ubiquitous asynchronous collaboration tool is email. You send an email and the receiver has time to digest the information. The drawback to asynchronous communication is that it’s inadequate as a means of communicating a sense of urgency. You have to wait for someone to respond to an email and that lack of immediacy can cause damage in troubleshooting or problem solving situations. Email replies tend to include the original message for context because you can’t rely on the timing of the message to keep the context.

On the other side of that collaboration coin, you have synchronous collaboration. Synchronous communication takes place like a conversation and more closely resembles a face-to-face interaction. A chat session is the ideal illustration of this concept. The obvious benefit to synchronous collaboration is the ability to send and receive information right away. This adds to the feeling of immediacy and tends to provide a more effective means of working together to solve issues in real-time.

Now that we have that covered, it’s time to move to the other defining aspect of collaboration: whether it’s vertical or horizontal. Vertical collaboration consists of problem solving tools that are domain-specific. An example of a vertical collaboration tool is one built for certain parts of the business like Salesforce for the Sales & Marketing teams or Rally for the development team. Horizontal collaboration consists of non-domain specific tools, meaning that they have potential use across all teams of an organization. Project management tools like Campfire or remote meeting solutions like Join.me or WebEx are examples of horizontal collaboration tools which tend to be widely adopted but lack the ability to really go very deep in helping in a domain-specific way.

Check out the quadrant I created to better show where different companies currently fit into the ever-growing collaboration landscape, based on the definitions provided above.

When looking at the state of collaboration tools, it’s interesting to note where most companies land within the quadrant. Why so few vertical synchronous ones? I’ll answer that next time.

April 11, 2014
Programmability in the Network: Stop a Bleeding Heart…

It is not often the case that a security vulnerability can get the entire Internet talking. And not just the security community on the Internet, but everyone. End-users and IT alike are looking for answers and trying to mitigate Heartbleed. It has its own web site and logo. It’s that big of a deal.

Many service providers have already patched their systems, but there are a whole lot of sites on the Internet and it’s estimated that a significant number of them are vulnerable.

Netcraft notes that Heartbleed, based on OpenSSL, “affects around 17% of SSL web servers which use certificates issued by trusted certificate authorities.” One of the most trusted sources of data regarding web server software in use today, Netcraft’s “most recent SSL Survey found that the heartbeat extension was enabled on 17.5% of SSL sites, accounting for around half a million certificates issued by trusted certificate authorities.”

But it’s not just server software that may be vulnerable. Any device in the data path that terminates SSL and is based on OpenSSL could be vulnerable. Any “thing” that might use OpenSSL to enable secure communication and accepts connections could be vulnerable. In other words, hearts are bleeding all over the Internet right now, and it could take quite a while for the Internet in general to plug the holes.

The hole itself is fairly simple – it takes advantage of the mishandling of what seems like an innocuous meta-protocol that runs atop TLS and DTLS that consists of two message types: HeartbeatRequest and HeartbeatResponse. It’s used to “ping” the other party in a secure connection to make sure the peer is alive. It can be used to offset idle times that might otherwise cause the connection to time out.

It can also, thanks to a missing bounds check on the message structure, arbitrarily and completely silently grab 64 kilobytes of the server’s memory. That memory might include sensitive things like the private key used to encrypt communications or passwords or pieces of critical documents.

Whatever might be in memory when the vulnerable software goes to grab it is leaving the building faster than Elvis – and without a trace. There’s no logs, no evidence of the theft, nothing. It’s clean and remotely executable by anyone on the Internet. That may explain the attention it’s currently getting.

The answer to any vulnerability is ultimately to patch it. Depending on how many servers or systems are impacted, that could take some organizations days – assuming patches are available immediately. In the meantime, the heart is still bleeding and the only way to prevent more information from leaking is to shut off the hose.

Except it’s not the only way.

Programmability in the data path can mitigate this, now. Right now. Like in the next five minutes.

Let’s review programmability in the network for a brief moment: you have, in the data path, a device. That device (or software) is capable of programmatically intercepting, inspecting, and transforming the data that’s traversing it. That means such a system can intercept a request to initiate SSL/TLS traffic, inspect it and, if it finds a request to include Heartbeat support it can reject the connection.

Interestingly, one of the primary motivators of early SDN architectures was the ability to support new or experimental protocols by inserting them into the data path in the network. While that’s a driver for academics, a more realistic driver for businesses is the ability to interact with protocols in flight and “fix” them – especially when the “fix” involves preventing the exploitation of a vulnerability.

Programmability in the network enables this kind of rapid response to threats. Being able to deploy a solution very quickly to prevent further data leakage provides protection for both the organization and those who interact with its properties (employees, partners and customers). It also offers time for IT to patch and update all impacted systems without remaining vulnerable or taking the drastic step of going completely dark until the situation is resolved.

By enabling a view of “infrastructure as code”, programmability not only enables a more software-oriented approach to deploying services, but also a software-oriented approach to quickly creating and deploying an appropriate response to emerging threats.

A devops approach can facilitate this capability, and push protection into the network as code.

April 10, 2014
Services As A Model For DevOps Communications
DevOps is an interesting movement. As with any effort of this magnitude, there will be many opinions on what is entailed in adoption and implementation. The issue is that businesses are so diverse and varied in culture that no single method for adoption is guaranteed to fit all businesses. In large IT organizations, this issue is additionally hampered by the fact that multiple cultures and dominions exist.

Thus, I am not a believer that DevOps adoption comes from the approach that the business commits to adopting DevOps. The business has committed to many initiatives over time that have not panned out. This is not necessarily failure, but perhaps just shifting priorities, shifting political landscape, lack of training or simply a lack of desire to change. “The best laid plans of mice and men ….” and all.

If commitment is not hampered by people, then there’s a good chance that DevOps adoption may be hampered by silos. This does not make silos our enemy. Silos help to compartmentalize and provide focus. Still, silos limit the ability to orchestrate process without involvement and participation from each of the silos.

Regardless of the reason, the challenge stands, how to align a process without the use of force. The answer is protocol. With protocol, an individual or group does not need to buy into the entire process, but just needs to understand their role. Short of incompetence or insubordination, or even both, which will eventually result in the removal of these particular types of hurdles, following protocol ensures that every role in an activity will complete their action and, thereby, facilitate the whole with roughly an 80% success rate.

I posit that the service is the means for defining protocol and of communicating that protocol into an IT organization, such that DevOps is the natural activity and outcome. A proper service entails defining a number attributes, for example:
- Metered
- Valued (priced)
- Advertised / Catalogued
- Governed
- Monitored
- Supported
- Documented
- Quality
- Available
- Intuitive
It is up to each area of IT participating in the DevOps process—Design, Develop, Test, Stage, Deploy, Operate—to contribute to defining and providing their completed artifacts that pertain to formulation of the service. The service will guide the development in the understanding of operational characteristics and will allow operations to ensure they can deliver and support the service to the level of expectations of the consumer. Moreover, in those cases where the service is not meeting appropriate levels, it will allow them to provide development with the appropriate data and metrics to take corrective action, thus leading to fewer and shorter outages and failures.
April 10, 2014
Automation versus Orchestration

Yes, Virginia, there is a difference.

One of the things devops practitioners are tasked with is the provisioning and configuration of all sorts of infrastructure. Application servers, web servers, load balancers, proxies and database servers are among the lengthy (and no doubt growing) list of “boxes” devops needs to get up and running to support just about any given application today. One of the key value propositions of a devops approach to operations is that it can reduce the time it takes to get applications to market by getting them up and running in production faster. That’s increasingly important as we move into the Era of Things, driven by extreme connectivity of everything, because all those “things” need are going to need to be talking to applications on the back side.

So speed is of the essence, but not at the cost of accuracy. We’re encouraged, then, to automate tasks and orchestrate processes to achieve that speed. The side effect of codification of tasks is higher accuracy (aka fewer mistakes).

So far so good.

But you may notice that I have (and will continue to) clearly delineate between “automate” and “orchestrate”, between “task” and “process”.

The reason for that is that, simply, they are not the same thing. We should strive not to confuse these two because while the two share some benefits, there are other benefits that are achievable only by one or the other – or the combination thereof.

Process optimization, for example, can’t be achieved by simple automation. Automation is concerned with a single task – launching a web server, configuring a web server, stopping a service. Orchestration, however, is concerned with automating, if you will, the execution of a workflow – of a process. A provisioning process may be comprised of multiple tasks and involve multiple systems. An “application” is not just a single server, after all, it’s likely several servers – web, app and database in a traditional three-tier architecture.

The goal of orchestration is not just to automatically execute a service, which brings speed to the deployment process and gets applications into production faster. It also affords an opportunity to streamline – to optimize – those processes for even greater gains in deployment velocity.

One of the simplest ways to optimize a process is to eliminate repetitive steps. This entry on Vagrant explains it fairly well :

“We could just SSH in and install a webserver and be on our way, but then every person who used Vagrant would have to do the same thing. Instead, Vagrant has built-in support for automated provisioning. Using this feature, Vagrant will automatically install software when you vagrant up so that the guest machine can be repeatably created and ready-to-use.” [emphasis mine]

— Vagrant Docs “Getting Started | Provisioning”

Now certainly the step of installing a web server on a new box with vagrant is hardly all that time consuming, it’s just a single line of text. But every line of text must be manually typed, and that introduces the possibility of fat fingering the command, which either consumes time in fixing it or introduces an error into the system. Neither are desirable results.

Ensuring that the process is repeatable and orchestrated by automating all the tasks that comprise that process an optimize it by eliminating redundancy. That results in greater deployment velocity.

Recognizing that processes should be comprised of composable tasks offers a higher degree of reuse across operations. The task of provisioning and configuration a web server with the appropriate base software means consistency, aka standardization, across all applications. Processes that take advantage of standardization result in greater infrastructure stability, another benefit often cited by organizations that have adopted a devops approach to operations.

The key here is to recognize the difference between automation and orchestration and how they relate to one another. Automation is about codifying tasks and orchestration is about codifying processes. Orchestration takes advantage of automation by reusing these basic building blocks.

There is a difference, and done right, the two play off one another to result in streamlined, more accurate execution of operational processes that speed up successful application deployments.

April 10, 2014
A Winning DevOps Approach for Established Enterprises: Pace-layered
Does your bank offer the ability to deposit a check by taking its picture with your smartphone? When that feature debuted a few years back, I remember thinking it was interesting and convenient enough that if my bank didn’t offer it soon, I’d be willing to switch to one that did.

I bring this example up for two reasons: one, it’s a microcosm of the larger race to innovation that is happening in every company and industry as we speak; and two, it involves banks, which are good touch-points for a discussion about the unique challenges established enterprises face as they attempt to implement DevOps practices in their IT organizations.

We know that DevOps means incorporating best-practice application delivery methods that improve communication, integration and collaboration between development and operations, but what is its ultimate goal? Quite simply, getting high-quality services to market faster than competitors – at a cost that aligns with both IT budgets and business goals. And as my check-depositing story illustrates, if you’re not first in the market, you at least need to be nimble enough to catch up, or you risk losing customers.

For long-time enterprises, this can be easier said than done. In my previous blog about DevOps culture, I used Etsy as an example of an organization that has had great success with its DevOps transformation, but the e-commerce company has been around for less than a decade. Its young IT architecture is apples to the oranges of established enterprises, whose complex environments often comprise a mix of legacy and modern technologies that have been cobbled together over time.

As a result, the DevOps journey is typically longer and more difficult for such organizations – but that doesn’t mean they can afford not to pursue it.

Improving Deployment Practices at Every Infrastructure Layer

When you look at the IT infrastructure of an established enterprise, you normally find three layers of technology. Let’s call the first “systems of record.” These are the legacy transactional systems (often mainframes) that have been with a company since the beginning. They do their jobs well, but their size and inflexibility make them resistant to the type of Agile development that DevOps typically embraces.

The next layer is “systems of differentiation,” or the various hardware, software and middleware components an organization accumulates as it grows, merges with/acquires companies and enters new markets. While these systems are often more Agile-ready than the systems of record, their decoupled/distributed nature can complicate and slow down development and deployment processes.

Finally, there is the “systems of innovation” layer, which is made up of mobile and Web applications and leverages the latest technologies. Because of its light footprint and flexibility, this layer is the most adaptable to DevOps practices. In fact, as you move up the layers from systems of record to systems of innovation, the holy grail of high-quality, fast and low-cost deployment becomes more and more attainable.

What does this mean from a macro perspective? Basically that established enterprises need to take a pace-layered approach, where application delivery occurs at different paces within the different areas of their architectures. The point, though, is to improve in each layer.

So if an organization pushes out a mainframe update once per year, perhaps it can move to a bi-annual schedule (with support from the right people, processes and technology, of course). On the other hand, a Web team that publishes updates once or twice a week could leverage DevOps tactics and technologies to adopt a daily update cadence. Bottom line: Different deployment paces are fine across layers, as long as improvements are occurring across the board.

Kick-starting the Journey with Service Virtualization

Now that we know why the DevOps journey can be difficult for established enterprises, let’s look at how one such organization, namely Union Bank, overcame these challenges and achieved success.

According to an article on Bank Systems & Technology, Union Bank was looking for ways to compete with top-tier banks, so it began “acquiring other institutions, updating legacy systems and launching new front-end applications.” In order to support these efforts, the bank implemented a service virtualization solution, which simulates constrained or unavailable systems across the software development lifecycle (SDLC). This allowed its developers, testers and performance teams to work in parallel for faster delivery and higher application quality and reliability.

Here are some of the results Union Bank has achieved since:
- Cut testing environment setup time from 42 days to 3 days
- Slashed three and a half months from mobile and Web application development time
- Reduced the cost of testing cycles by 65 to 85 percent
Obviously, a service virtualization solution is only one piece of the DevOps puzzle, but Union Bank stands as a powerful example of how a long-standing enterprise can leverage such a tool to be more innovative and nimble in the marketplace.
April 9, 2014