devopsy.kinsta.cloud

Category: Features

DevOps Contrarian

I admire and appreciate the ideas of DevOps as a result of living through two decades of working in IT and experiencing the pain of what I would call the “failed state of IT”. The failed state of IT being something inherently broken, something that couldn’t survive on its own, something that eats its budget and provides no value and something where the people working in such a system don’t even see the worth of the system and stop challenging it all together. In order to correct or prevent these “failed states” I often apply my experiences learned through DevOps and Lean processes and values.

DevOps to me has been about recognizing and pulling yourself out of this “failed state” concept of IT and doing so with a little bit of empathy, a little bit of common sense and a lot of science. I don’t necessarily mean pure science in trying to come up with an underlying law that describes and predicts your system of work, but the foundations of science – The ability to question everything, the ability to collect data, analyze data, measure data and develop analysis thereof. By thinking “DevOps Science” you choose to develop models that describe your work, define your flow, show the value of your system and tailor it to your business needs and to research the best models to achieve your end goal and strive to continually improve those models. DevOps is about putting this science to work, and doing so with collected intelligence & experimentation behind it.

Where I find myself being a DevOps contrarian of sorts is in the often over simplified representation of DevOps as an applied model. I don’t have any issue whatsoever with the concepts described below by themselves; they create amazing discussions and speak to very core foundations of DevOps, but they’re often only spoken of superficially.

DevOps is not just about Culture.

I often see people speak of DevOps as an applied “Cultural Invention”, an innovation found to be useful to a group of people in their behavior. While that is a great description from a 30 foot view as if you’re looking from the outside, it really doesn’t help one understand culture in any sense to help foster it. DevOps culture isn’t about copying what others do and hoping for success, it’s about developing it within realizing that its affected by both forces encouraging change and forces resisting change and finding the balance there of is that is important. You can’t simply force DevOps on an organization and call that culture. The real power of DevOps as a cultural invention is being able to develop DevOps values that directly apply to your organization so that it can be passed down generation to generation and survive on its own. Thinking scientifically, we learn from these shared examples of cultural invention and their successes by experimenting with them to see how they fit our organization as there is no universal law of culture but merely models we can research and identify with. It’s great to build upon the success for others, but the only way to validate those successes is to test them yourself!

Another danger of thinking applied culture is the development of cultural optimums, I parallel this to what Garret Hardin coined “The Tragedy of the commons”, whereby individuals, acting independently and rationally according to each one’s self-interest, behave contrary to the whole group’s long-term best interests by depleting some common resource. I won’t claim to define what your resource is that you’re depleting, but it’s something to keep in your mind as you think of culture because it’s important to understand culture external of yourself as much as it is within. The value of culture isn’t in how you rationalize it for your own goals, but how it operates on its own and if that culture is appropriate for the good of the entire system (business). The strongest metaphor in DevOps for depleting the wrong resource is wasted resources not working towards your common goal. Are you actively engaged on the right tasks? Right automation? Right flows? Are you merely creating more work or are you adding value? DevOps isn’t something you apply methodologically, its something you experiment with methodologically.

DevOps is not just about automation.

Automation is the process for operating with minimal or reduced human intervention. If you’re focused on automating processes are you still improving that process or did you think to yourself “I automated that, now I’m free to work on other stuff”? Automation can tempt you into doing “more work” if not kept under control and that creates a real danger of losing focus, having too much work in progress and actually hurting your overall performance.

Often times the best value in the systems we call “Systems Automation“ – the Chef, CFEngine, Puppet, Ansible & Saltstack systems is not necessarily their amazing capability to automate processes but their ability to document processes. Those cookbooks, manifests, scripts and DSL’s you’re using are absolutely the best documentation of your system you can have. They provide a great resource when done in such a way to easily describe the processes that they’re automating so that they’re well enough documented for improving them. You can’t automate that which you don’t understand and the best way to understand them is to make sure the processes that describe them are easily readable, documented, tested, and validated. They should be part of your experiment, not just testing for failure but testing for improvement.

Through the lens of science, we can look at automation more as standardizing on a process and applying the best resource to that process. Sometimes it is compute power instead of human power but don’t fall into the trap that the process is said and done (Automated all the things!) and time to move on! Science is a never ending process by where you build upon your conclusions!

One conclusion I and many others have come across is that people are the most flexible resource you have! It may be a better value to your business to have the flexibility of a human resource if the process you’re trying to build requires flexibility. Even when planning for automation, requiring human intervention is a perfectly valid result and a justification you can sell to your business that has been proven through experimentation & data. Don’t ride the automation bandwagon when your data shows otherwise!

DevOps is not just about measuring.

What are you doing with the data you measure? What value does measuring give your organization? When you have the flu and you take your own temperature, what are you measuring for? Do you have the cultural aptitude in your organization to accept the ramifications of what you measured? Can your organization stop working when it’s “sick”, take the time to heal and start back up after taking corrective or remediation actions?

Measuring is extremely important, but measuring isn’t what DevOps is. Measuring is a part of what makes your organization responsive, culturally adept, and accountable for itself. It’s what allows you to convey value, to compare itself against and strive for improvement. If you’re not measuring, what are you improving against? Inversely, If all you’re doing is measuring, what is the value of your measurements? What are they the basis of? I like to think of biological systems when I start thinking of computing systems where there is a cost of energy to do something. Are you utilizing that cost the best way you can by measuring what you are or should you evolve the system to spend that energy elsewhere? Are you collecting the right metrics to even comprehend your systems? Do your metrics allow for capturing exaptation and evolution thereof? When you experiment, measure, rinse & repeat you get patterns; measuring helps identify those patterns. Creativity, experimentation and the process of science allow you to act upon those patterns and create value from those patterns. Some people call that innovation.

DevOps is not just about sharing.

There are people who believe that unless you are open source, have a GitHub repo and publish your code, you’re not really sharing and you’re not DevOps. I certainly hope we’re not going down this path and certainly hope we can appeal to the broader ideas of sharing. Sharing is about the joint use of a resource and sharing is knowledge charity. Sharing goes hand in hand with culture in not just expressing your needs and sharing your knowledge, but having the ability to listen, feel empathy and accept what others share as well. When it comes to collaboration & sharing it’s much easier to accomplish if everyone sees it through the lens of science, not only do you speak to DevOps values more rationally but you can even accept opposing views and ideas much easier because you have evidence to them and aren’t emotionally tied to their success, you’re tied the process that challenges their success as a way to improve them!

DevOps through the lens of Science.

The beauty of DevOps through the lens of science is that science is about challenging ideas, accepting failure, learning from history/prior experiments and having the strength to not only say those successes and failures are a valid output of your experiments but also come up with a new experiment or new hypotheses and do the process all over again. This helps define your culture, this helps setup what you should automate, this helps you validate what to measure and this develops what you should share. Through the lens of science you learn through experimentation, testing & verification what works for you, what creates value, what brings you closer to your goal. When honestly done you also learn what doesn’t work for you, where you failed, and how you can use those lessons in failure to continuously improve. A failed hypothesis after all is still good data!

Why so much focus on the sciences? Personally, I enjoy astrophysics not because there is direct monetary value in understanding the cosmos around us but because of the philosophical nature of being, the beauty of knowing and the questions it gets you asking as the more you know, the more you wonder. That wonder, is what makes DevOps so great to me. That wonder is what drives continuous improvement. That wonder is what drives the “what next…”. That wonder is what pushes “question everything”. That wonder is always what pushes me to accept and understand the science we do know and trust it because it’s based upon the values that are respected by thousands of scientists all over the world through peer review and consensus. That foundation is not the assumption that we think we know it all, but that we know enough to describe something and we trust that description good is enough that we challenge you to break it. I would highly recommend that any DevOps practitioner cross pollinate their technology passion with the sciences as there is nothing more exciting than realizing commonalities of biology, physics, astrophysics and the laws of nature within your own thoughts and patterns as they’re applied to DevOps.

So be a DevOps contrarian; for its not accepting DevOps as the answer, it’s the challenge there of that will make it aspire to bigger and better things. It’s not that Culture, Automation, Measuring and Sharing is wrong by any means, but they’re just scratching the surface of what you need to understand to make DevOps “Work” for your organization.

I’m not only a DevOps contrarian, but I’m an OS contrarian, a Developer contrarian, a Systems Contrarian. I’ve made the best of running Windows, Oracle Linux, Oracle Database, eBusiness Suite, Weblogic and much more. Not because I’ve personally identified or I am idealistically aligned with any of these tools & systems, but because they solve the needs of the business; they create value, they operate and perform admirably and we have proven their worth. It’s no utopian system by any stretch of the imagination, but I find great reward in the continuous experimentation of making these systems work and improving upon them in a very DevOps way.

After practicing DevOps & Lean principals and having over two decades of IT experience, I’ve realized the “failed states of IT” and people who thrive in them are people who stopped critical thinking, stopped challenging the status quo and stopped experimenting. They stopped having a contrarian.

March 25, 2014
Config Management & F***ing Shell Scripts

Joke tech projects pop up all the time. Some, like the lamentable titstare, should never have seen the light of day. Others, like Mark Zuckerberg’s Facemash start out as a joke and grow into something that is anything but. Most come and go without anyone really noticing.

Whilst their lives are typically short we often learn more from people’s reactions to these projects than we do from their use of them. An excellent example of this popped up recently when a couple of frustrated Chef and Ansible users came up with a simple configuration management alternative, the regrettably named F***ing Shell Scripts (FSS). As they explain it, as in “Why can’t we just use f***ing shell scripts?”.

Whilst usability issues with configuration management tools – particularly in the Enterprise – is a topic close to my heart, I would not normally have had much more than a passing interest in FSS. What grabbed my attention was the reaction it elicited when posted to Hacker News. With over 150 comments FSS clearly touched a nerve.

Now to clarify, the authors of the project were quite definitive in assuring the mob that they had created FSS as a joke:

We wrote it literally in an afternoon, and were giggling like schoolgirls the whole time.

People either missed this, or chose to ignore it:

“This is simply brilliant! I’ve dreamt about such tools.”
– artellectual

“I’d love to see a tool like yours be brought to full maturity, then I think I’d love to use it.”
– sdegutis

It provided a perfect platform for many to voice their frustrations with current tools:

“Even for an experienced sysadmin, using Puppet and Chef to do even a trivial project (replacing a ~10 line shell script) took a painful couple of days.”
– mattjaynes

“In my experience, there’s a huge amount of work involved in just getting Ansible (and other CM systems) to work right in the first place”
– s_kilk

“We’re using Chef right now, and evaluating other options. They seem to either be between way too over-engineered, or not flexible enough, but nowhere in between.”
– sdegutis

“Keep it simple. puppet/chef have horrendous DSL’s that make it really complicated to reason about.”
– sp1982

In the interest of balance I should point out that there were some voices of support amongst the crowd. Ansible in particular won praise for the effort its team has put into simplifying things.

Overall though I was still blown away with the level of frustration being expressed. Automation tools have been around for years now; CFEngine, the grandfather of the modern tools was created in 1993. Despite this, a tongue in cheek project suggesting ditching them all in favor of the technology they set out to replace (or at least abstract away), shell scripts, is considered credible. For me this is akin to people ditching virtualization for physical boxes. It just doesn’t make sense.

Unless of course something is still fundamentally broken in the approach the current set of tools is taking; that they are still too hard; that they still fail to address key prerequisites like visibility and accountability. I believe that this is the case because I’ve experienced it first hand; I’ve seen it a hundred times in businesses I’ve talked to, and threads like the Hacker News one mentioned above leave little room for alternative interpretations. Heck, we’ve based our product GuardRail on this entire premise; these are not empty words on my part.

Incremental improvements such as those offered by Ansible are a good thing, but what people seem to be searching for is a step change. As things stand this is less likely to come from the current batch of tools, and far more likely to come in the form of a new approach. Docker anyone?

March 24, 2014
What “Programmable” Means

One of the most often used words these days to describe emerging architectures and technology is (drum roll please) “programmable.”

Now, from my used-to-be-a-developer-and-architect-turned-infrastructure-geek perspective, I know what that means. It means I get to write code. But to what purpose that code is put to is a question that’s rarely answered. So today we’re going to answer it because, well, my used-to-be-a-developer-and-architect-turned-infrastructure-geek persona likes to talk about code.

When you hear the term programmable used in the context of Software-Defined Networks (SDN) they’re talking about the same things that “programmable” means in the context of Devops: programmatic control of both the data and the control path.

The easiest way to visualize the difference between these impressive sounding terms is to recognize that the data path is inside and the control path is outside.

What that means is if you’re interacting with the data path, you are participating in it. You can inspect it, evaluate it, and transform it and, perhaps more importantly, you can make decisions based on data in flight. That’s powerful for the network, because a variety of Devops patterns are based on this very concept and require the network for implementation. Consider, for example, the implementation of an A/B testing pattern. The basic premise of the pattern is that somehow the identity of a user is known and, based on their membership in a defined group (beta testing customers, roles within the organization, volunteers from the community, etc…) they are routed to different versions of the same application to determine how differences in the UI impact behavior.

Notice the term routing, which implies it is a function of the network. The reason programmability is critical is two-fold: one, you must be able to identify the user as belonging to a specific group and two, you must be able to dynamically route based on that information. The former, identification, could ostensibly be accomplished using fixed lists IFF (If and Only If) it is possible to map a user to some variable readily accessible to the network, which traditionally means things like IP and MAC addresses and TCP ports. Most folks will immediately recognize that today this isn’t enough (and actually it hasn’t been enough since the days of the mega-proxy, but I digress). The information that’s needed is much higher in the networking stack at the application layer, aka in the OSI model as Layer 7. Which means you’re probably going to have to dig into HTTP headers and maybe the payload to get the data you want. That requires programmability in the data path, inside.

To really make the pattern scale (and to make it flexible enough to support a phased approach to beta testing) you probably don’t want group membership to be a static, fixed thing. So now you’re going to want to extend your implementation to include making a call to a database or identity store (LDAP, AD, etc…) to get current information. That requires programmability in the data path, inside.

So what programmability in the data path means is the ability to use code to direct the behavior of some “thing” in the network, be it an application proxy or a load balancer or what have you. You can’t easily do this through configuration-based proxies because they don’t allow for the execution of the domain logic necessary. There are a limited number of interactions you can achieve with configuration-based systems because of the logic they’re hard-coded to execute,and most of them don’t include calling out to a data store to look-up arbitrary information.

Data path programmability = Code in Flight. So you now you know where the name of the blog came from. This developer-and-architect-turned-infrastructure-geek finds the intersection between programmability and networking irresistible.

Okay, so that’s data path programmability. What about control path programmability?

Well, that’s the easy part. Control path programmability is on the outside. It’s the APIs that enable automation and orchestration of provisioning and configuration. Control path programmability is how you integrate infrastructure into command control systems like OpenStack, Puppet and Chef. Infrastructure comes with varying degrees of API-enablement, some extremely granular and others less so, depending on the particular function and maturity of the vendor.

So what you can do with the control path depends on the API, but in general it enables things like provisioning new services – think configuring a new virtual server in Apache and all the settings that accompany it – or start up and shut down functions that enable rolling maintenance across a set of devices or servers to avoid downtime.

But all control path programmability is on the outside, enabling integration and management without having to learn a CLI or configuration file syntax.

SDN and Devops use both programmability to reach relatively the same goals: scale of operations, rapid provisioning of services, to decrease the possibility of human error (the fat finger factor) and to introduce flexibility into systems that were previously bound to fixed processing and configuration-based paradigms.

Programmability in the network is going to enable a variety of new patterns, processes and services and as traditional infrastructure becomes more and more programmable through the adoption of architectural approaches like SDN, Devops is going to see more and more opportunity to apply its maturing expertise in leveraging programmability to a much wider audience.

March 24, 2014
Don’t let automation be the tail wagging the dog

There is quite a bit of talk right now about DevOps and how important automation is. This could be compared to GM in the 1980s when the company tried to transform the way they built cars through automation and robot. In the early 1980’s GM spent $90B dollars in an attempt to speed up the way cars were delivered to the market. This failed with famous stories of robots painting each other and others welding doors shut. Ultimately this project failed and the robots were scraped. It was later reported that the money lost on the project could have been spent to acquire both Toyota and Nissan at the time.

Automation dominates the conversation when it comes to DevOps. If you do not learn how to build cookbooks you are not doing DevOps. I fear this is going to set companies trying to transform IT off on the wrong track, focused on the outcome and not on the scientific process to get there. Alan Sharp-Paul recently wrote that you cannot automate what you don’t understand. The best way to understand it is in two key ways, change and performance.

Let Science set you free

This was the secret that Toyota used to dominate the automobile industry. They were able to transform all of their employees into little scientist. Constantly tinkering to squeeze our inefficiencies in the system. Every worker was empowered to think how things could be done better.

At the heart of the agile and DevOps processes is the practical application of the scientific method. Think back to your 8th grade science fairs. No not the volcanos but the ones that won best of show. These had a clear hypothesis that stated what they were going to try and prove. They were able to understand what they were going to change and measure the difference between the before and after. They could clearly state what changed and if it was a successful experiment or a failure.

This thinking is key for companies trying to transform to a DevOps IT organization. If a company does not understand the experiments that are running and what the outcomes should be then automation will just make the problem worse. That is why change management and performance management are foundational for transforming to a DevOps culture.

Tracking your experiments

Any good scientist knows exactly what he is working on. He knows every experiment that is running anticipating the outcome. He knows all the details about each change that has been made to see how they impact the outcome. This is the same way that IT should be tracking every change happening.

Every change done in IT is a scientific experiment running. Tracking every change allows for understanding of the outcome of changes within the application. Developers and Operations will be more informed about the cause and effect of their changes. There are multiple different ways that you can track changes but the key is that you are doing it very well before you start to apply automation.

Measuring the outcome

There is no need to track the experiments if there are not being measured. A scientist is very deliberate in how they measure the outcomes. Good enough just does not get this done. This is beyond just monitoring. Monitoring in a lot of cases is after the fact, in production. This is about measuring the difference for every experiment regardless of where it is being tests. This creates the need to test changes all the way to when the first functional tests are done, measuring the changes of every code update. Following it through all of testing and to production. Understanding how each experiment impacts the system as a whole.

Efficiency through automation

Once there is good understanding of all the experiments and a standard platform for measuring the outcome then automation starts to return exponential benefit. Because every experiment it known the lower priority ones can be identified and automated. Risk is better assessed. Processes can start to speed up because there is visibility and accountability. Applications are no longer classified as high risk or low risk only change is classified that way. Determining what can be automated just becomes another experiment that is tried and measured for success and and failure. Automation becomes just another outcome of DevOps to the business and not the route to achieve it.

March 21, 2014
Busting The 4 Biggest DevOps Security Myths

Last month when the IT security world convened in San Francisco for the annual RSA confab, a number of security experts tackled DevOps in talks that covered the crossover between continuous improvement and risk management. Among them was a lively panel mythbusting some of the most prevalent misconceptions about DevOps held by the security community.

Moderated by Dwayne Melancon of Tripwire, the panel was made up of Josh Corman, CTO of Sonatype, Nick Galbreath, vice president of engineering at IPONWEB, Gene Kim, author of The Phoenix Project, and David Mortman chief security architect and distinguished engineer for Dell Enstratius, all of them security professionals and very active DevOps evangelists. The banter backed-and-forthed between the participants, with four myths bubbling to the surface as particular highlights from the discussion.

Myth #1: DevOps Pace Will Leave Security In The Dust

One of the most prevalent myths about DevOps within the security community is that the rapid deploys of the approach will throw all risk management precautions to the wind and leave security practices behind as a result. Quite the contrary, panelists argued. They believe that DevOps patterns make it more possible than ever to bake security practices into IT processes more than before, when it is frequently bolted on “at the end of the caboose,” as Galbreath puts it.

“You’re not throwing out your current security system,” he said. “What you’re doing is removing a lot of the junk work that eats up so much time and keeps people underwater. You’re really enabling different teams to all improve security together.”

As Corman explained, security people say all the time that complexity is the enemy of security.

“It also happens to be the enemy of stability,” he said. “That is the reason DevOps people come willingly with open arms to the security community.”

Not to mention, Kim added, “because you can’t be highly available and have stuff coming continuously through the pipeline if it is not secure.

Myth #2: DevOps Destroys Compliance And Control Frameworks

This myth came up during the audience Q&A portion of the panel, where one audience member from the financial industry wondered at how possible it would be to institute DevOps in a highly regulated industry. According to Mortman, as seemingly chaotic and fast-and-furious as the DevOps pace can seem to the uninitiated, it is actually far more process-oriented than people realize.

“People say ‘Oh, there’s no process to this.’ But it is actually very process oriented,” he said. “It’s just not very bureaucracy-oriented. There’s not a lot of paperwork. In its very guts, it is very service-oriented and very delivery-oriented.”

As he puts it, every compliance framework in the planet says you need to log, audit and approve changes. But none of them stipulate a workflow where each change has to be approved one at a time.

“The workflow can be ‘Please approve the following list of actions that I can do without notifying you first,’” he explained, “and as long as you have an audit trail showing that that has been done properly, I have yet to meet an auditor who will not buy into that.”

Myth #3: DevOps Gives Developers Too Much Power

Many dyed-in-the-wool security pros fear DevOps because they believe it is a cowboy environment that gives developers too much power.

“I’ll be honest, when i first started observing DevOps patterns, developers doing their own deploys seemed immoral to me; irresponsible, reckless, just shouldn’t be done,” Kim said. “No sane person would want to be associated with that, right?”

The thing was, after Kim dove in and also did a benchmark study with Tripwire several years ago across 4,000 organizations, he found that when comparing groups where ops did the deploys to those where developers shepherded code through deploys, change success rates and mean time to repair were not adversely affected, and in fact, “it is often considered indispensible that developers take code all the way through the deployment pipeline and you know they are getting great outcomes.”

And from a security controls perspective, the truth is that in a DevOps situation developers “are not logging individually into each server and SSH’ing the code up, untarring it and installing it,” Mortman said. Instead these actions are being done by automated systems.

“They’re actually going to give you much better timestamps, much more accuracy, and more consistent results and visibility than a manual process being done by someone in ops,” Mortman said. “So, actually, you’re getting better change management out of that system than you are in other situations.”

Myth #4: DevOps Cuts Important Testing

One discussion point also brought up during audience Q&A was the possibility that DevOps would somehow excise important testing functionality from the IT process. The panel quickly put that one to bed.

“You can do bad testing in any model,” Corman said. “If DevOps starts eliminating any kind of testing, its a bad DevOps program. I think that’s orthogonal to some of the benefits or opportunities of DevOps.”

Kim pointed to Google as an example of how testing fits into the DevOps methodology.

“So, Google does about 5,500 code commits per day, they have about 15,000 software engineers, and 75 million hours of testing is run each day,” he said. “You can’t get that massive amount of deployment frequencies without a tremendous amount of testing.”

March 20, 2014
Partnership Between Tufin, Puppet Labs Indicates Need for Security Baked Into DevOps

A recent partnership between the IT automation developers at Puppet Labs and the security policy orchestration gurus at Tufin Technologies could start setting the pace for how orchestration of security policies can more easily be baked into DevOps practices. Announced at last month’s RSA Conference, the partnership yielded an integration between Puppet Enterprise and Tufin Orchestration Suite. This will make it possible for joint customers to streamline the configuration and provisioning of security policy changes to iptables, a host-based firewall commonly found on physical and virtual Linux servers.

“We expect Security Policy Orchestration to become a core requirement for our customers,” says Nigel Kersten, CIO at Puppet Labs. “Integrating with Tufin enables our customers to quickly and effectively address connectivity requirements across the enterprise without compromising security.”

The integration is an indication of the market’s need for better tools to help DevOps shops speed up the pace of application and network changes while maintaining a strong security and compliance posture. Today’s firewall and security policy changes are often handled in a manual, error-prone process—even at some of the largest enterprises. A recent survey commissioned by Tufin found that the vast majority of IT professionals reported having to correct 20 percent to 60 percent of security policy changes in their organizations after the fact.

“This is incredibly useful to our customers and ensures that security is woven into the server’s connectivity requirements,” says Reuven Harrison, CTO of Tufin. “As the convergence of security and network operations continues, this module enables our customers to manage change with automation that delivers tangible business value.”

According to Harrison, firewalls play a dual role in network operations, both providing security through network segmentation and connecting applications to the network across multiple touch points.

“It’s the latter that makes them relevant to DevOps folks,” he says. “Firewalls see the both application and the network, which makes them an ideal source of information for modeling applications. Load-balancers are also good for this.”

This is why the industry is starting see a convergence between these two technologies, and also why Tufin has pushed for an evolution from firewall management into the sphere of security policy orchestration.

“Security policy orchestration aids with use cases for firewalls and other network infrastructure as they relate to applications,” Harrison says. “It ensures security policy changes are automated and account for security and compliance considerations not just for network access but also for application connectivity.”

Harrison believes that while security may not necessarily be the chief selling point for evolving to DevOps practices at some firms, the movement could present opportunities to more closely fold security into a more rapid application release cycle. He believes it will be an imperative in complex cloud environments.

“As cloud infrastructure matures and gains acceptance from larger and more traditional companies, security will have to be baked into the application release process,” he says. “That means securing not just the application code but also the infrastructure on which it resides.”

As things stand, Harrison believes that security is typically split into two distinct layers. There’s the application security layer, typically dominated by code security and penetration testing and then network security that is preoccupied by zone segmentation, encryption, authentication and the like.

“Eventually these two layers, along with some emerging technologies such as virtual networks, will converge,” he says, “and DevOps will handle it end-to-end.”

March 19, 2014
Perforce: The biggest little secret in DevOps
Quick can you name a company that is 15 years old, never took a dime of venture money, has over 250 employees, over 10,000 customers and was widely profitable from day one? Until last week me neither. That was when I was briefed by Dhruv Gupta, Director, Product Marketing and Bob Dever, Director, Corporate Marketing for Perforce.

Perforce is a DevOps powerhouse. Take a look at the customer cloud to the right to get an idea of who some of the 10,000+ customers they have are. Perforce is all about version control. They do that and do it well. Whether you are talking about version control of your code, documents or analytics, Perforce has a solution for you.

When you think about it, you can see why Perforce is such a DevOps play. DevOps is all about continuous delivery or at least rapid release. How can you accelerate your releases without keeping track of the different versions? Without version control you quickly have a mess. Perforce has in some ways enabled continuous delivery and rapid iteration. They truly are the biggest, little secret in DevOps.

For many years Perforce was really the only choice in town for version control (there were others, but it became a leader in the space pretty quickly) but the rise of GIT, a distributed version control system (DVCS) gave some an alternative. But GIT lacked the enterprise features (read security and team management) that larger organizations needed. Its popularity among lone wolf and smaller dev teams though was unmistakable. It is easy.

Perforce has recognized this and has just released a new hybrid version of their solution. This hybrid version allows developers and dev team managers to use GIT and Perforce’s P4 in one hybrid version control platform. A best of both world solutions give DevOps teams the ability to move fast with GIT and have the security and enterprise functionality that P4 allows.

When you start thinking about DevOps Unicorns like Netflix for instance or Salesforce, the sheer magnitude of what Perforce is managing comes into clarity. In Salesforce for instance you have 3,000 Engineers making 4,000 – 9,000 Code Changes a day with 550,000 Tests / Hour. Keeping versions straight is not the kind of thing you do in a spread sheet.

It in a Continuous Delivery environment where Perforce really shines. They are making a big bet on enterprises adopting the Continuous Delivery model. However there are some fundamental questions to answer first. According to Perforce though you need to define what does Continuous Delivery mean to you? They conducted a survey:

According to Perforce they think of Continuous Delivery as this:
- Deliver working product to users as quickly as possible
- Every change (check-in) leads to a potential release
- Give business the option to release – what, when, to whom
- A change in process and culture
Most importantly though, Continuous Delivery is not just something Perforce thinks is crucial. Waterfall release models are obsolete and even Agile is just a way stop towards Continuous.

According to Perforce’s own survey they see Continous Delivery attaining critical mass:

Is this a case of when you are a hammer everything is a nail or are we at the dawn of the age of Continuous Delivery? Perforce is making a bet that it is. They are already very successful, any more and they may just lose their status as the biggest, little secret in DevOps.
March 18, 2014
DevOps: An Opportunity to Build a Better Resiliency

Security professionals like to check code for security-related defects before the code is delivered. When they start hearing about multiple builds a day, or even a few builds a week, and a radical decrease in development time along with operational changes, they are bound to give pause. After all, most are having a hard time keeping up even now.

The industry is moving in this direction. An Evans Research survey of more than 600 software development professionals in the UK and US found that 51% of respondents have already started the practice of continuous delivery across a number of teams. The same survey found that both culture and technology are the primary stumbling blocks, with 53% believing that they have the technology in place for continuous delivery and 59% believing that their staff is ready.

However, those that do successfully make the move to continuous delivery and embrace DevOps find that the real impact on security will be much brighter than the naysayers predict, according to those interviewed. In fact, it is automation that underpins continuous delivery and much of the DevOps movement, when done properly, goes a long way to enhance security and system resiliency.

Security often gets blamed for slowing down the development process, explains Andrew Storms, senior director of DevOps at CloudPassage. But the reality is that it slows it down no more than QA and other development checks. “Those who are at the end of the chain are typically the ones who get the fingers pointed at them. They are the ones who are blamed for slowing down time to market, or delays in getting product in front of the customers,” says Storms “DevOps is an opportunity to integrate all of that together.”

“One of the interesting things about moving to DevOps is that it requires a very high level of automation,” says David Mortman, chief security architect at Dell Enstratius and a contributing analyst at security research firm Securosis. “Part of the whole continuous deployment model is the move to continuous building and continuous integration. So you need to start writing security-oriented tests that can be automated against that code; when you are running your regression, integration, and other tests, you are testing for security states as well,” Mortman says.

Jeff Sussna, founder of the IT service innovation consultancy Ingineering.IT, agrees “In continuous delivery and DevOps you start thinking, from a nonfunctional requirements perspective, about continuous integration—and that’s about finding software problems sooner, when they’re easier to fix. That’s what security professionals have been fighting for, for years,” Sussna says.

That certainly makes DevOps an easier sell to information security teams. “When you suggest to security teams that security scans be completed with every build so that problems are found sooner, my experience is that security folks, who don’t seem to smile a lot, suddenly start to smile,” Sussna says..

It’s not just talk, “it works,” says Sussna. “When we run those kinds of scans, we find problems earlier, and it’s easier to fix them. When it comes to running through the traditional gates that security puts up, it’s a lot less painful because they tend to run clean within the processes.”

Another, and perhaps much more subtle, yet deep, benefit, is how continuous security testing keeps security at the forefront of developers’ minds. “Even when a build comes back clean, and the notification says, ‘Security violations, zero,’ it puts in their minds—daily—that security is something they need to think about it. And it’s not confrontational; DevOps puts it within the development process because the developers are seeing the feedback from their reporting tools, and so they are getting instant feedback that they need to fix security issues. Therefore, the feedback isn’t a nag from QA or information security; it’s coming from their systems,” .

The benefits of DevOps can run still deeper. “Forget the deployment. Forget the automation. Just having this culture of sharing and working together that DevOps fosters, as opposed to being combative, helps to drive excellence,” says Mortman.

March 18, 2014
The DevOps Journey
Maintaining a constant thirst for learning is essential for anyone dedicating their career to the IT, computer and networking industries. It’s more than just learning about the latest technologies, rather, it’s imperative we embrace new methodologies and paradigms for doing work, thinking about problems and creating solutions. Enter DevOps, which is a healthy mix of all the above.

You might find things a bit confusing if you’re new to DevOps. There are many perspectives on what it means. For myself, there were two significant milestones on my journey to understand and embrace DevOps; 1) rethinking how we design large, dynamic applications and manage the underlying infrastructure in cloud environments such as Amazon AWS, and 2) reading the book The Phoenix Project by Gene Kim, Kevin Behr, and George Spafford.

At first blush, I thought DevOps suffered the “definition bloat” of so many other industry buzzwords (for example, “cloud” comes to mind.) Depending on who you talked to or what you read, DevOps definitions frequently encompassed one or more of the following:
1. embedding development resources within business units for better alignment and reduced cycle time,
2. high frequency software releases into production (multiple per day, hour, etc.)
3. automating infrastructure and system administration tasks vs manual provisioning and sysadmin
4. embedding operations functions within development organizations
5. operations performed by developer, even doing away with traditional operations in some cases
If these seem like a somewhat confusing, loosely assembled list is ideas, don’t fret — they struck me the same way at first too. These ideas actually fit very well together around a few core ideas essential to DevOps.

Quicker Time To Value – Being Fast and Agile

DevOps is about changing traditional paradigms, enabling us to more rapidly deliver production updates, new capabilities and improved operations. Outcomes include reducing software releases from months to multiple, micro releases per day, and even hours. To do this requires much more than can be achieved through traditional operational efficiencies. It requires tight alignment, coordination and execution by product, development and operations functions. Embedding multiple disciplines together helps gain this alignment, shorten communication paths, setting priorities quicker, enabling lightning fast coordination. DevOps promotes new practices like embedding development and operations functions within other parts of the business, breaking down long standing barriers between IT / Ops and development / business groups.

Automated Provisioning, Operations and SysAdmin

From app development to infrastructure operations, DevOps is about automation. DevOps is about faster software testing and integration, quality improvement, and working in much more dynamic and rapidly changing operating environments. For example, rather than building system images by hand (stick built), automation is used to rapidly generate and spin up new virtual environments on a small or large scale. Systems wake up, check in to learn their personality, install, configure or update code (OS, database, web server, application code revs, etc) and join the network dynamically. This can occur with a single instance or with hundreds simultaneously. Initial upgrades and feature updates can occur on smaller or isolated sections of the network, then are rapidly rolled out across the network, and rolled back quickly if necessary. Sysadmins, be they developers or ops people, use scripts, code and automation tools to manage much larger environments than would ever be possible using manual methods. Automation is an essential differentiator in any DevOps environment.

Culture of Open Collaboration, Communication and Failing Fast

DevOps breaks down silos and places a premium on collaboration and communications amongst all players involved. It’s no longer us vs. them or this department vs. that other department. The blame game is gone and the focus is on making work happen, and fast. Communication lines are drastically shortened and conflicts in chains of command are collapsed. It’s about how “we” get things done, no longer about what “they” didn’t do. Failures are viewed much differently. Failures happen and from them we learn and develop newer, better, more innovative solutions to solving problems and prevention problems in the future. Because of the speed DevOps can bring, recovery from failure is much quicker. If a new release into production is causing issues, we quickly flip back to a previous known-good version running on other virtual instances. Updates can also be rolled out much more quickly.

I hope these and coming views on DevOps are valuable to you as a reader. I know our conversation will help me and the teams I work with, as we learn and grow our expertise in DevOps. Please share your ideas, disagreements and learnings as we move along this journey together into the fast moving and quickly changing world of DevOps.
March 16, 2014
A Q & A with Gene Kim: DevOps and What Separates High Performing Enterprises

We recently caught up with Gene Kim (@realgenekim) as he was wrapping up the final touches of a draft of the upcoming DevOps Cookbook. We wanted to get his thoughts on what causes IT deployment failure, the DevOp movement, and finally what separates high performing IT organizations from low performing ones. If you’re following DevOps, no doubt you are familiar with Gene. In case you are not, Gene is an award winning CTO, researcher and author. He founded the security firm Tripwire, and was the CTO there for 13 years. To date, he’s written three books, including The Visible Ops Handbook and The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win. Gene is a huge fan of IT operations, and how it can enable developers to maximize throughput of features from “code complete” to “in production,” without causing chaos and disruption to the IT environment. Now, let’s jump into our discussion.

devopsy.kinsta.cloud: When organizations decide they want to move to DevOps, or continuous delivery, what is a barrier that they immediately run into? And is DevOps something all types of companies can embrace?

Gene Kim: In my mind it’s the unicorns (leading web app companies) that are sort of showing the way out. Just to show why I’m so interested in this, you know I’ve been studying high performing IT organizations since 1999. What I’ve noticed is that there is this downward spiral that happens in almost every IT organization. It led to more fragile application in production, longer deployment times, building up of technical debts, and the business goes slower and slower. I think one of the reasons that I cared so much about is that it led to preordained failure. Where people felt, especially downstream (operations, test, security) trapped in a system where we were powerless to change the outcomes. No matter what we did, failure has been preordained. I think the way out of the downward spiral is what organizations like Google, Amazon, Twitter, LinkedIn, GitHub, are doing. These organizations are doing tens, hundreds, or even thousands of deploys a day. In a world where most of us are stuck with one deploy every nine months with catastrophic outcomes. I think that the first objection is that we’re not an “Amazon,” we’re not a “Google,’ we’re a bank, we’re telco, we’re a retailer. That it doesn’t apply to them. However, it turns out if you look at who’s adopting DevOps it’s Nordstrom, Walmart, General Motors, SAP. One of the statistics that I saw was, that Google has 15,000 engineers, or developers. HSBC, the European bank, has 15,000 developers. I think every company is an IT company regardless of what business we think we are in. I think it just shows that anything we try to get down relies on IT. So for DevOps not to be a core competency is absurd.

devopsy.kinsta.cloud: And even increasingly software has to become a core competency, as more devices become API-enabled? All companies are software companies now.

Gene Kim: I love that Jeff Immelt, CEO of GE, quote: “Our biggest fear is being disintermediated by software. So I’m going back to school on software and big data.” I imagine that they run into that same objection internally. “Hey, we’re not Netflix, we’re not Amazon.” Right? They are trying to sell the idea of moving to DevOps or becoming more agile. Anyone can do it and benefit from it. Even in general conversations that was the first objection that always came up. Well that’s good for so and so, but we’re not so and so. I think that makes it a belief issue. So that’s what we’re talking about here, right? It can’t work here. I think it a serious of problems and it’s very predictable what the bottlenecks or obstacles are. The first one is that it takes too long to do the deployments, actually you can go a step before that. One of the biggest problems is that operations is always in the way. Whenever a dev or test wants a test environment it’s not uncommon in typical enterprises, that it takes 40 weeks to get one. Back in my Tripwire days, I use to think that developers didn’t like testing the code, because that’s what I saw. The truth is even worse than that; the developers really want to test their code. Their problem is they can never get an environment when they want one or they need one. So the survival mechanism kicks in, they’ll always go to the guy they know that’s really good at getting an environment, Their Corporal Klinger or Nash. They don’t want to ask any questions about where it came from, because you know someone is missing a server somewhere. This person probably just stole an environment from another project. With virtualization and infrastructure code there’s no almost no excuse not to be able to create environments available on demand. One of the things that we did, with Puppet Labs, is we reached out to me to help design their DevOps benchmark. This was awesome, when I found out about it I jumped out of my chair because, I saw this as a logical continuation of ten years of benchmarking we had done.

devopsy.kinsta.cloud: What have you learned that separates high-performers and makes DevOps possible?

Gene Kim: There were three findings about it that blew me away. The first one is the high performers were massively outperforming their non-high performing peers. Thirty times more frequent code deployments. They were able to complete the deployments from code committed to running a production 8,000 times faster. In other words they could do a deployment measured in minutes or hours. Whereas low performers it took them weeks, months, or quarters. Furthermore, they actually had better outcomes. Twice the change success rate and when they caused outages they could fix it 12 times faster. There were 4,200 that completely responded and it just showed that you could be more agile and reliable at the same time. I think if we just connect some dots, they’re also going to be more secure, they’re probably going to have a faster find fix cycle time. We didn’t actually test that but, if that were true it would mirror what we found in the previous benchmarking work I had done. So that’s surprise number one the extent that they were out performing their non-high performing peers. Second surprise, we found two behaviors that were present in every high performer but they were universally absent in every non-high performer. The first one was that they are they checking in all their production artifacts and version control. Meaning, they could reproduce a production environment based off of what was in version control. Versus only it being an indescribable asset that only one person understood, if they knew about it at all. The second one was is there an automated code deployment process? Which includes being able to create environments available on demand. The next one is the code deployment process. I think it’s very typical in most enterprises that whenever you want to do a new release or deploy it takes days or weeks to complete. Involving tons of hand-offs from the developers, DBA, release team, back to the developers to the security people, and the firewall change and so forth. All that creates delays and is error prone and so forth. In the high performer there usually a one click to deploy or an automatic deploy. If you look at Google, Amazon, Twitter, Facebook, in general they have a continuous deployment process so that when a developer checks in code it automatically runs all the tests. Then it puts it into an integration test environment and runs all the tests there. There might be some manual work, but then in general it automatically migrates to production. Clearly, that’s what’s required to do multiple hundred of deploys a day.

devopsy.kinsta.cloud: That is incredible to learn, Gene. A lot of fascinating findings. Thanks so much for your time.

Gene Kim: Thank you, George. Always a pleasure speaking with you.

March 16, 2014