devopsy.kinsta.cloud

Category: Features

  • What “Programmable” Means

    What “Programmable” Means

    One of the most often used words these days to describe emerging architectures and technology is (drum roll please) “programmable.”

    Now, from my used-to-be-a-developer-and-architect-turned-infrastructure-geek perspective, I know what that means. It means I get to write code. But to what purpose that code is put to is a question that’s rarely answered. So today we’re going to answer it because, well, my used-to-be-a-developer-and-architect-turned-infrastructure-geek persona likes to talk about code.

    When you hear the term programmable used in the context of Software-Defined Networks (SDN) they’re talking about the same things that “programmable” means in the context of Devops: programmatic control of both the data and the control path.

    The easiest way to visualize the difference between these impressive sounding terms is to recognize that the data path is inside and the control path is outside. 

    programmabilty-illustratedWhat that means is if you’re interacting with the data path, you are participating in it. You can inspect it, evaluate it, and transform it and, perhaps more importantly, you can make decisions based on data in flight. That’s powerful for the network, because a variety of Devops patterns are based on this very concept and require the network for implementation. Consider, for example, the implementation of an A/B testing pattern. The basic premise of the pattern is that somehow the identity of a user is known and, based on their membership in a defined group (beta testing customers, roles within the organization, volunteers from the community, etc…) they are routed to different versions of the same application to determine how differences in the UI impact behavior.

    Notice the term routing, which implies it is a function of the network. The reason programmability is critical is two-fold: one, you must be able to identify the user as belonging to a specific group and two, you must be able to dynamically route based on that information. The former, identification, could ostensibly be accomplished using fixed lists IFF (If and Only If) it is possible to map a user to some variable readily accessible to the network, which traditionally means things like IP and MAC addresses and TCP ports. Most folks will immediately recognize that today this isn’t enough (and actually it hasn’t been enough since the days of the mega-proxy, but I digress). The information that’s needed is much higher in the networking stack at the application layer, aka in the OSI model as Layer 7. Which means you’re probably going to have to dig into HTTP headers and maybe the payload to get the data you want. That requires programmability in the data path, inside. 

    To really make the pattern scale (and to make it flexible enough to support a phased approach to beta testing) you probably don’t want group membership  to be a static, fixed thing. So now you’re going to want to extend your implementation to include making a call to a database or identity store (LDAP, AD, etc…) to get current information. That requires programmability in the data path, inside. 

    So what programmability in the data path means is the ability to use code to direct the behavior of some “thing” in the network, be it an application proxy or a load balancer or what have you. You can’t easily do this through configuration-based proxies because they don’t allow for the execution of the domain logic  necessary. There are a limited number of interactions you can achieve with configuration-based systems because of the logic they’re hard-coded to execute,and most of them don’t include calling out to a data store to look-up arbitrary information.

    Data path programmability = Code in Flight. So you now you know where the name of the blog came from. This developer-and-architect-turned-infrastructure-geek finds the intersection between programmability and networking irresistible.

    Okay, so that’s data path programmability. What about control path programmability?

    Well, that’s the easy part. Control path programmability is on the outside. It’s the APIs that enable automation and orchestration of provisioning and configuration. Control path programmability is how you integrate infrastructure into command control systems like OpenStack, Puppet and Chef. Infrastructure comes with varying degrees of API-enablement, some extremely granular and others less so, depending on the particular function and maturity of the vendor.

    So what you can do with the control path depends on the API, but in general it enables things like provisioning new services – think configuring a new virtual server in Apache and all the settings that accompany it – or start up and shut down functions that enable rolling maintenance across a set of devices or servers to avoid downtime.

    But all control path programmability is on the outside, enabling integration and management without having to learn a CLI or configuration file syntax.

    SDN and Devops use both programmability to reach relatively the same goals: scale of operations, rapid provisioning of services, to decrease the possibility of human error (the fat finger factor) and to introduce flexibility into systems that were previously bound to fixed processing and configuration-based paradigms.

    Programmability in the network is going to enable a variety of new patterns, processes and services and as traditional infrastructure becomes more and more programmable through the adoption of architectural approaches like SDN, Devops is going to see more and more opportunity to apply its maturing expertise in leveraging programmability to a much wider audience.

     

  • Don’t let automation be the tail wagging the dog

    Don’t let automation be the tail wagging the dog

    There is quite a bit of talk right now about DevOps and how important automation is.  This could be compared to GM in the 1980s when the company tried to transform the way they built cars through automation and robot.  In the early 1980’s GM spent $90B dollars in an attempt to speed up the way cars were delivered to the market.  This failed with famous stories of robots painting each other and others welding doors shut.  Ultimately this project failed and the robots were scraped.  It was later reported that the money lost on the project could have been spent to acquire both Toyota and Nissan at the time.

    Automation dominates the conversation when it comes to DevOps.  If you do not learn how to build cookbooks you are not doing DevOps.  I fear this is going to set companies trying to transform IT off on the wrong track, focused on the outcome and not on the scientific process to get there.    Alan Sharp-Paul recently wrote that you cannot automate what you don’t understand.  The best way to understand it is in two key ways, change and performance.

    Let Science set you free

    This was the secret that Toyota used to dominate the automobile industry.  They were able to transform all of their employees into little scientist.  Constantly tinkering to squeeze our inefficiencies in the system.  Every worker was empowered to think how things could be done better.

    At the heart of the agile and DevOps processes is the practical application of the scientific method.  Think back to your 8th grade science fairs.  No not the volcanos but the ones that won best of show.  These had a clear hypothesis that stated what they were going to try and prove.  They were able to understand what they were going to change and measure the difference between the before and after.  They could clearly state what changed and if it was a successful experiment or a failure.

    This thinking is key for companies trying to transform to a DevOps IT organization.  If a company does not understand the experiments that are running and what the outcomes should be then automation will just make the problem worse. That is why change management and performance management are foundational for transforming to a DevOps culture.

    Tracking your experiments

    Any good scientist knows exactly what he is working on.  He knows every experiment that is running anticipating the outcome.  He knows all the details about each change that has been made to see how they impact the outcome.  This is the same way that IT should be tracking every change happening.

    Every change done in IT is a scientific experiment running. Tracking every change allows for understanding of the outcome of changes within the application.  Developers and Operations will be more informed about the cause and effect of their changes.  There are multiple different ways that you can track changes but the key is that you are doing it very well before you start to apply automation.

    Measuring the outcome

    There is no need to track the experiments if there are not being measured.  A scientist is very deliberate in how they measure the outcomes.  Good enough just does not get this done.  This is beyond just monitoring.  Monitoring in a lot of cases is after the fact, in production.  This is about measuring the difference for every experiment regardless of where it is being tests.  This creates the need to test changes all the way to when the first functional tests are done, measuring the changes of every code update.  Following it through all of testing and to production.  Understanding how each experiment impacts the system as a whole.

    Efficiency through automation  

    Once there is good understanding of all the experiments and a standard platform for measuring the outcome then automation starts to return exponential benefit.  Because every experiment it known the lower priority ones can be identified and automated.  Risk is better assessed.  Processes can start to speed up because there is visibility and accountability.  Applications are no longer classified as high risk or low risk only change is classified that way.  Determining what can be automated just becomes another experiment that is tried and measured for success and and failure.  Automation becomes just another outcome of DevOps to the business and not the route to achieve it.

  • Busting The 4 Biggest DevOps Security Myths

    Busting The 4 Biggest DevOps Security Myths

    Last month when the IT security world convened in San Francisco for the annual RSA confab, a number of security experts tackled DevOps in talks that covered the crossover between continuous improvement and risk management. Among them was a lively panel mythbusting some of the most prevalent misconceptions about DevOps held by the security community.

    Moderated by Dwayne Melancon of Tripwire, the panel was made up of Josh Corman, CTO of Sonatype, Nick Galbreath, vice president of engineering at IPONWEB, Gene Kim, author of The Phoenix Project, and David Mortman chief security architect and distinguished engineer for Dell Enstratius, all of them security professionals and very active DevOps evangelists. The banter backed-and-forthed between the participants, with four myths bubbling to the surface as particular highlights from the discussion.

    Myth #1: DevOps Pace Will Leave Security In The Dust

    One of the most prevalent myths about DevOps within the security community is that the rapid deploys of the approach will throw all risk management precautions to the wind and leave security practices behind as a result. Quite the contrary, panelists argued. They believe that DevOps patterns make it more possible than ever to bake security practices into IT processes more than before, when it is frequently bolted on “at the end of the caboose,” as Galbreath puts it.

    “You’re not throwing out your current security system,” he said. “What you’re doing is removing a lot of the junk work that eats up so much time and keeps people underwater. You’re really enabling different teams to all improve security together.”

    As Corman explained, security people say all the time that complexity is the enemy of security.

    “It also happens to be the enemy of stability,” he said. “That is the reason DevOps people come willingly with open arms to the security community.”

    Not to mention, Kim added, “because you can’t be highly available and have stuff coming continuously through the pipeline if it is not secure.

    Myth #2: DevOps Destroys Compliance And Control Frameworks

    This myth came up during the audience Q&A portion of the panel, where one audience member from the financial industry wondered at how possible it would be to institute DevOps in a highly regulated industry. According to Mortman, as seemingly chaotic and fast-and-furious as the DevOps pace can seem to the uninitiated, it is actually far more process-oriented than people realize.

    “People say ‘Oh, there’s no process to this.’ But it is actually very process oriented,” he said. “It’s just not very bureaucracy-oriented. There’s not a lot of paperwork. In its very guts, it is very service-oriented and very delivery-oriented.”

    As he puts it, every compliance framework in the planet says you need to log, audit and approve changes. But none of them stipulate a workflow where each change has to be approved one at a time.

    “The workflow can be ‘Please approve the following list of actions that I can do without notifying you first,’” he explained, “and as long as you have an audit trail showing that that has been done properly, I have yet to meet an auditor who will not buy into that.”

    Myth #3: DevOps Gives Developers Too Much Power

    Many dyed-in-the-wool security pros fear DevOps because they believe it is a cowboy environment that gives developers too much power.

    “I’ll be honest, when i first started observing DevOps patterns, developers doing their own deploys seemed immoral to me; irresponsible, reckless, just shouldn’t be done,” Kim said. “No sane person would want to be associated with that, right?”

    The thing was, after Kim dove in and also did a benchmark study with Tripwire several years ago across 4,000 organizations, he found that when comparing groups where ops did the deploys to those where developers shepherded code through deploys, change success rates and mean time to repair were not adversely affected, and in fact, “it is often considered indispensible that developers take code all the way through the deployment pipeline and you know they are getting great outcomes.”

    And from a security controls perspective, the truth is that in a DevOps situation developers “are not logging individually into each server and SSH’ing the code up, untarring it and installing it,” Mortman said. Instead these actions are being done by automated systems.

    “They’re actually going to give you much better timestamps, much more accuracy, and more consistent results and visibility than a manual process being done by someone in ops,” Mortman said. “So, actually, you’re getting better change management out of that system than you are in other situations.”

    Myth #4: DevOps Cuts Important Testing

    One discussion point also brought up during audience Q&A was the possibility that DevOps would somehow excise important testing functionality from the IT process. The panel quickly put that one to bed.

    “You can do bad testing in any model,” Corman said. “If DevOps starts eliminating any kind of testing, its a bad DevOps program. I think that’s orthogonal to some of the benefits or opportunities of DevOps.”

    Kim pointed to Google as an example of how testing fits into the DevOps methodology.

    “So, Google does about 5,500 code commits per day, they have about 15,000 software engineers, and 75 million hours of testing is run each day,” he said. “You can’t get that massive amount of deployment frequencies without a tremendous amount of testing.”

  • Partnership Between Tufin, Puppet Labs Indicates Need for Security Baked Into DevOps

    Partnership Between Tufin, Puppet Labs Indicates Need for Security Baked Into DevOps

    A recent partnership between the IT automation developers at Puppet Labs and the security policy orchestration gurus at Tufin Technologies could start setting the pace for how orchestration of security policies can more easily be baked into DevOps practices. Announced at last month’s RSA Conference, the partnership yielded an integration between Puppet Enterprise and Tufin Orchestration Suite. This will make it possible for joint customers to streamline the configuration and provisioning of security policy changes to iptables, a host-based firewall commonly found on physical and virtual Linux servers.

    “We expect Security Policy Orchestration to become a core requirement for our customers,” says Nigel Kersten, CIO at Puppet Labs. “Integrating with Tufin enables our customers to quickly and effectively address connectivity requirements across the enterprise without compromising security.”

    The integration is an indication of the market’s need for better tools to help DevOps shops speed up the pace of application and network changes while maintaining a strong security and compliance posture. Today’s firewall and security policy changes are often handled in a manual, error-prone process—even at some of the largest enterprises.  A recent survey commissioned by Tufin found that the vast majority of IT professionals reported having to correct 20 percent to 60 percent of security policy changes in their organizations after the fact.

    “This is incredibly useful to our customers and ensures that security is woven into the server’s connectivity requirements,” says Reuven Harrison, CTO of Tufin. “As the convergence of security and network operations continues, this module enables our customers to manage change with automation that delivers tangible business value.”

    According to Harrison, firewalls play a dual role in network operations, both providing security through network segmentation and connecting applications to the network across multiple touch points.

    “It’s the latter that makes them relevant to DevOps folks,” he says. “Firewalls see the both application and the network, which makes them an ideal source of information for modeling applications.  Load-balancers are also good for this.”

    This is why the industry is starting see a convergence between these two technologies, and also why Tufin has pushed for an evolution from firewall management into the sphere of security policy orchestration.

    “Security policy orchestration aids with use cases for firewalls and other network infrastructure as they relate to applications,” Harrison says. “It ensures security policy changes are automated and account for security and compliance considerations not just for network access but also for application connectivity.”

    Harrison believes that while security may not necessarily be the chief selling point for evolving to DevOps practices at some firms, the movement could present opportunities to more closely fold security into a more rapid application release cycle. He believes it will be an imperative in complex cloud environments.

    “As cloud infrastructure matures and gains acceptance from larger and more traditional companies, security will have to be baked into the application release process,” he says. “That means securing not just the application code but also the infrastructure on which it resides.”

    As things stand, Harrison believes that security is typically split into two distinct layers. There’s the application security layer, typically dominated by code security and penetration testing and then network security that is preoccupied by zone segmentation, encryption, authentication and the like.

    “Eventually these two layers, along with some emerging technologies such as virtual networks, will converge,” he says, “and DevOps will handle it end-to-end.”

  • Perforce: The biggest little secret in DevOps

    Perforce: The biggest little secret in DevOps

    Quick can you name a company that is 15 years old, never took a dime of venture money, has over 250 employees, over 10,000 customers and was widely profitable from day one? Until last week me neither. That was when I  was briefed by Dhruv Gupta, Director, Product Marketing and Bob Dever, Director, Corporate Marketing for Perforce.

    perforce customersPerforce is a DevOps powerhouse. Take a look at the customer cloud to the right to get an idea of who some of the 10,000+ customers they have are. Perforce is all about version control. They do that and do it well. Whether you are talking about version control of your code, documents or analytics, Perforce has a solution for you.

    When you think about it, you can see why Perforce is such a DevOps play.  DevOps is all about continuous delivery or at least rapid release.  How can you accelerate your releases without keeping track of the different versions? Without version control you quickly have a mess. Perforce has in some ways enabled continuous delivery and rapid iteration. They truly are the biggest, little secret in DevOps.

    For many years Perforce was really the only choice in town for version control (there were others, but it became a leader in the space pretty quickly) but the rise of GIT, a distributed version control system (DVCS) gave some an alternative. But GIT lacked the enterprise features (read security and team management) that larger organizations needed.  Its popularity among lone wolf and smaller dev teams though was unmistakable. It is easy.

    Perforce has recognized this and has just released a new hybrid version of their solution. This hybrid version allows developers and dev team managers to use GIT and Perforce’s P4 in one hybrid version control platform. A best of both world solutions give DevOps teams the ability to move fast with GIT and have the security and enterprise functionality that P4 allows.

    When you start thinking about DevOps Unicorns like Netflix for instance or Salesforce, the sheer magnitude of what Perforce is managing comes into clarity. In Salesforce for instance you have 3,000 Engineers making 4,000 – 9,000 Code Changes a day with 550,000 Tests / Hour. Keeping versions straight is not the kind of thing you do in a spread sheet.

    It in a Continuous Delivery environment where Perforce really shines. They are making a big bet on enterprises adopting the Continuous Delivery model. However there are some fundamental questions to answer first. According to Perforce though you need to define what does Continuous Delivery mean to you? They conducted a survey:

     perforce cd

     

    According to Perforce they think of Continuous Delivery as this:

    • Deliver working product to users as quickly as possible
    • Every change (check-in) leads to a potential release
    • Give business the option to release – what, when, to whom
    • A change in process and culture

     perforce cd pipeline

    Most importantly though, Continuous Delivery is not just something Perforce thinks is crucial. Waterfall release models are obsolete and even Agile is just a way stop towards Continuous.

     perforce cd arrow

     

    According to Perforce’s own survey they see Continous Delivery attaining critical mass:

     perforce cd adoption

     Is this a case of when you are a hammer everything is a nail or are we at the dawn of the age of Continuous Delivery? Perforce is making a bet that it is. They are already very successful, any more and they may just lose their status as the biggest, little secret in DevOps.

  • DevOps: An Opportunity to Build a Better Resiliency

    DevOps: An Opportunity to Build a Better Resiliency

    Security professionals like to check code for security-related defects before the code is delivered. When they start hearing about multiple builds a day, or even a few builds a week, and a radical decrease in development time along with operational changes, they are bound to give pause. After all, most are having a hard time keeping up even now.

    The industry is moving in this direction. An Evans Research survey of more than 600 software development professionals in the UK and US found that 51% of respondents have already started the practice of continuous delivery across a number of teams. The same survey found that both culture and technology are the primary stumbling blocks, with 53% believing that they have the technology in place for continuous delivery and 59% believing that their staff is ready.

    However, those that do successfully make the move to continuous delivery and embrace DevOps find that the real impact on security will be much brighter than the naysayers predict, according to those  interviewed. In fact, it is automation that underpins continuous delivery and much of the DevOps movement, when done properly, goes a long way to enhance security and system resiliency.

    Security often gets blamed for slowing down the development process, explains Andrew Storms, senior director of DevOps at CloudPassage. But the reality is that it slows it down no more than QA and other development checks. “Those who are at the end of the chain are typically the ones who get the fingers pointed at them. They are the ones who are blamed for slowing down time to market, or delays in getting product in front of the customers,” says Storms “DevOps is an opportunity to integrate all of that together.”

    “One of the interesting things about moving to DevOps is that it requires a very high level of automation,” says David Mortman, chief security architect at Dell Enstratius and a contributing analyst at security research firm Securosis. “Part of the whole continuous deployment model is the move to continuous building and continuous integration. So you need to start writing security-oriented tests that can be automated against that code; when you are running your regression, integration, and other tests, you are testing for security states as well,” Mortman says.

    Jeff Sussna, founder of the IT service innovation consultancy Ingineering.IT, agrees “In continuous delivery and DevOps you start thinking, from a nonfunctional requirements perspective, about continuous integration—and that’s about finding software problems sooner, when they’re easier to fix. That’s what security professionals have been fighting for, for years,” Sussna says.

    That certainly makes DevOps an easier sell to information security teams. “When you suggest to security teams that security scans be completed with every build so that problems are found sooner, my experience is that security folks, who don’t seem to smile a lot, suddenly start to smile,” Sussna says..

    It’s not just talk, “it works,” says Sussna. “When we run those kinds of scans, we find problems earlier, and it’s easier to fix them. When it comes to running through the traditional gates that security puts up, it’s a lot less painful because they tend to run clean within the processes.”

    Another, and perhaps much more subtle, yet deep, benefit, is how continuous security testing keeps security at the forefront of developers’ minds. “Even when a build comes back clean, and the notification says, ‘Security violations, zero,’ it puts in their minds—daily—that security is something they need to think about it. And it’s not confrontational; DevOps puts it within the development process because the developers are seeing the feedback from their reporting tools, and so they are getting instant feedback that they need to fix security issues. Therefore, the feedback isn’t a nag from QA or information security; it’s coming from their systems,” .

    The benefits of DevOps can run still deeper. “Forget the deployment. Forget the automation. Just having this culture of sharing and working together that DevOps fosters, as opposed to being combative, helps to drive excellence,” says Mortman.

  • The DevOps Journey

    The DevOps Journey

    Maintaining a constant thirst for learning is essential for anyone dedicating their career to the IT, computer and networking industries. It’s more than just learning about the latest technologies, rather, it’s imperative we embrace new methodologies and paradigms for doing work, thinking about problems and creating solutions. Enter DevOps, which is a healthy mix of all the above.

    You might find things a bit confusing if you’re new to DevOps. There are many perspectives on what it means. For myself, there were two significant milestones on my journey to understand and embrace DevOps;  1) rethinking how we design large, dynamic applications and manage the underlying infrastructure in cloud environments such as Amazon AWS, and 2) reading the book The Phoenix Project by Gene Kim, Kevin Behr, and George Spafford.

    At first blush, I thought DevOps suffered the “definition bloat” of so many other industry buzzwords (for example, “cloud” comes to mind.) Depending on who you talked to or what you read, DevOps definitions frequently encompassed one or more of the following:

    1. embedding development resources within business units for better alignment and reduced cycle time,
    2. high frequency software releases into production (multiple per day, hour, etc.)
    3. automating infrastructure and system administration tasks vs manual provisioning and sysadmin
    4. embedding operations functions within development organizations
    5. operations performed by developer, even doing away with traditional operations in some cases

    If these seem like a somewhat confusing, loosely assembled list is ideas, don’t fret — they struck me the same way at first too. These ideas actually fit very well together around a few core ideas essential to DevOps.

    Quicker Time To Value – Being Fast and Agile

    DevOps is about changing traditional paradigms, enabling us to more rapidly deliver production updates, new capabilities and improved operations. Outcomes include reducing software releases from months to multiple, micro releases per day, and even hours. To do this requires much more than can be achieved through traditional operational efficiencies. It requires tight alignment, coordination and execution by product, development and operations functions. Embedding multiple disciplines together helps gain this alignment, shorten communication paths, setting priorities quicker, enabling lightning fast coordination. DevOps promotes new practices like embedding development and operations functions within other parts of the business, breaking down long standing barriers between IT / Ops and development / business groups.

    Automated Provisioning, Operations and SysAdmin

    From app development to infrastructure operations, DevOps is about automation. DevOps is about faster software testing and integration, quality improvement, and working in much more dynamic and rapidly changing operating environments. For example, rather than building system images by hand (stick built), automation is used to rapidly generate and spin up new virtual environments on a small or large scale. Systems wake up, check in to learn their personality, install, configure or update code (OS, database, web server, application code revs, etc) and join the network dynamically. This can occur with a single instance or with hundreds simultaneously. Initial upgrades and feature updates can occur on smaller or isolated sections of the network, then are rapidly rolled out across the network, and rolled back quickly if necessary. Sysadmins, be they developers or ops people, use scripts, code and automation tools to manage much larger environments than would ever be possible using manual methods. Automation is an essential differentiator in any DevOps environment.

    Culture of Open Collaboration, Communication and Failing Fast

    DevOps breaks down silos and places a premium on collaboration and communications amongst all players involved. It’s no longer us vs. them or this department vs. that other department. The blame game is gone and the focus is on making work happen, and fast. Communication lines are drastically shortened and conflicts in chains of command are collapsed. It’s about how “we” get things done, no longer about what “they” didn’t do. Failures are viewed much differently. Failures happen and from them we learn and develop newer, better, more innovative solutions to solving problems and prevention problems in the future. Because of the speed DevOps can bring, recovery from failure is much quicker. If a new release into production is causing issues, we quickly flip back to a previous known-good version running on other virtual instances. Updates can also be rolled out much more quickly.

    I hope these and coming views on DevOps are valuable to you as a reader. I know our conversation will help me and the teams I work with, as we learn and grow our expertise in DevOps. Please share your ideas, disagreements and learnings as we move along this journey together into the fast moving and quickly changing world of DevOps.

  • A Q & A with Gene Kim: DevOps and What Separates High Performing Enterprises

    A Q & A with Gene Kim: DevOps and What Separates High Performing Enterprises

    We recently caught up with Gene Kim (@realgenekim) as he was wrapping up the final touches of a draft of the upcoming DevOps Cookbook. We wanted to get his thoughts on what causes IT deployment failure, the DevOp movement, and finally what separates high performing IT organizations from low performing ones. If you’re following DevOps, no doubt you are familiar with Gene. In case you are not, Gene is an award winning CTO, researcher and author. He founded the security firm Tripwire, and was the CTO there for 13 years. To date, he’s written three books, including The Visible Ops Handbook and The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win. Gene is a huge fan of IT operations, and how it can enable developers to maximize throughput of features from “code complete” to “in production,” without causing chaos and disruption to the IT environment. Now, let’s jump into our discussion.

    devopsy.kinsta.cloud: When organizations decide they want to move to DevOps, or continuous delivery, what is a barrier that they immediately run into? And is DevOps something all types of companies can embrace?

    Gene Kim: In my mind it’s the unicorns (leading web app companies) that are sort of showing the way out. Just to show why I’m so interested in this, you know I’ve been studying high performing IT organizations since 1999. What I’ve noticed is that there is this downward spiral that happens in almost every IT organization. It led to more fragile application in production, longer deployment times, building up of technical debts, and the business goes slower and slower. I think one of the reasons that I cared so much about is that it led to preordained failure. Where people felt, especially downstream (operations, test, security) trapped in a system where we were powerless to change the outcomes. No matter what we did, failure has been preordained. I think the way out of the downward spiral is what organizations like Google, Amazon, Twitter, LinkedIn, GitHub, are doing. These organizations are doing tens, hundreds, or even thousands of deploys a day. In a world where most of us are stuck with one deploy every nine months with catastrophic outcomes. I think that the first objection is that we’re not an “Amazon,” we’re not a “Google,’ we’re a bank, we’re telco, we’re a retailer. That it doesn’t apply to them. However, it turns out if you look at who’s adopting DevOps it’s Nordstrom, Walmart, General Motors, SAP. One of the statistics that I saw was, that Google has 15,000 engineers, or developers. HSBC, the European bank, has 15,000 developers. I think every company is an IT company regardless of what business we think we are in. I think it just shows that anything we try to get down relies on IT. So for DevOps not to be a core competency is absurd.

    devopsy.kinsta.cloud: And even increasingly software has to become a core competency, as more devices become API-enabled? All companies are software companies now.

    Gene Kim:  I love that Jeff Immelt, CEO of GE, quote: “Our biggest fear is being disintermediated by software. So I’m going back to school on software and big data.” I imagine that they run into that same objection internally. “Hey, we’re not Netflix, we’re not Amazon.” Right? They are trying to sell the idea of moving to DevOps or becoming more agile. Anyone can do it and benefit from it. Even in general conversations that was the first objection that always came up. Well that’s good for so and so, but we’re not so and so. I think that makes it a belief issue. So that’s what we’re talking about here, right? It can’t work here. I think it a serious of problems and it’s very predictable what the bottlenecks or obstacles are. The first one is that it takes too long to do the deployments, actually you can go a step before that. One of the biggest problems is that operations is always in the way. Whenever a dev or test wants a test environment it’s not uncommon in typical enterprises, that it takes 40 weeks to get one.  Back in my Tripwire days, I use to think that developers didn’t like testing the code, because that’s what I saw. The truth is even worse than that; the developers really want to test their code. Their problem is they can never get an environment when they want one or they need one.  So the survival mechanism kicks in, they’ll always go to the guy they know that’s really good at getting an environment, Their Corporal Klinger or Nash. They don’t want to ask any questions about where it came from, because you know someone is missing a server somewhere. This person probably just stole an environment from another project. With virtualization and infrastructure code there’s no almost no excuse not to be able to create environments available on demand. One of the things that we did, with Puppet Labs, is we reached out to me to help design their DevOps benchmark. This was awesome, when I found out about it I jumped out of my chair because, I saw this as a logical continuation of ten years of benchmarking we had done.

    devopsy.kinsta.cloud: What have you learned that separates high-performers and makes DevOps possible?

    Gene Kim: There were three findings about it that blew me away. The first one is the high performers were massively outperforming their non-high performing peers. Thirty times more frequent code deployments. They were able to complete the deployments from code committed to running a production 8,000 times faster. In other words they could do a deployment measured in minutes or hours. Whereas low performers it took them weeks, months, or quarters. Furthermore, they actually had better outcomes. Twice the change success rate and when they caused outages they could fix it 12 times faster. There were 4,200 that completely responded and it just showed that you could be more agile and reliable at the same time. I think if we just connect some dots, they’re also going to be more secure, they’re probably going to have a faster find fix cycle time. We didn’t actually test that but, if that were true it would mirror what we found in the previous benchmarking work I had done. So that’s surprise number one the extent that they were out performing their non-high performing peers. Second surprise, we found two behaviors that were present in every high performer but they were universally absent in every non-high performer. The first one was that they are they checking in all their production artifacts and version control. Meaning, they could reproduce a production environment based off of what was in version control. Versus only it being an indescribable asset that only one person understood, if they knew about it at all. The second one was is there an automated code deployment process? Which includes being able to create environments available on demand. The next one is the code deployment process. I think it’s very typical in most enterprises that whenever you want to do a new release or deploy it takes days or weeks to complete. Involving tons of hand-offs from the developers, DBA, release team, back to the developers to the security people, and the firewall change and so forth. All that creates delays and is error prone and so forth. In the high performer there usually a one click to deploy or an automatic deploy. If you look at Google, Amazon, Twitter, Facebook, in general they have a continuous deployment process so that when a developer checks in code it automatically runs all the tests. Then it puts it into an integration test environment and runs all the tests there. There might be some manual work, but then in general it automatically migrates to production. Clearly, that’s what’s required to do multiple hundred of deploys a day.

    devopsy.kinsta.cloud: That is incredible to learn, Gene. A lot of fascinating findings. Thanks so much for your time.

    Gene Kim: Thank you, George. Always a pleasure speaking with you.

  • Is DevOps Ready to Cross the Chasm to Mainstream?

    Is DevOps Ready to Cross the Chasm to Mainstream?

    There are just a handful of business books that I have read that have had a profound influence on me.  One was Gene Kim’s “The Phoenix Project”, another was Eliyahu Goldratt’s “The Goal”. A third one was Geoffrey Moore’s, “Crossing the Chasm: Marketing and Selling Disruptive Products to Mainstream Customers”.  Even though written in the early 90’s, it is as relevant today as it was then.  Though our technology has changed (dare I say advanced), the adoption curve of who uses it has remained static.

    When I look at the world of DevOps today, to me it is on a classic chasm model.  The real question is will it or rather how quickly will it make the hop over the chasm into mainstream. The picture above is a great representation of the chasm. Most technologies go through the enthusiast period, moving to the visionary.  I would say that DevOps has been in this visionary period for a few years now.

    In order for any technology to jump the chasm it needs to build up enough momentum and critical mass to successfully make the jump.  Now there are some technologies that never build up enough critical mass to successfully jump over to the pragmatist’s cliff of mainstream adoption.

    Back in my StillSecure day’s NAC had a tough time making the jump. It was not until mobile technology and BYOD served as ignitions that NAC made the cross-over to mainstream.  But enough about NAC lets return to DevOps.  For DevOps the Cloud and virtualization have been ignition points.  Using software to replace hardware has allowed for rapid deployment, automation and continuous delivery models that characterize DevOps.

    With so many pundits and analysts predicting that this is the year Cloud goes mainstream (I thought it did already), DevOps will not be far behind. As disruptive as Cloud is, could it be as disruptive if not for DevOps? There is a symbiotic relationship between the two.

    But it is not just Cloud alone. DevOps has other boosters that will help it over the chasm. SDN is another big boost.  Software will eat the world according to Marc Andresseen. It may very well start with networking hardware.  But software is eating more than hardware.

    One of the hottest trends in security (and one you will see featured here in the SecOps neighborhood of blogs on devopsy.kinsta.cloud) is Software Defined Security.  It may sound like marketing speak to you, but it involves putting security right into the virtual, software stack.  It is DevOps for security.  In fact DevOps and security is something you are going to hear more about from the likes of Josh Corman, Dave Mortman and Dwayne Melançon and others.  Their “Rugged DevOps” discussion is ripping through the security industry for the last year or more.

    All of the above aside the real key for DevOps to making the leap is that instead of just enthusiasts and visionaries being involved, what Moore calls “the Pragmatists” get involved.  These pragmatists are not hung up on labels or new versus old.  They just want to get the job done. If DevOps will help them get the job the done, that is good enough for them.

    The pragmatists make up a good chunk of the mainstream market.  The silent majority if you will.  To win these pragmatists over DevOps just has to show that its benefits are real, quantifiable and worthwhile. The good news is that the Pragmatist are the first group on the other side of the chasm.

    Having the Pragmatist as the first part of the mainstream is a good thing, because after that it gets harder to gain traction.  According to the Chasm model after the Pragmatists come the Conservatives.  Though not quite reactionary, this group doesn’t like change. They won’t change until they are nearly forced to by business pressures. They will resist adopting DevOps just because it is different than what they do now.  Generally technology has to be pretty mature, its benefits well established before this group will consider doing anything more than a small trial.  With DevOps representing as much a change in philosophy and culture as it does new tools, this is a really bitter pill to swallow for the Conservatives.

    I would think we are years away from DevOps making real traction into this group. Especially when there are still some folks who claim that DevOps is DOA, DevOps is much ado about nothing, that it is doomed to failure. But hey there are people who still say that the science doesn’t point to climate change and that we should teach creationism in schools.

    Call me the eternal optimist though. I do think that as Cloud and the other catalysts behind DevOps continue to expand, DevOps will be swept up even into the conservative base.

    Between Conservative and Pragmatist, a large slice of the market is spoken for. Added to the pre-chasm market and DevOps is well beyond critical mass.  Whether the last segment of the market, the skeptics ever adopt is not even relevant.  Many of these skeptics have a bone to pick. As I have grown older I am convinced that they are just wired to be skeptics.  They literally look at everything through a half empty glass.

    So there is little doubt in my mind that DevOps will not be a stillborn technological movement. It has the catalysts, the benefits and most importantly timing and circumstances are lined up to help push this from the early adopter crowd and right over the chasm into the mainstream. What’s more is I think this is already under way and will be much quicker than most people think.

    This is all a big reason why we launched devopsy.kinsta.cloud.  There is a big enough critical mass now to support a site dedicated to DevOps like this.  But as we cross the chasm to mainstream, we want to ride the crest of the wave, helping to bring DevOps to pragmatist, conservative and anyone else who would benefit from it.

  • Let the Code do the Heavy Lifting

    Let the Code do the Heavy Lifting

    One of the first jobs I had when I started in IT was as the IT administrator for a dot.com startup. IT admin at a smaller company like that was typically an umbrella term used to describe a one-man IT department, which is essentially what I was.

    The company was growing exponentially—both in terms of customers and revenue, as well as traffic to our website. What that meant for me was a steady stream of UPS shipments from Dell, and a virtually full-time job of mounting servers on racks in the data center, getting them properly configured, and deploying them to the network.

    Dealing with 50 or 100 hundred servers in 2000 was a daunting task, but dealing with hundreds or thousands of virtual servers in 2014 is—for all practical purposes—impossible. That is, it would be all but impossible doing things the old-fashioned way. However, the DevOps revolution changes the rules of the game because it uses code to do the heavy lifting, and automate those tasks which can be automated so the IT admins of today are free to engage in more productive tasks.

    If we were still dealing in physical servers, the benefits of automation that you get in a DevOps environment would still not be possible. What allows so much of the traditional “racking and stacking” today to be automated is that we are dealing primarily with virtual servers. Although a virtual server may act like a server, and fulfill the same duties as a physical server, it is essentially just a software program. The fact that a virtual server is comprised of nothing more than code means that virtual servers can be created, configured, and deployed automatically through code as well.

    It’s a good thing, too. The fast pace of a DevOps culture would be challenging to maintain without the efficiency of automation. The ability to create, provision, destroy, or move virtual servers around on a whim is part of what facilitates agile development and accelerated execution by DevOps teams.

    There are also some adverse consequences of automation. People have lost their jobs in manufacturing plants as machines have come along to automate tasks they once performed. There is a risk that automating many of the routine tasks normally delegated to IT personnel could result in fewer people being necessary, but that is not a reason to not embrace automation. Automation enables people to develop more advanced skills and focus on more important tasks—those functions that can’t be easily automated.

    One of the primary aspects of functioning in a DevOps environment is a faster development cycle, and more frequent deployments. Traditional organizations might deploy updates once a quarter, or once a month, but DevOps teams often deploy updates at least daily—sometimes multiple times per day. Automation reduces the possibility of error—it is a simple, repeatable process that works reliably and predictably.

    A survey of DevOps Conference attendees published by JumpCloud and SoftLayer found that about 75 percent of the respondents use a configuration automation tool such as Puppet or Chef. More than half use scripts of some sort to execute tasks, and about one-third reported using a patching tool of some sort. Many of those surveyed admitted that they have the tools they need, but there simply isn’t enough time to get things done.

    Perhaps that explains why 82 percent of survey respondents plan to automate more. Breaking the data down a little farther, nearly a third are looking to increase automation of configuration and deployment, and about 10 percent want to automate user management, and a variety of tasks like log management, scripting, testing, patching, and monitoring all came in around five percent or below.

    It’s still possible to create virtual servers, and configure them manually one at a time. That may work if you’re bored, or nostalgic for the “good ole days” of IT, but it’s impractical, and frankly just dumb. It’s also possible to cut your lawn with a pair of scissors, or paint your walls with a Q-tip, but it is highly inefficient and ill-advised.

    Don’t waste time and resources performing tasks that can be automated. Identify the functions you routinely perform that are consistent and repeatable, and embrace automation to take care of those things for you so you can focus on more important things.