After the success of the first two years of the U.S. DevOps Enterprise Summit, people have been asking us to host a similar conference in other locations, specifically in Europe. This summer, we’re hosting our first DevOps Enterprise Summit in London!
This conference is for technology leaders who are driving transformations in large, complex organizations. These are successful enterprises that have been around for decades (and, in some cases, for centuries!), with thousands (or tens of thousands) of employees, and billions of dollars in annual revenue. These are organizations with massive and complex daily operations, and all the resulting processes and bureaucracies that come with it. In many ways, these complex operations and bureaucracies are the very things that create the worst-case conditions for DevOps transformations!
We’ve heard amazing case studies from Barclays Capital, GE Capital, Disney, Marks & Spencer, Bank of America, Capital One, Telstra and many more, where leaders have created amazing DevOps outcomes that are more typically associated with organizations such as Google, Amazon, Facebook and so forth.
For this inaugural year of DevOps Enterprise Summit London, we’ll be bringing some of our “greatest hits” from the United States, but the rest of the programming will feature DevOps transformations in organizations residing in the U.K. and continental Europe. (And we’re still holding the DevOps Enterprise Summit in San Francisco, so feel free to join us there as well!)
New World and Old World DevOps
The speakers at DevOps Enterprise are pioneering the practices of how large, complex organizations transform and deliver software. It’s my belief that these principles are universal, regardless of what continent or country the organization resides in.
One of the biggest surprises from co-authoring “The Phoenix Project” has been how many people have told me that the book describes their organization, seemingly independent of company size, industry vertical, for-profit or nonprofit or the continent they’re writing from. This validates that the problems that result when Dev and Ops can’t get along are universal.
The good news is that the DevOps solutions we deploy to fix the problems also have lots of commonalities!
It’s always inspiring to hear how these leaders have overcome the challenges they faced. I see our speakers as genuine heroes who have put themselves in personal jeopardy, often risking their careers and livelihoods, to achieve something they believe is genuinely important. They possessed the clarity and conviction that the capabilities they were creating for their organizations were needed to win in the marketplace—or maybe even just to survive.
I think one of the interesting statistics is that more than 25 percent of DevOps Enterprise speakers have been promoted in the last two years—some more than once! It’s a fantastic testament that the organization values the contributions they have made, and their risk-taking and heroism paid off.
The DevOps Movement Matures
Every year at the DevOps Enterprise Summit, I have asked all speakers to share what problems they didn’t know how to fix or the problems that they were looking for help on. What I’ve found is that the answers have evolved. From year to year, we are seeing trends on the obstacles facing the DevOps Enterprise community.
In 2014, attendees’ stated problems were about needing better strategies and tactics for creating automated tests for legacy applications, as well as the culture and leadership aspects during transformation, information security and compliance practices and identifying metrics to best improve performance with DevOps initiatives.
In 2015, the problems were focused on people and organizational design. Examples included the need for effective strategies and methods for leading change in large organizations. Speakers and attendees also wanted to know what the organization charts looked like for organizations successfully adopting DevOps. What were the respective roles and responsibilities, and how had they changed from more traditional IT organizations?
Now, as we approach DevOps Enterprise Summit 2016, we are seeking to address the next phase of challenges facing the community: What are the modern architectural and technical practices that every technology leader needs to know about? What are the technology practices for managing infrastructure, environments and automated testing? And what are the transition patterns to get there?
From the HR perspective, how do you go about reskilling the workforce and employing these new design patterns? Lastly, we are seeing compliance and security discussed more often, including: What are concrete ways for DevOps to bridge the information security and compliance gap to show auditors and regulators that effective controls exist to prevent, detect and correct problems?
The programming committee always invites a handful of speakers to return from year to year. While to some this might appear redundant, it’s actually quite fascinating to witness their journeys unfold. It’s like an episode of, “Where Are They Now,” and the speakers never cease to amaze!
Solving Problems Together
As a vendor-neutral conference, DevOps Enterprise Summit curates stories that are from the enterprise point-of-view. Most of the presentations follow a very specific “experience report” structure—speakers describe their organization and their industry, the business problem they faced, where they started their transformation, what they did and the business outcomes. It’s a simple format but very powerful because it helps other people facing the same problems learn how other people solved the problem, and it often reveals practices that can be used in other contexts.
I’m so excited to share some of the initial speakers we have slated for DOES London:
Chris Jackson, Director of Cloud Product Engineering at Pearson
Robin Hughes, Head of Development Services at LV=
Justin Dean, SVP, Platform and Technical Operations at Ticketmaster
Jeremy Waite, Chief Digital Strategist at Salesforce
Jason Cox, Director, Systems Engineering at The Walt Disney Company
Rosalind Radcliffe, Distinguished Engineer, Chief Architect for DevOps for Enterprise System at IBM
What also amazes me about DevOps Enterprise is the concentration of talent in the technology leaders who attend—they’re all people who are passionate about technology and helping teams win, and willing to do disruptive things to get there.
Ross Clanton from Target (a large U.S. retailer) told me he valued being surrounded by people who were not only trying to solve the same problems that he was facing, but were the best in the game—and there’s a huge intrinsic value of being surrounded by those kind of people. I can definitely relate; I learn something in every one of my interactions at the conference, and I have no doubt that the practices this group is pioneering will be commonplace within a generation.
Like in any aspect of life, peers are important. As Buzz Aldrin said, “Show me your friends, and I will show you your future.”
DOES is a unique opportunity to meet some very talented and inspirational new friends who can help shape your career path. We hope you’ll join us in London.
Legacy information technology infrastructure has a strong effect on the potential success of DevOps. In environments where different teams have always used the same servers, change can be difficult due to inter-team coordination and limited server transparency and application inventories, says David Blank-Edelman, a 30-year systems administration/DevOps/SRE veteran and currently Technical Evangelist at Apcera.
“Unraveling the Gordian knot to bring DevOps principles and practices into that environment will be quite a challenge,” he says.
In such a legacy environment, the enterprise walks a topological tightrope stretched between one or more development machine clusters shared by multiple teams with no real isolation between teams and projects, Blank-Edelman notes. “In most cases, there is a second parallel set of machines for production and, if the business is very fortunate, for test/QA as well. There is seldom anything, however, to prevent a production workload from talking to a database server in development by mistake, which can lead to all sorts of fun.”
How to Move to CD
One solution is a policy system approach, in which the organization provides for a policy system to migrate a legacy information technology environment to a platform that uses continuous delivery (CD), Blank-Edelman says.
To create the policy system, he notes, the organization will need to address questions such as:
Are the workloads allocated/using only the resources that they should?
What is inside the workload? What is its provenance? Is it acceptable and in good order from security, coding and business requirements perspectives?
Where is the workload allowed to run? In which cloud? In which environment? In which country? Near what other containers?
Can the workload communicate with only the things it should? If I move it to another location, is this still true?
“The process of developing a policy system and then enacting it can lead an organization on a journey that brings their existing infrastructure to a much better place. If that platform can also automatically enforce the decisions that organization made during this question and answer discovery process, you can be certain that the answers to these questions stay answered,” says Blank-Edelman.
Other approaches to moving to continuous delivery include tool-first approaches in which the organization adopts one to several CI/CD/workflow tools and then makes an effort to implement those, possibly by shoehorning existing processes and runbooks into that tool or toolset.
A better tactic is the people-first or culture-first approach that focuses on how the development and operations personnel will work together to create a product vs. a project, says Blank-Edelman. While tools still have a role to play in the people-first approach, they are not the priority; rather, organizational change comes first. Once the people have made the commitment and the change, everything else will follow.
Challenges During the CD Move
It’s not the existing information technology infrastructure that presents moderate challenges to CD that is the issue; it’s the infrastructure that poses either very many challenges or very few that is the hurdle. “If you start with a more challenging situation, this will incentivize people to work very hard to reduce the associated pain points in their life,” Blank-Edelman says.
On the other hand, he notes, when an organization begins a move to CD with an infrastructure that is already in pretty good shape, you won’t often hear its people say that they should stop working on the product and work on CD all the time instead. People will still claim they have plenty to do already.
Continuous Deployment Can’t Exist Without CD
Continuous delivery starts with a code base and takes logical steps through that code base to increase confidence that it is ready for production, using a process that is as smooth, automated and loosely coupled as possible, he says. With confidence in that pipeline comes confidence in moving the result of that process into production automatically, which is continuous deployment, he adds.
I realize the title of this article may conjure up ideas about the impacts of entanglement from quantum computing, but no, we are not there yet. Instead, I am referring to the benefits an organization can achieve from determining which environments in its software development life cycle (SDLC) should remain permanent and which are better candidates to be transient to understand the discreet costs of innovation (ideally if virtualized).
First, to keep an even playing field, make sure your DevOps services are using the “build once, deploy many” philosophy. Under this thinking there is a single build constructed that is deployed unilaterally into each class of SDLC environment (not a unique build for each class). Uniformity under test eliminates variability of results. Uniformity of the basic process of deployment, builds consistency and predictability also will reduce errors and the time it takes to resolve them.
Second, the actual computing capacity you deliver any environment on may be virtualized, whether you provide your own bare metal assets (private cloud foundation) or if you use a public cloud entirely is less the issue. Understanding whether a class of environment is transient or permanent only affects how you plan for the costs and policies of operating it, and it leads back to the discreet cost of innovation as a result.
The Cost of Creation
When this topic is usually considered the first and immediate point of view is usually R&D, or the development class environment. And yes, the ability to provision a DEV class environment and then tear it down when the effort has completed is the normal use-case to justify virtual infrastructure at all (public, private or hybrid cloud usage). The obvious benefits include having no “throttle” on how many concurrent efforts might be spun up.
Transient DEV environments also build in a concept of failure, or elongated “playing,” not intended to produce a specific result but intended to dabble with innovation to see what “might” be possible. All of this without the capital investment it would take to build and maintain an R&D lab to accommodate it, or set aside fixed computing capacity for it. DEV efforts with intent (i.e., part of a formal project) can be tracked for costs discreetly with that planned innovation. DEV efforts without intent (i.e., more like what used to occur in R&D labs) can be associated to the overhead costs of all innovation.
The Cost of Evaluation
But beyond DEV, the next major class of environment to consider is the integration (INT) class. If your company has more than one application, has apps that work together or has applications with upstream or downstream dependencies, you likely need an INT class environment. The major responsibility of this class environment is to provide “something to point to.” Your app (undergoing change) has an environment where it can point to all the other applications to test how the interconnections work. Logically, this class of environment is a better candidate for permanence. This is because you can rarely predict which apps may need this service at which time across the entire portfolio. Many apps undergoing change may be using this at one time to insure they all work with each other (i.e., assessing their interoperability with future versions, early, or prior to production).
Another reason for maintaining a permanent INT class environment may involve your interaction as a vendor to other companies who “point” their apps at yours as a service provider. Even organizations who have little interaction between their own apps, may well be a significant provider to a wide swath of customer applications. If your customers need a place to point to, the INT class environment is typically the one you want. So in this instance permanence is the logical need.
Next up, becomes the performance test (PT) class of environments. Emulating production, or being able to project differences between the platform you use, and what production provides is key to success. However, this class of environment is a perfect candidate for transience (especially if your development and production environments are already virtual). Why maintain expensive hardware on the floor, only to conduct periodic testing, which requires massive reconfiguration and preplanning. Dump it. Instead, use a virtualized platform where provisioning is near instant and can exactly mimic both development and production – remember: build once, deploy many.
Another significant class of environment in the SDLC to examine is the User Acceptance Testing (UAT). Logically, this class environment may get a lot of usage, but ideally only after the app has passed testing in development, INT and PT. Because of this, the UAT should be transient as well. A good candidate to tear down when not in use. You may wish to plan for a setup and tear down of this class to mimic what happens in formal development efforts.
Finally, the quality assurance (QA) or preproduction (PreProd) class environments, if in use, are designed to emulate the current state of production completely, allowing the app-under-change to run against a more sophisticated type of INT class environment (with possibly copied production data, etc.). This happens more often when the earlier INT class environment contains a great number of other applications under change. This one provide a “cleaner” place to point to, only production operating apps. The QA environment may also be used as an after action, when an error is discovered in production for debugging or more extensive monitoring. Obviously, if your organization finds a need for this class of environment, it will carry the permanence characteristic.
The Cost of Operations
Production is by nature permanent. Virtualizing production offers the ability to scale on demand. But a scalable production solution benefits from more mature DevOps services (whether you own the data center or not). Consider for a moment, the number of errors attributed to code post a DevOps embrace. The number of errors goes down, the time to resolve them goes down, the size of the effort gets smaller for less risk, and higher speed. This combination means production is going to suffer far less from aberrant code running amok (asking for tons of memory, CPU or disk due to error). Those kinds of problems get detected earlier (because of DevOps) in the SDLC process and hardly ever reach production after DevOps is engaged.
With a mature use of DevOps, it means that any kind of scalable solution of capacity-on-demand will get engaged for only two reasons: a dramatic increase from customer demand (the ideal solution) or a denial of service attack attempting to drive your costs higher without real customer use. The monitoring solution you employ will help decipher the difference. If your application has logging features built into it (i.e., you produce a log detailing what customer is performing what key feature in your app and when) you should be able to avoid the latter scenario with an automated monitoring detection effort.
New Abilities to Measure the Cost of Innovation
Understanding the cost delta of transient and permanent environments is worth examining. It will help you understand better the cost of innovation on an annualized basis. And it may also reveal how much real effort goes into testing of each kind of class. If user adoption is suffering for example, and you determine that UAT class environments are largely unused or torn down, you know better where to address the problems earlier in the cycle. The same can be said for PT as well, how much test, for what kind of scaled performance in production.
If your organization maintains INT or QA class environments you may want to allocate their costs to SDLC overhead as they tend to be permanent, and value shared. The development, PT, and UAT environments, if transient, represent the variable cost of a particular innovation effort and can be tracked with good discretion. When considering the timing, value, and benefits of virtualization, you may wish to factor in where you are on the DevOps continuum. If you have had DevOps running for a while, the timing for virtualization and scalable capacity on demand may be better, than if you are just starting out.
Keep in mind that most organizations who have transitioned away from waterfall and into agile, will lose a good deal of the former financial controls that worked in waterfall, and do not work as well in agile. DevOps delivery is also capable of moving change at orders of magnitude higher speeds than former manual methods. This combination makes trying to cost innovation discreetly more difficult. Being able to associated transient environment costs with formal development efforts (ones with intent) may restore some of that financial discretion. It may also help identify the true overhead costs of innovation located in the operation of environments classed better as permanent.
To continue the conversation, feel free to contact me.
My encounters with commercial clients in the DevOps space have been exciting, curious, political, and challenging. What most don’t realize is that DevOps is not just about infusing tool automation into a delivery pipeline, but it also has to do with re-mobilizing and challenging the status quo when it comes to “who does what”. As teams become more tightly integrated, functional lines blur, and uncertainty may set in.
I find more and more, that clients and colleagues want to know:
What happens to the operations/run tower people if tools like UrbanCode Deploy automate the promotion of applications and releases to run-time environments?
What happens to our system administrators and “build out” teams, if tools like Chef, Puppet, Vagrant and pattern-based provisioning systems like IBM Cloud Orchestrator and UrbanCode Deploy with Patterns can perform solution stack build outs with minimal custom network, hardware, software, and middleware configurations?
If developers can provision and deploy their own apps, what do the sys admins do?
Will Operations Staff be replaced by tool automation?
The truth is, now, more than ever, a savvy, astute, and innovative operations staff is essential to successfully execute effective DevOps within the enterprise.
As development and test teams are being empowered to deploy their own applications with tool automation, Operations will need to shift the focus to enabling and empowering these teams with standardized, reusable automation assets. Design, development, maintenance and management of these automation assets will be the core foundation of DevOps success within the enterprise. A corollary to this is a set of standards and policies must be clearly defined and adopted as a framework for implementation.
Standardization is paramount to making DevOps successful on a wide scale in the enterprise. If we were to think of our applications as a “solution stack” of network, hardware/virtual machine, operating system, middleware, and custom application components, standardized assets in each of the solution layers would enable faster assembly, development, test and delivery of a solution.
When it comes to effectively enabling the enterprise for DevOps, you must become a hater of snowflakes. Snowflakes can be elegant, but they are unique and difficult to automate with tools. Examine applications and systems for uniqueness…and ask, can it be standardized? Typically, snowflakes occur higher up on the solution stack at the Application or Middleware layers. Network teams are not likely to implement new security policies for each application. However, moving up to OS and middleware, uniqueness is more likely to occur (OS release versions, MW versions and packages and so on).
Once opportunities have been identified for standardized assets, operations teams must mobilize to enable the enterprise for automation:
Encapsulate enterprise policies in standardized automation assets (which can be reused across the enterprise).
Centrally manage and maintain automation assets within the enterprise (to maintain standard and policy compliance).
Evangelize, encourage, and incentivize the use of automation assets in delivery of applications.
Provide a continuous delivery pipeline tool set (preferable, as a service) providing an infrastructure to enable the automation assets for development and test teams
Automation assets can occur at every layer in the solution stack, providing an opportunity for the new Operations team to inject enterprise influence and compliance at every level. This approach ensures standards and compliance, but also relieves application development and test teams from worrying about whether their application violates any corporate policies.
Examples of automation assets and some enabling technologies are shown below, but this is obviously not an all-inclusive list (even a hardened shell script can be an asset). The point is, for each layer in the solution stack, assets and corresponding enabling technologies can be chosen to invest in. Some technologies can address multiple layers in the solution stack.
There is clearly a lot of good, solid, critically important work for the new age Operations team to manage and maintain. Interfacing with enterprise architecture and corporate standards groups are essential. However, the focus remains largely around empowering and enabling delivery using standardized assets.
You can also watch this short video that I recorded recently. Please do share your feedback on this blog and how you see the evolving role of IT Operations in a DevOps Era.
Andrea Martinez is the DevOps Solution Leader for Cloud Services in IBM. Andrea’s mission is focused on delivery of DevOps solutions in cloud and traditionally hosted environments. Andrea has lead the solution architecture, implementation and initial adoption of one of IBM’s first DevOps Deployment Offerings. As a member of the IBM Academy of Technology, and a certified Thought Leader in application architecture, Andrea is keenly interested in enabling and empowering practitioners with innovation and technology to deliver solutions faster.
Continuous integration and continuous delivery (CI/CD) are often cited as pillars of successful DevOps. To establish and optimize the CI/CD development model and reap the benefits, companies need to build an effective pipeline to automate their build, integration and testing processes. This requires architecting CI/CD with five key ingredients which, when combined effectively, operate as a successful cornerstone of DevOps and provide myriad bottom-line benefits.
Building the Pipeline
Let’s first take a look at the five essential ingredients needed for a healthy CI/CD architecture, following up with a discussion of their benefits to the business:
Micro-agility in the form of testing and committing code frequently: The development life cycle begins with a request to development, which then takes the request and designs a solution, codes it, tests it and commits it a code repository. From here, the code is tested by QA and then deployed to production. While each step is important, we begin here, as local testing occurs most frequently and bugs found at this stage of the process help avoid the large expense of bugs found in production.To achieve micro-agility, developers must have a clear understanding of where they are expected to commit their code after changes are made and they should be expected to test and update their code frequently—many times a day. This means development testing should be easy to run locally and is a subset of QA. The result is that code pushed by development is by definition QA-ready.
Automated QA: In my work I see many organizations whose QA team is one of the largest bottlenecks in the process. It’s not for a lack of skill, but too few staff who spend a good portion of their time waiting on the QA test environments they need to do their work. To avoid this situation, the QA process should be fully automated, from QA ready code coming in the door to one-click promotion on the other side. In between the two, QA should have a constant feedback loop with development where communication about—and tracking of—bugs is clear and easy to follow, allowing development to easily repair issues and recommit the code for further testing. Moreover, automated QA in a CI/CD environment should focus on microcycles consisting of small test batches rather than weeklong macro tests. Automation and communication are the overriding imperatives here, exhibited by reports to developers that are communicated immediately and code commits that are fail-rejected automatically.
Production-like environments: To increase efficiency and effectiveness, development and testing environments should mirror production as closely as possible. They should follow the PRQ rule, as I like to call it—production-like, repeatable and quick. Specifically, it should be fast and easy to set up new environments, to the point where developers can, with the push of a button, deploy new environments for their development and testing needs. Doing so allows developers to fail fast, breaking and re-creating environments without fear as they know they can easily recreate the same environment every time. I often recommend using tools such as Jenkins to help with this process; if developers and IT operations use the same tool, it makes it that much easier to streamline the CI/CD process and achieve the promise of DevOps.
One-click promotion: As I alluded to earlier, a well-architected CI/CD pipeline provides one-click promotion to production. This ensures changes are seamlessly moved to the overall production environment with the least amount of friction possible between development and operations. To that end, this reduction in friction should occur across all the steps of the process, with efficient code delivery pipelines that are designed to balance security and agility from the outset. Cloud infrastructure coupled with container technology and automation can easily optimize the process across teams and throughout the DevOps journey.
Fully automated deployments: The last factor of an efficient CI/CD pipeline is full automation of deployment. Any configuration changes made by development and accepted by QA should automatically be applied. Continuous integration in this way helps avoid system downtime and human errors are decreased greatly. Deployments also should be continuously monitored so that IT is the first to learn of potential problems. This helps nip issues in the bud, allowing for faster rollback and return of code to development to fix anything that may not be functioning properly. Catching issues before customers do helps ensure customer satisfaction and strength of reputation.
Benefits of the Pipeline
In addition to these benefits, when combined effectively, these ingredients minimize the time and cost of local test, QA and production processes; reduce the time it takes to identify problems and their causes; and accelerate delivery of production-ready code. We can see this throughout the CI/CD pipeline:
Developers have greater control over the process and can respond more quickly to market demands by quickly bringing up new environments, testing against them and easily starting over, if need be. Moreover, developers can quickly adjust to new environments—an approach that has proven to decrease failures and resolution time when they do occur.
Operations benefits as reduced friction in the CI/CD pipeline created by automation decreases repetitive, manual work and with it the opportunity for “fat finger” mistakes and the greater opportunity to introduce inefficiency into the process.
And, less manual, repetitive work provides everyone in the process with more time for strategic work, which provides direct bottom-line value to the organization.
CI/CD is a pillar of successful DevOps and according to research by CA, organizations that apply DevOps discover an 18 percent reduction in time to market of new applications and a 21 percent increase in delivering new software that they would not otherwise be able to complete. Teams that master an efficient CI/CD pipeline deliver faster, higher-quality applications to market bringing with it a 19 percent increase in revenue. Mixing these ingredients together results in an efficient, streamlined CI/CD pipeline that serves as a fundamental pillar of DevOps success that, when mixed together, delivers direct business benefits.
About the Author/Aater Suleman
Aater Suleman, CEO & co-founder at Flux7, is an industry veteran in performance optimization on servers and distributed systems. He earned his PhD at the University of Texas at Austin, where he also currently teaches computer systems design and architecture. As a faculty member at UT-Austin, Mr. Suleman focuses on optimizing DevOps architectures involving the AWS cloud. Along with the Flux7 consultant team, he works at every step of the dev flow, from creating local developer environments using Vagrant and Docker, to continuous integration and continuous delivery using Jenkins or RunDeck, to configuration management using Chef/Puppet/Ansible, to creating efficient systems for deploying new code from Git using Capistrano/Fabric/bash. His background is as an architect, and he has experience with both hardware and software. This combination brings a unique perspective and broad view to development and operations.
Why DevOps is not for everyone and how release automation helps in bimodal world
So am I really going to write a blog calling DevOps a big hoax?
Well, not exactly, but as a follow-up to my previous blog on the difference between DevOps and Continuous Delivery, I want to alert you to a few key facts:
DevOps is not always the right methodology for developing and delivering software.
Agility and speed can be achieved even in a waterfall method if the right tools are in place.
Okay, so here goes: Why is DevOps not always the right way to go? Simply because not all systems are born equal. For a back-end master customer data system, stability trumps iteration. Fail fast and roll forward simply aren’t sustainable in many of today’s most core business applications such as banking, retail, media, manufacturing or other industry vertical. Clearly, these enterprises still need to innovate and provide new and exciting ways for their customers to engage through web and mobile, but not at the risk of a terminal business failure at the back end.
Delivery capabilities for these back-end systems need speeding so they don’t become the bottleneck for the entire business. However, restructuring teams and changing development along with test and release into tens and hundreds of small/tiny parallel updates cannot be supported by a legacy technology stack (say Siebel or SAP at the back end). It is likely not even advisable at all.
This doesn’t mean we should accept the realities of the last century, resort to the status quo and leave our release cycles to once or twice a year for key applications. Slowly and surely we would lose any competitive edge.
Luckily there is a middle ground, and this is why I believe release automation is bigger then DevOps. That middle ground is release automation. I will explain why, but I need to start by examining DevOps a little more first.
What is DevOps?
At its core, DevOps aims to get new ideas and functionality into the hands of users quickly and iterate fast. It achieves this by fixing two traditional weaknesses in the software delivery life cycle, namely “overhead” and “rework.”
Overhead is anything that developers do that isn’t directly related to developing new features or improving existing ones (manual deployments would be a big item in the overhead column for sure, as would debugging production deployment issues or handling configuration problems).
Rework occurs anytime developers have to go back over a piece of code they already have worked on. However, this does not include bug fixing, so rework occurs when test cycles take a long time and developers already have been working on new functionality, which now needs to be reworked to address a newly found problem.
DevOps tackles these challenges by attempting to shorten the cycles—for example, by adding smaller changes that developers and testers can test in real time or by using tools such as Puppet or Chef for provisioning quickly to test while ensuring consistency across systems.
This relies on Devs owning updates and rollouts across the life cycle, including production, which challenges everything in the way enterprise teams are traditionally organized. It calls for completely new processes, which often are a great fit for newly developed applications and modern architectures, but can be a complete misfit (and regulatory obstacle) for so many other core systems.
I’m sure you can see where I am going with this. Much like the story, “The Emperor’s New Clothes,” DevOps isn’t what people believe it is.
Why Release Automation is Bigger Than DevOps
The goals of DevOps are sound, however, which leads me to my last point: Release automation is bigger than DevOps. Done right, release automation can accommodate diverse organizational needs and different types of applications in this new, bimodal world. It improves the software delivery process and its final product by rooting out manual work, improving cycle times, reducing overhead and rework.
Crucially, release automation enables incumbent businesses to stay competitive and ensure survival without forcing them to completely abandon their DNA across the board in a “big bang” fashion. Such a process could put the very existence of a large enterprise at risk.
Before signing off on this blog entry I want to make one last thing clear: The argument I am raising here isn’t trying to downplay the benefits of DevOps as a valid methodology. Nor am I denouncing any of the DevOps tool chain technologies out there, which are all welcome advances, of course. This year, 2016, is (according to most industry experts) the year in which DevOps adoption goes mainstream, and I fundamentally believe that it is release automation that drives this adoption, not just pure DevOps conversions.
Have you ever seen the sign, “Will work for food”? If you deconstruct the premise a bit, we all work for the rewards systems that interests us. We “trade” our labor (ideally, focus our passion), for a set of acknowledgements that range from survival to pride. And ultimately it is our employer (whether the boss or the board of directors) who decides what our rewards systems will be. That is the common sense part. But when something such as DevOps enters the picture, it alters traditional thinking throughout an organization, and the ripple effects should (and eventually will) reach the HR department.
The most immediate effect of DevOps on HR occurs around talent acquisition. Finding a qualified DevOps engineer that it is possible to afford—and the company can retain—becomes the unicorn of recruiting departments from one coast to the other. The alternative is creating or adopting a training curriculum that can move a traditional programmer or engineer into a fully qualified DevOps engineer. Of course, advice varies on exactly what that is, and if it’s successful, your organization becomes a feeding ground for every other competitor to poach your folks right after training.
Because this HR acquisition problem is so great for most organizations, they can overlook some other underlying issues that continue to occur. These problems have always been there, but DevOps brings a spotlight to them that did not exist prior to its implementation. As stated earlier, we “do” what we are “paid” to do. In organizations that innovate in the software discipline, we have always “paid” that entire division to innovate. Prior to DevOps, the pace of that innovation, was slow and manageable. Once DevOps begins to take hold, the pace of innovation begins to enter the 400mph superhighway, where traditional control mechanisms no longer work.
In itself, the success of the controls over change—transitioning to a high rate of speed—will be determined by how we implement DevOps. It is the other half of DevOps that needs some focus. While our Dev organization is cranking out change exactly as we have always paid them to do, our Ops organization traditionally has been paid for exactly the opposite characteristic. We pay our Ops organization for stability. We want the trains to run, on time. We do not want a 2 a.m. phone call from an angry CEO—or, much worse, an angry customer—because our operations have “broken.” And, of course, the No. 1 culprit for causing breakage in operations is … [drum roll, please] … “change.”
In the yesteryear of Waterfall construction and manual delivery, our Ops teams had time to digest what change was coming, prepare for it, adjust to it and get it implemented smoothly (or at least pretend to). In the current day of Agile construction and DevOps delivery, I might have moved six new builds into production in the time it has taken you to read this article. Ops has no time anymore to digest and prepare for change, so if the quality of that change is not significantly better than it used to be, the angry-phone-call potential goes up exponentially. And yes, there are technical remedies to ensure quality (I will reserve discussing those for another time).
But the tradition of how we compensate remains in place. We comp Dev to innovate, and we comp Ops to remain stable. This makes not only silo thinking, but also mortal enemies during the adoption of DevOps into an organization. All of a sudden, each views the other as potentially causing harm to their ability to continue earning compensation. And they are right to see it that way, as HR has yet to alter the models to create partnerships instead of combatants.
To fix this, the compensation models must be adjusted to pay Ops teams to get involved in the software development lifecyle (SDLC) process from the upfront. The Ops teams should have extensive visibility into the testing cycle of new changes and how that works, and become the organization always asking Dev how it can “prove” its change works. The Ops teams must become the in-process auditors of the Dev teams. This keeps the Ops teams from being compensated for conditions that only exist at the end of the SDLC process, where they feel “dumped-on” by more rapid change delivery. The new comp model for Ops must focus on the number of changes successfully introduced into the production environment (i.e., the quantity of change that works), not just the stability they traditionally maintain.
The Dev team, on the other hand, must begin to be compensated only for after-action success. Dev teams must look at their code after delivery to production—and only if the code is validated (i.e., it works), is adopted by its intended customers (a completely different angle) and performs within expected timing parameters (i.e., it scales well). This becomes the criterion that is the foundation of the Dev team’s measurement of success. This puts an Ops set of glasses on the programmer instead of making him blind to those business issues. The Ops teams become the key advisors to the Dev teams on how to prove quality. And the joint discussions lead to a more sensible monitoring strategy, which is needed across the board.
There’s Compensation, and Then There’s the Bonus
Changing the point of view for the compensation is only a start. Next up: Begin to alter bonus compensation to a more lean model. Instead of large annual bonuses, where the basis for achievements often are colored by my boss’s mood over the last six weeks, organizations should increase the frequency of bonus payouts to match the duration of epic completions—or, if possible, of sprints. The idea is to tie meaningful bonus payouts to the length of time it takes to produce only successful change. If the change works, the company offers additional bonus comp; if not, then the team must improve to secure the next bonus opportunity.
Tie the bonus duration (whether to epics or sprints) to meaningful business impacts. In this way, each contributing employee understands “why” the work he or she offers has meaning to the company or its customers. This kind of bonus system allows for extreme flexibility in the priority of what teams work on throughout the year. Bosses can change priorities or shift them to match what executives see on the horizon. Employees are not tied to some list of annual work accomplishments that in a DevOps rapid world they had little shot of achieving unless they did them in a week. Instead, they have real-world goals, tied to real world time frames, where employees see the reward as they succeed at the work.
Organizations can add even more bonus comp this way if over-and-above work is needed. And “if” organizations compensate both Ops and Dev only for successful change (which implies quality in the SDLC and stability in Ops) they create a new rewards system that enables adoption rather than repel it. Beyond finding the talent they need, organizations need innovative methods of retaining them. Substantial rewards in real time might just be one way to do that.
Finally, being able to see the business impact of the change the joint effort of Dev and Ops teams produce allows for better ability to offer “appreciation” ideas as well. A simple “thank you” goes a long way in employee satisfaction, and when the executive and the team member understand “why” they are being appreciated, they go even longer. DevOps is not just a catalyst reserved to the “techies” who practice it; DevOps is instead a new way of thinking about how a company innovates. This kind of infectious thinking can easily spread to tertiary support organizations such as HR and Legal, if you let it.
To continue the conversation, feel free to contact me.
DevOps and continuous delivery are the same thing, right? Far from it. We peer through the fog to understand the difference between the two related but distinct methodologies.
We live in an era of apps. More and more of our personal, social and work life is governed by apps—communicating with one another through social apps, booking cinema tickets through your mobile device or scheduling a business trip through the airline app. This digitization of services has arrived. Organizations that understand this sea change have a unique opportunity to jump ahead of their competitors, acquire customers more quickly and grow revenues.
The battleground for this service development is DevOps. It’s not a product, but rather a methodology for helping organizations build teams and build software. DevOps is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other IT professionals while automating the process of software delivery and infrastructure changes. It creates a culture and environment where building, testing and releasing software can happen rapidly, frequently and more reliably.
Organizations are marching with their feet. According to a recent study, market demand for DevOps skills is growing, and DevOps engineers are among the highest paid IT practitioners today. And it looks set to stay that way. Research in the U.K. shows that 30 percent of 300 organizations surveyed in the run up to the 2015 DevOps Summit had either started to implement a DevOps strategy or had already merged Dev and Ops, with another 13 percent planning to go down this route.
Two Related Children: DevOps and Continuous Delivery
So now we arrive at continuous delivery. If DevOps is about accelerating the delivery of new, high-quality services, surely continuous delivery is the same thing, isn’t it? In other words, ensuring a steady stream of new services? You might look at it as a manufacturing production line: DevOps is the machine that builds the service, while continuous delivery is the conveyer belt that rolls the services off the production line—one big unified service development cycle.
That’s not the case. Although the distinction between DevOps and continuous delivery can be a little grey and cause confusion, there is in reality clear water between the two. Continuous delivery has become an essential ingredient for teams doing iterative and incremental software delivery. It is an approach whereby teams ensure that every change to the system can be released, and that any version can be released at the push of a button. Think of it this way: Continuous delivery aims to make releases dull and reliable, so organizations can deliver frequently, at less risk, and get feedback faster from the end users until deployment becomes an integral part of the business process and competitiveness of the enterprise.
Continuous delivery and DevOps do share common traits, though. Both methodologies are aimed at agile and lean thinking: each delivers small and quick changes; each relies on tight business an IT collaboration; and each share the common goal of faster time to market for new services.
However, the goal of DevOps is merge dev and ops roles together—and the processes they manage—to achieve business goals. It’s all about a common, shared culture and enhanced collaboration. About clearly defined business processes.
Automation has an essential role to play in both DevOps and continuous delivery. Automating DevOps helps your organization manage and secure continuous delivery to the business and IT users. The result is a faster, low-risk migration to a new world of composite applications, faster business innovation and better support for rapidly changing business process needs.
In part 1 of this series we talked about server provisioning tools. In that post, we mentioned some of the more prominent application provisioning tools on the market with regard to integrations. This post will focus on the application layer of full layer automated provisioning (FLAP).
Application Provisioning Who’s Who
Application provisioning technologies most likely are familiar: Puppet, Chef, Salt, etc. We’ll take a look at each, how they compare in the areas of control mechanisms, operating systems support and ease-of-use features (integration with server provisioning tools is mostly covered from a server provisioning tool perspective in part 1 and part 3 of this series, because integration must be managed in the server provisioning tool). A blog is too short to go much deeper than that, but this will help you determine which ones should be considered in your environment. Information here is valid as of the day this blog is written, but the market is moving fast, so don’t hesitate to check at the provided links for improvements. Because this topic is a popular one, don’t hesitate to look for up-to-date information provided by other sources, either.
It is worth mentioning that all of these tools are moving to support configuration of network equipment. For a guy who spent years worrying about programmability and interoperability at a networking gear vendor, that is very cool news. I haven’t had time to actually try any of that support yet, so I’ll leave evaluations to you.
A word of advice: This entire marketplace has cutesy-naming disease when it comes to really important concepts. While you are grimacing at “recipes” or “grains” or whatever, learn that terminology. It may be cutesy, but the heart and soul of these products hides behind those names and the details of what concepts they encapsulate. During your evaluation period preferably—but certainly once one is chosen—you should get those terms and their meanings down pat. It will make your life much easier. The same is true with the variety of scripting formats. They’re all communicating basically the same information, but so do English and Japanese—knowing one doesn’t automatically makes you an expert in the other. Take the time to truly understand the scripting language for your tool of choice.
Ansible is a full-app provisioning tool that allows users to install and customize applications through a combination of a command line and scripts. The tool is well-documented, easy to install and works very well at its designed task. Ansible offers support for Red Hat and Windows running on Docker, OpenStack and AWS. Notably missing is VMware support, but since the machines are available via IP and the OSes are standardized, VMware virtuals can be managed/orchestrated with Ansible.
The open source version of the tool comes with command line interfaces and management is handled via command line and editing scripts. The premium add-ons such as Tower include UI, analytics and programmable APIs.
Since the purchase by Red Hat, it is reasonable to expect an increased focus on RHEL for this tool and a commensurate reduction in non-Red Hat OSes. Not that this is assured, but history says it is a likely long-term result. If you’re a Red Hat shop, Ansible is a good place to start—though looking at Satellite—and trying to predict how Red Hat’s various open and add-on provisioning tools will evolve and emerge—is a good idea for any evaluation at this point in time.
Chef is one of the original application provisioning tools out there. It is managed through command line and scripts in the open source version, and through these plus a GUI in premium. Unlike Ansible, many Chef APIs are bundled with the open source product, only APIs for premium services like analytics and reporting are premium.
Chef supports most variants of Linux and Windows, with most cloud and virtualization vendors available as platforms. In terms of completeness of support, it really is up toward the top of the list.
If you have a largely heterogeneous environment, Chef is a good tool to look at for automating application provisioning—be it for deployment, refresh or continuous delivery.
Juju advertises itself as an orchestration layer above the other products in this space. While there is a small amount of truth to that claim, most of what you can do in terms of orchestration with Juju can be done with the others. In fact, Puppet has kind of found a home in entire complex multi-server systems. Juju comes with GUI in the open source version, but the product does not have a traditional API.
Juju supports Ubuntu and Windows, and is interoperable with most of the major cloud providers. Deep server provisioning integration is really only provided for MaaS (another Canonical project), but users have made it work with other server provisioning tools. Juju scripts (charms) are primarily written in Ruby, though other languages are definitely possible. For Ubuntu shops, Juju would be the item I’d place on the top of the application provisioning list. For Windows shops, it would definitely be worth considering. For Red Hat shops, other solutions are probably more appealing.
Puppet should need no introduction to readers of devopsy.kinsta.cloud. The standard in application provisioning, it uses scripts and command lines to deploy applications with a pretty broad base of user support that equates to lots of user shared scripts. Other vendors in this list also have vibrant communities contributing scripts, but Puppet, by sheer weight of numbers (and marketing), has a larger selection available.
Supporting all major operating systems, Puppet’s documentation is solid. The only quibble is that you have to be careful when coming from a search engine about whether you’re on the open source documentation or the Professional Edition documentation. Puppet comes with APIs, though the GUI is only available in the commercial version. Puppet is the progenitor of Razor, the server provisioning tool, and while it has been used with all of the provisioning tools in the first iteration of this article, expect it to work best and provide the most value with Razor going forward.
For highly heterogeneous shops, or shops that include multiple server provisioning tools, Puppet is a solid bet. While it will work fine in other situations, the value-add of a more focused product is worth considering if your environment maps to another tool’s strengths.
Salt is an interesting member of the application provisioning crowd. They don’t share name recognition with some of the other players, and yet have a respectable user base. Their product is certainly comparable to the others in this market, and the user community is pretty active. Not being deeply associated with any of the server provisioning tools, SaltStack has been integrated with most of them.
The Salt Master runs on almost all flavors of Linux, and can manage all major Linux distributions plus Windows, OS X and Solaris.
Like all of these other tools, SaltStack can manage any machine that is accessible over the network and is running a supported operating system., Additionally, the dev branch has Salt Cloud built in to the Salt Master, which spins up cloud images, automatically puts them into salt and starts managing their configuration. Expect that this type of functionality will become the norm for the market (Juju does some of this also, and others have varying levels of cloud support)—it just makes sense, and it compliments server provisioning. There is a tiny bit of overlap in the VM space, but then you can choose whether to use server provisioning or Salt Cloud to handle those instances.
For highly heterogeneous shops that wish to use best-of-breed (actually “best fit” is a better name) server provisioning along with a solid application provisioning tool, SaltStack is a heavy hitter worth considering. If you have a more specialized environment focused on a single OS, SaltStack will work, but its advantage of broad support for both OS’s and server provisioning tools will not come into play.
Satellite is Red Hat’s provisioning solution. Unlike most of the others in this list, it is a cross between server and application provisioning. In actuality, it is that “layer above provisioning” that Juju positions itself as. Under the covers, Satellite (as of release 6.0) is using Foreman for server provisioning and Puppet for application provisioning. The reason it still makes this list then is because it does do application provisioning, and does orchestration too. You can use it to spin up MariaDB or OpenStack, and it is a marketed and supported product, so it fits the list.
As one might wholly expect, Satellite supports RHEL. It can manage instances of RHOSP and RHEV, along with VMware and Amazon, and does bare metal through Foreman also. Satellite is deeply tied to Foreman and Puppet, so if you have issues using this tool set, it’s not going to be your first choice. Because Satellite extends and improves Red Hat Network access, for Red Hat shops it should be No. 1 on the list of viable options, but if you use a lot of non-Red Hat machines, including (as of version 6) Fedora and CentOS, it is less appealing. It used to be that Spacewalk was the viable option for those CentOS and Fedora installs, but now they are separate code bases, meaning you have to maintain two different systems. At that point, it is worth looking at other options available on the market.
The newer version 6 architecture is designed to provide a platform moving forward that is more powerful and adaptable, so expect to see good things from it if you are a Red Hat customer, possibly even cancelling out the disadvantage of no support for CentOS and Fedora.
Summary
All of these products are very good in their best environment, and most of them will give you pain in their very worst environment. The question is, which environment do you have? While only employees can answer that for a given environment/tool combination, hopefully there are some pointers here—and elsewhere—that can help make the decision.
In the third and final installment, we’ll look at the combination of server and application provisioning to offer full stack automation.
If you ever wanted to learn about Rugged DevOps (some call it DevSecOps), sit down for a spell with Shannon Lietz, Ian Allison and Scott Kennedy from Intuit. We discussed a number of important topics including internal war games, culture hacking, gamification of Rugged DevOps and starting as a small team. There are 100 gold nuggets in this conversation for novices and experts alike.
Derek Weeks: I have some of the Intuit DevSecOps team here with me today. We’re going to talk to them a little bit about Rugged DevOps and how things work over at Intuit. Let’s start with some introductions.
Allison: I’m Ian Allison. I help run the Red Team at Intuit, which is, I guess you’d say, an interesting way of taking control of security at our company. We try to get ahead of the attackers by basically being the attackers. We’re essentially ethical hackers. We go after all of our own stuff to make sure we can find where the deficiencies lie in all of our software.
Lietz: I’m Shannon Lietz. I’ve been working at Intuit 3 1/2 years and helped to found the 24×7 DevSecOps capability at Intuit, leading the Red Team, our security operations capability, our cyber SOC, and what we also consider “blue teaming”: being able to hunt for defects.
The organization has really had to transform how we do software development because we’re a 30-year-old software company. We are now seeing the traditional way of putting together software really embracing DevOps. For us, it’s been exciting to really work in the industry with Rugged DevOps, trying to help build security into the DevOps movement.
Kennedy: I’m Scott Kennedy. I run the forensics and threat intelligence part of cyber work.
Weeks: Shannon (@devsecops), tell me a little bit about software supply chains and how that vision of software development has impacted the way you see things at Intuit.
Lietz: That’s really a great question. It was interesting when Josh Corman and I first talked; we had a lot in common. One of those things was the software supply chain. What I really love about the concept is being able to have processes driven a certain way so that you can reduce defects.
Having worked for Toyota in the past and understanding the supply chain mentality, you get a sense of how you could put something together better, incrementing on it, figuring out how to share that process, and then really figuring out what things are important. Having that notion of fewer, better suppliers was really a core concept.
I love the idea of transparency, building things a certain way, and really getting into continuous improvement. You need to look at things from an opportunities perspective—making sure you’re not just looking to make things perfect. You’re looking for those opportunities to improve over time.
https://www.youtube.com/watch?v=nRaURhaTH9o
Weeks: As we think about Rugged DevOps within your security team, how do you measure the success of what you’re doing? What kind of metrics are you looking at that matter to the business?
Lietz: We measure everything. For example, mean time to remediation (MTTR). Once somebody finds a defect, we analyze that defect from the time it got into the supply chain to when it actually gets resolved. We track everything from mean time to remediation to when the ticket was created, to being able to look at when the code actually got published, to when it actually got found, and then we work on those things over time. We really try to uplift.
____
Once somebody finds a defect, we analyze that defect from the time it got into the supply chain to when it actually gets resolved.
____
We leverage JIRA [Software] just like a software development team does. We register our defects and figure out how to get development teams to take responsibility for those ideas. It goes through their process of release and regression testing. As part of that, we look back to see where our opportunities are.
As an example, we started out where things may have taken weeks. We then reduced it down to days and ultimately got it down to hours. We’ve seen defect resolution where it’s now minutes. When it’s something we’ve discovered that was just a mistake by an engineer, we realize “mistakes do happen.” We found that our cycle times also help us to find fault stack vulnerabilities in real time because we get to do end-to-end testing more aggressively utilizing this method.
Weeks: How has consistency in your operations helped with Rugged DevOps and has it fragility within the organization?
Allison: One of the things we do is to utilize a golden image for all of the AMIs (Amazon Machine Image) we use, for all of our customers, and we require everybody to use these AMIs. We’ve also built some really interesting automation around scanning these AMIs. So one thing we realized quickly when we first started native U.S., when we try to do full vulnerability scans against another system, if it’s set up to autoscale, we all of a sudden have 50 systems. Right? We can’t … it’s really hard to do a full vulnerability scan right against the system, so we came up with a way to share back all of the AMIs with a special account. Then we bring those up and we scan them. Then we grade them.
Based upon the vulnerabilities that are found, you’ll get a letter grade, like A through F, based upon the system you have. While we always strive to have our base image be an A, people continue to run on older images. But they get graded, and those grades get pushed up, so everybody in their org structure gets to see what the grade is for their account. I think by being a little standardized, basically with these images, lets us know what’s in everything, and we have a grade for everyone. It helps everyone have a really good idea of where they stand when it comes to a security standpoint.
____
Based upon the vulnerabilities that are found, you’ll get a letter grade, like A through F … so everybody in their org structure gets to see what the grade is for their account.
____
Weeks: That’s not only a grading but a policy enforcement governance kind of role that grading plays. How rapid is the feedback loop in that grading system for the teams that you’re working with?
Lietz: It’s really quick, and we’ve discovered through some science that having component-based resources like AMIs provides us with an advantage when doing things like remediating vulnerabilities. Using AMI-based resources, we have seen that when there’s a defect in it, we can find and remediate all of the defective AMIs quickly. That improves everyone’s security across the company.
So if you’re just picking out really good components, keeping track of those components and adding security into them, then you’ll actually see a different effect across our pipeline. A single change can actually have a dramatic effect on reducing the problems within the pipeline.
Allison: It’s really interesting. This morning I got an email from somebody that said, “Why did our baseline AMI go from an A to a C today?”
We had just received notice of a new vulnerability. Our stuff caught it, we scanned it, and we pushed the grade out to our portal where all our customers go to look at the grades. Our customers were able to see that change quickly.
They could now say, “Wow, it changed from an A to a C in less than 12 hours.” I think the feedback is really important. The other important thing is that we have people going and looking. I wouldn’t be getting emails about why has this changed if people aren’t actually looking and wanting to make their grades better.
Weeks: You mentioned customers. Are these internal customers?
Allison: Internal.
Lietz: Yeah, for our development teams, we as a security team really have changed how we think about things. It used to be that the security team would go out and govern. Basically, you got the fear of the security team coming in, descending upon you.
We’ve really changed how that happens within our organization. We grade our resource components, and we grade the way in which our applications come together. That changes how developers want to operate because they really want to figure out how to get better grades in security. And it creates a learning dynamic that incentivizes somebody to improve continuously.
___
That changes how developers want to operate because they really want to figure out how to get better grades in security. And it creates a learning dynamic that incentivizes somebody to improve continuously.
___
Weeks: Does it create a competitiveness or gamification of who has better grades?
Lietz: Absolutely, which is why we did it in the first place. To your point there, gamification is something where when you start to grade components like that, you can actually start to leverage a leaderboard concept. We do have leaderboards as part of this. We have APIs where you can actually pull down your grades and include them in your automation. With these, you can make governance decisions.
If you sort of have that “game afoot,” right, your leaders can then ask for specific grades within their pipeline. That up-levels the system, and you just see a continuous improvement lifecycle come to bear. Ultimately, you see fewer defects, and ultimately, you get to the notion of Six Sigma in our way of thinking. DevOps is really about continuous improvement and embracing automation. Embracing that concept allows us to get to fewer defects faster.
___
DevOps is really about continuous improvement and embracing automation. Embracing that concept allows us to get to fewer defects faster.
___
Weeks: As you embraced continuous practices and DevOps practices, were there points when you realized certain old ways of doing things weren’t going to enable you to move forward?
Kennedy: In looking at the progression of what we’ve been doing, one of the decisions that was made in Intuit and one of the things that I saw was really unique was the way they decided we were going to migrate into AWS. Our idea was to have the chaos team be the first people out, and that’s the security team. So the security team was the one that was going out and finding out how to use each of the products that AWS has and creating the concept of whitelisting. Each product was rated as to whether or not it met security’s requirement.
Therefore, no team can go ahead and pull down this new cool tool that AWS released yesterday and use it in production because it’s not been “whitelisted.” That can go into their scoring. Their scoring is not only used by the development teams but also is useful when reporting to the board. When the board asks, “How are we doing as a company across the entire organization?” we can say that product A got a lower score than product B, and then they turn to the VP in charge of it and say, “Well, why?”
___
When the board asks, “How are we doing as a company across the entire organization?” we can say that product A got a lower score than product B, and then they turn to the VP in charge of it and say, “Well, why?”
___
We decided to not rush into the cloud but to take a careful, considered approach and migrate in a very intelligent and well-thought-out way. At the same time, we gave the chaos team the ability to make the mistakes and grow and learn, so they can immediately turn around and share the mistakes with everyone else. They could say, “Hey, these are the things that didn’t work for us. We came across a lot of problems, especially when you look at things like accounts and account roles.”
How do you control when you have thousands of accounts and you need to have some sort of administrative control?
You can either have a gigantic effort to force your namespace and your Active Directory to be the source of control. Or you can use the vendor-specific tools like IAM and have each account have their own separate islands, but with the concept of cross-account roles, you can then do remote administration from a centralized account. You have it locked down. You have the capability to have a restricted group and be able to remotely go in and do the necessary actions.
That also gives you an audit trail. That also gives you multifactor built in because the AWS products get those things added to them.
Lietz: I think when it comes down to it, I think culture-hacking your environment can have a profound effect, especially when you’re going through a DevOps transformation.
Weeks: What is culture hacking?
Lietz: That’s a great question. We use it when really trying to figure out how we as a security team can change and transform. A lot of the things that take place in a company are really based on traditional processes: What has worked before, and why would we change something that is working, right? If you’re really going to go into an innovative frame; if you’re really going to get into that next-generation innovation; if you are trying to figure out what’s going to work in that … it’s never going to be the thing that is working. It’s going to be the thing that you’re going to learn as you go to that next step.
Culture hacking is really about looking at the people who are operating right now and trying to figure out how you’re going to help them go from A to B, making that change. What is that the experience going to be like?
What we have done, to Scott’s point, is we’ve forced our security team to have empathy for the DevOps teams. We go through the process of developing something in the cloud, utilizing it as a method of taking their paranoia and trying to balance the notion of getting something done within a specific time frame. We try to really wrangle what it takes to do those things securely and safely but, ultimately, still be able to deliver for the business.
I think that culture hacking really comes into play when you’re trying to figure out how to move somebody from the rock they’re on to the rock you need them to be on and trying to figure out what those mechanisms are.
___
Culture hacking really comes into play when you’re figuring out how to move somebody from the rock they’re on to the rock you need them to be on.
___
Weeks: Part of your security practice is looking at open source and third-party components and your own binaries. Can you shed some light on how Intuit is using Sonatype solutions to better manage those vulnerabilities?
Lietz: Yeah, Sonatype Nexus is a fantastic platform. We love the Nexus repositories. We love how you guys put together a community. We learn a lot.
A lot of our DevOps practice is working together with it. We’ve put together our Nexus repositories to do code signing and figuring out how to really secure our pipelines a certain way. We are taking advantage of the fact that we can pick up components, track them and then scan them [for known vulnerabilities].
That’s allowed us to reduce the defect count that goes to production. Actually scanning and looking for vulnerabilities within our components and our open source libraries allows us to make better decisions about what we’re including in our software.
Weeks: When you govern what open source, third-party or proprietary components are being used by developers, is there any kind of feedback from the teams saying, “Hey, you’re restricting my behavior, not improving my innovation”?
Lietz: What we’ve found is that the notion of security approvals, exceptions and gates really doesn’t work. Quite often, you just create a culture where developers are going to go out and do it, and then you’re going to find out about it. When it comes to really partnering and being boundary-less about how you think about security in your business, it’s all about transparency. It’s all about benefits. It’s creating things like a security markdown file within your repository manager. It’s about taking responsibility and accountability for the things that you’re doing from a security perspective in your development process. It’s ultimately having an attacks.md file, keeping track of what’s out there, keeping track of your open source, understanding what components you’re leveraging, and why you made the decisions that you made to bring those things into your project.
___
It’s about taking responsibility and accountability for the things that you’re doing from a security perspective in your development process.
___
At a top level, all of those things work. But really having tools that can help the decisions that were made by some of the other open source programmers that you’re getting contributions from is really necessary. All of the things that they might be deciding are also part of your decision tree, and ultimately, you’re rolling all of that and bundling it together. The attack surface is not just the decisions that your team is making, but the ones that you share across the code base that you’ve got.
Weeks: Your practices are very mature. You’ve clearly developed them over a long time, and some people watching this might think, “Well, Intuit’s a huge organization,” and it may be daunting to them if they haven’t started down the path of Rugged DevOps. Can you be a small team and have success in these kind of practices?
Lietz: We’re not exactly a huge organization, but we are relatively large in size now. When we got started, I believe I was one of maybe three people that started this, only a couple years ago. We have hired into our group pretty extensively to help grow it, and some of the things that we’ve done have really allowed us to operate differently, to bring in people and have them immediately be successful. Our practices allow someone “day one” to be able to work with the environment, to be able to develop code, to be able to contribute code that week.
We do things like weekly demos, where we actually do video demos. A person has to come in, program something, secure something, operate it and create a demo, all within their first week. So having the right bar for those folks is really important, but more importantly, our Red Team leader here (points to Ian), he came in and just is amazing, has created a Red Team pretty much out of thin air. So is having somebody from forensics, who’s just done an incredible job to help us, to make it so that we have a life cycle where we can snapshot something and be able to learn from it when it’s actually offline.
___
A person has to come in, program something, secure something, operate it and create a demo, all within their first week.
___
Those are the types of practices where you start to extend yourself past the normal baseline practices of processes today and really think past that about how you’re going to support innovation. You get into it very quickly. You get a learning culture. You get people who know that making mistakes—and figuring out how to learn from them—is okay. That’s a really important part of that actual culture that you’re putting in place.
Allison: Yeah, I was going to say, it’s all about iteration, right? We started small, and we just continually iterate on what we’re doing to try to get better and be better at what we all do.
When I first started this journey, I was a security guy—a pen tester. It was always the developer’s fault. Developers always made the mistakes. I always had to clean up after them. But after six months of developing Ruby APIs and Ruby and working my butt off in code, the empathy was there.
I really understand what the developers are going through and why they make the choices they do. But I think by allowing us to help them, by creating tooling that allows them to self-serve, to understand it without making them … helps them make themselves more secure without them having to become a security professional. I think that’s kind of our ultimate goal.
Shannon: Being friendly hackers, right? Basically going out and attacking them so their applications don’t get attacked by external attackers is really part of that frame.
Kennedy: The Red Team shift at the company has been profound because you see how people react. When the Red Team started, it was not as well shared, and a lot of people suddenly were very upset that they were attacked by the Red Team. But when it was pointed out, “Well, what would you rather have happen? Would you rather have somebody in China do this to you, that didn’t work with you, didn’t sit next to you and help you fix the product, or would you like to have a friend who, by the way, their job is to attack?”
When we went through several drills and actually practiced the muscle of defending the company against an attack, people were upset. “Oh, I had to do all this work.”
My response to them: “Well, you did the right work.”
“You did the right thing. You saw something bad. You did it. You did good. You practiced the muscle. Now when it happens again and it’s not the Red Team, I know that you’ll know what to do. You know that the process works, and we can actually defend the company faster and more securely.”
___
You know that the process works, and we can actually defend the company faster and more securely.
___
Weeks: Yeah. That’s an incredible story. Thank you for sharing it.
My final question: If you could pick a superpower in dev, security or ops that you would have in the organization, what would it be?
Allison: To me, they’re all alike; they’re the same, right? That’s what we do, DevSecOps, right? We try not to actually separate them out because I think once you start to separate them out, you start to lose perspective.
Kennedy: Yep.
Allison: There’s a good thing about having them all be one thing, so I’d choose them all.
Kennedy: It’s been pretty consistent. DevSecOps is the answer. What was the question? (Laughter)
Lietz: I think the reason we went out and created DevSecOps was just simply to change how we thought about doing development and technology—and to really to get ahead of it, to realize that attackers weren’t setting up appointments or meetings to help you figure out how they were going to attack your software, and so then why were we? Why were we operating at a fragile level?
I think that the superpower that I would like to have is DevSecOps because I know that we are going through the process of creating a less-fragile capability in security that will allow us to get ahead of attackers, make it much harder for them to go after the software that gets built, and we’re seeing those improvements. That’s actually a great thing.
Weeks: It sounds really exciting, and it’s very cool, so thank you all very much. I really appreciate it.
All: Thank you.
If you loved this interview and are looking for more great stuff on Rugged DevOps, I invite you to download this awesome research paper from Amy DeMartine at Forrester, “The 7 Habits of Rugged DevOps.”
As Amy notes, “DevOps practices can only increase speed and quality up to a point without security and risk (S&R) pros’ expertise. Old application security practices hinder speedy releases, and security vulnerabilities represent defects that can leave a company open to cyberattacks. But DevOps practitioners can leap forward with both increased speed and quality by including S&R pros in DevOps feedback loops and including security practices in the automated life cycle. These new practices are called Rugged DevOps.”
Aater Suleman, CEO & co-founder at Flux7, is an industry veteran in performance optimization on servers and distributed systems. He earned his PhD at the University of Texas at Austin, where he also currently teaches computer systems design and architecture. As a faculty member at UT-Austin, Mr. Suleman focuses on optimizing DevOps architectures involving the AWS cloud. Along with the Flux7 consultant team, he works at every step of the dev flow, from creating local developer environments using Vagrant and Docker, to continuous integration and continuous delivery using Jenkins or RunDeck, to configuration management using Chef/Puppet/Ansible, to creating efficient systems for deploying new code from Git using Capistrano/Fabric/bash. His background is as an architect, and he has experience with both hardware and software. This combination brings a unique perspective and broad view to development and operations.
