devopsy.kinsta.cloud

Category: Features

DevOps Drives Innovation, Enterprise Differentiation

It’s hard to believe that it’s been more than a decade since Nicholas Carr wrote his controversial article “Does IT Matter,” where he argued that IT had become such an integral part of business that IT disappeared into the infrastructure in the way roads, rails, electric utilities, and engines have in industry. IT, like such infrastructure, became so commonplace that it no longer provided a competitive advantage.

While Carr nodded to the transformative nature of cloud computing, he didn’t see the competitive advantage made possible by businesses that could develop rapidly and innovate. And that’s at the heard of what DevOps, and the associated processes and collaboration is all about. And enterprises that are able to identify opportunities and rapidly innovate using IT more effectively than their competitors would succeed.

However, to succeed, IT investments need to be viewed as more than just ways to cut costs in applications or IT infrastructure. IT has to be looked at as a way to actually drive innovation. And DevOps itself needs to be viewed as more than just a way to reap increased efficiencies out of existing processes and technologies: but as a way to also more tightly couple business needs with IT efforts. One of the most effective ways to do this is to build applications that matter to business users.

When it comes to building great business applications, enterprises often don’t get apps right – however that is a gap DevOps can help to close. When it comes to building apps, organizations often try to build the same apps on mobile devices that they’ve been providing on desktops for years. Or, they try to build applications that attempt to accomplish too much, and end up being overly complicated and largely unused as a result.

“Businesses don’t always get apps built right,” says Brian Katz director, head of end user experience and innovation at Sanofi. “In fact it’s common that they don’t. They try to develop complicated apps that people don’t actually use instead of developing apps that users really need,” says Katz.

“If good app design isn’t part of the equation, all you get is operations and development tools and processes that are optimized to deliver poor app quality,” says Katz. “Design isn’t always part of DevOps, but I think it should be. But design is one of those things that is critical to driving innovation, but people don’t always think about it that way,” he adds.

One way DevOps can help address app development speed bumps is to make sure that these teams are integrated more closely with those who are designing the apps and know what the end users want to do – and how they want to do it. This is especially true when it comes to mobile development, when users may have seconds to input or grab data while they are on the go.

Chris Cera, CEO at mobile and web product development firm Arcweb agrees, and says that testing and design are crucial to building the apps users want to use and to driving IT innovation. “You want to test the user interface and get feedback. It’s so crucial to developing the apps that will not only be apps users want to use, but it’s the key to driving innovative design too,” says Cera.

Those enterprises that do manage to successfully integrate great app design with their DevOps efforts will prove themselves formidable competitors in the years ahead. “The organizations that embrace DevOps are seeing what good happens when the business and IT collaborate, and in the years ahead we will probably see more technology leaders show how IT can lead the organization forward,” says Gene Kim, co-author of The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.

The data are starting to prove such claims valid. According to the 2014 State of DevOps Report from Puppet Labs, DevOps enterprise adoption is increasing and those organizations that embrace DevOps are deploying code 30 times more frequently and with half as many failures as non-DevOps enterprises. That data helps to support the findings of a CA Technologies study published last year, TechInsights Report: What Smart Businesses Know About DevOps, which found that respondents claim to experience a 17 to 23 percent improvement in certain business outcomes, such as increased revenue, and faster time-to-market.

That all certainly sounds like IT as a way to innovate and differentiate. While the day will come when “IT Doesn’t Matter” – that’s a time that is likely decades away.

December 2, 2014
Choose Digital Chooses Jenkins For Continuous Deployment

Challenge: Shrink Time To Innovate

Choose Digital offers a white label media store solution (think iTunes) for enterprises that want to reward brand loyalty with opportunities to use miles, points, or cash to buy digital products. As a competitive media company, Choose Digital was eager to test and adopt innovations to its software platform as fast as its developers could produce them. The media firm was ready for a model that lifts the burden of maintaining tools in-house.

“We were looking for a way to move from a straight compute environment to a DevOps approach that would enable us to push lots of code and new builds out daily while quickly iterating a lot of new ideas for improvements to our digital marketplace,” says Mario Cruz, Co-Founder & CTO, Choose Digital. A managed service could also relieve the headaches of supporting all the development applications internally. Choose Digital chose CloudBees’ Jenkins , a Java-based pipeline and orchestration tool for continuous deployment, which was also available as a hosted service, to meet those needs.

Solution: Simplification

Once Choose Digital made its decision, CloudBees made it simple to get up and running. “We opened our account with CloudBees and were productive with our Dev and Stage environments in two days using a fully functional Jenkins trial implementation,” says Cruz. Choose Digital moved production into the cloud 30-days later.

Jenkins enabled Choose Digital to find and fix bugs and try new things fairly quickly, without waiting for monolithic releases. “Jenkins is how we innovate, test our code, and enable the coder to immediately fix what he breaks,” says Cruz. Jenkins provides continuous integration services for software development and enables Cruz’s team to run build tests and functional tests.

“Jenkins gives us a quick snapshot so we can find broken code right away, not five weeks from now,” says Cruz. Enterprises that are still using older approaches don’t typically have time to fix small bugs, and certainly can’t fix them right away.

Choose Digital uses Jenkins in the cloud (DEV@cloud), which CloudBees fully manages and hosts on AWS EC2 http://aws.amazon.com/ec2/. Cruz doesn’t worry about maintaining the environment or whether his media platform has enough space or capacity. “CloudBees manages all the updates and issues with Jenkins CI. They support all the plugins for Jenkins, even if it is not their own plugin,” says Cruz.

As a managed service, Jenkins simply works from machine to machine. “My developers can focus on the product. They don’t have to worry about whether they have the right software build, the right apps on their machines, or the right versions of those apps,” says Cruz.

“As a result, development is up to 5 times faster in our continuous deployment environment than it would be if we had set up our own hardware or used an IaaS solution. Things that could take months or years we can now do in days,” says Cruz.

A Bump In The Road

As Cruz and his team moved to Jenkins, the biggest challenge was communication. “The toolset is not the challenge, it’s having everyone thinking with and getting used to the new DevOps mindset,” says Cruz; “while we were fully functional in two days, it took two weeks to get used to the new mindset.”

Today, Cruz’s software team communicates across stakeholders and team members in a DevOps approach that leverages Slack https://slack.com/ (IRC) and HipChat https://www.hipchat.com/ plugins for Jenkins. “DevOps is more a culture than it is tools; DevOps is all about communication. We communicate to our team, to the QA department, and to product managers,” says Cruz.

Current State: Everyone Knows, So We Can Fix Everything

Jenkins brings transparency so that what only some could see—such as what a developer has broken—becomes everyone’s knowledge. “Things get flushed out pretty quickly. By using Jenkins CI, you see the hidden things that no one knows about quickly,” says Cruz.

Cruz has some realistic advice for his peers: The sooner you get to an environment and mindset like Choose Digital uses, says Cruz, the sooner you will see the benefits and flush out bigger problems. “It’s not perfect the first time around but it gets better and better on every build. You get rid of the technical debt you didn’t know you had,” says Cruz.

November 25, 2014
Q and A: ITinvolve’s Matthew Selheimer on DevOps and Culture Change

The intersection of DevOps and culture is a common subject here at devopsy.kinsta.cloud. Is it tools and processes that help support the culture, or does the culture of an organization drive toolset and process building decisions? Perhaps a little of each. We recently had a talk with Matthew Selheimer, Chief Technical Evangelist at ITinvolve, a Houston, TX-based maker of IT collaboration and workspace management applications.

devopsy.kinsta.cloud: Tell us a little about ITinvolve, and the IT challenges the company tries to solve?

Selheimer: One of the core challenges that the founders of ITinvolve identified was the fact that organizations really didn’t have an effective understanding of their environments, and all of the associated dependencies. It’s a reflection of how today’s configuration management efforts, using their databases and discovery tools, fall short. They only filled in part of the picture, and a lot of times organizations spend a significant amount of money investing in CMDB initiatives only to have people not even bother to use the data. Maybe they didn’t trust the data sources, or they had their own private systems, spreadsheets and things they were accustomed to using, and they kept using those familiar toolsets.

The founders took a close look at what could be achieved if we apply the principles of Enterprise 2.0 to the way IT is managed. What if we actually embrace the notion that it’s all about people, process, and technology, and in that order, instead of not putting technology or the process first. That is, historically, the approach of many management vendors. So ITinvolve was founded on the principle of embracing people as the most important asset in the IT organization.

I know you’ve been in the industry for a long time, so you’ve lots of anecdotes you can look back on your career where it was either that the tribal knowledge created the great success – or it ended up creating the great failure. And when people left an organization, the business was at risk because it was so dependent upon tribal knowledge, or even a single person. So, when they left the company, there was a big freak-out because they knew how everything was configured and why it was configured that way and that tribal knowledge was something that couldn’t be lost.

devopsy.kinsta.cloud: With that dynamic in mind, are you seeing DevOps impact traditional CMDB usage? Are organizations using them widely, or are they still more a promise than reality?

Selheimer: I think these are real problems that organizations continue to struggle with, that there is so much risk involved in change because we’ve cobbled together all of these architectures and infrastructures over the years. That was the core idea behind the company. And the initial effort product was to take discovered data – CMDB data, SharePoint data, Visio diagrams, all the scattered data sources – and bring them together in a way that is visual, logically organized, and within an application that people want to use.

We were initially focusing more on the traditional ITSM market, but what led us to the DevOps market is that DevOps is the area that has everybody’s attention, because it is so much closer to what the business really cares about. At the end of the day, resolving incidents and delivering better customer support and customer experiences, are critical parts of the IT organization. But how do you actually get applications developed and employed, how do you modernize legacy infrastructures and legacy applications, which aren’t going to go away? How do you mobilize the front end of them and those kinds of things?

DevOps: Automation helps, but it’s not about just automation for the sake of automation. The danger there is you end up just accelerating bad process.

Selheimer: Kevin Behr and I were talking about this a couple of weeks ago. We concluded that automation is the last thing you should do. What you really need to be doing first is figuring out how to actually get the mindset shift necessary for people to start thinking about end-to-end flow? It’s not about optimizing a workstation, it’s about thinking in manufacturing terms. It’s about flow all the way across, and that starts with requirements and what we’re trying to achieve from a business standpoint. That carries all the way through to the actual deployment and their ongoing operation.

But there are huge communications gaps when it comes to attaining that end-to-end flow that are not being addressed by traditional communications – whether that’s conference calls, whether that’s email and instant messaging, whether it’s SharePoint sites. We see pockets of collaboration, so you have GitHub on the source code side of things. You’ve got some collaboration products that are doing some level of basic Wiki and other kinds of collaboration. But there is no real collaboration engine that’s really orchestrating that flow.

So we see that there are a lot of gaps in this DevOps tool chain. Things are still getting passed on down to operations, which doesn’t have enough clarity as to what is needed, why it’s needed, or when it’s needed. Organizations have been able to overcome these challenge on a small scale. A lot of the DevOps science projects that are out there haven’t fully seen this issue yet. I was just reading O’Reilly’s book DevOps in Practice, which profiles a case study with Nordstrom and a service provider for the Texas government.

I think that is a great book. It is a great example of the maturity of DevOps. So I can take a small project and a small team and I can say, let’s go stand up this project around how we are going to start doing middleware configuration and automation better. And bite off that little piece of it – and have some degree of success in being able to have infrastructure as code and that well-bounded box. But, once we start trying to think about taking this up to full-scale applications – once we start thinking about taking this outside of a new generation web company that’s grown from the ground up using agile principles but doesn’t have a lot of legacy – and start trying to apply this to JP Morgan Chase, or to Merck, or to big companies that have duct tape and baling wire holding all of these legacy applications from mainframe to everything altogether, that’s something different altogether.

I think that’s where these communication problems are really going to be amplified in a big way. And that’s what we’re seeing some of the dialog that we’re having with prospects and the marketplace – around how to really do DevOps at an enterprise level. How do we take it out of these small-scale projects where you can have this startup mindset and actually start applying that to 300 people scattered across five, six, eight different time zones and eight different nationalities?

devopsy.kinsta.cloud: How do you that? Is that culture? Is it learning how to work across silos?

Selheimer: Everybody has been saying that culture is the challenge, right? It’s the ability of the organization to adopt a different mindset; it’s the ability of people to think more in terms of the end-to-end process as opposed to their particular area. It’s the old silo versus flow debate. My position on that is we have to have silos, and we have to have expertise. There are still heart surgeons in the world and there will always be heart surgeons in the world. But what we need is this kind of T-shape. We need the vertical siloed expertise, but we need the horizontal flow coming across as well. And we have to tie the conversation together and create opportunities for serendipity and for people to engage and to collaborate in ways they haven’t been able to do before. That’s what success looks like.

Enterprises try to do this in an ad-hoc way today. It’s with the water cooler conversation, but generally that ad-hoc happens within a function of the organization, rather than a cross-function. So, one thing I recommend is to have your developers and system administrators have lunch once a week. Have your teams get to know each other better and understand each other’s work better. That’s what is important when it comes to culture change.

November 20, 2014
Detours Ahead: Overcoming DevOps Challenges

Saying that you’re interested in DevOps is another way of saying you’re interested in change. That’s what DevOps promises any organization. No matter the problems a company is trying to solve or the goals it’s trying to reach, DevOps depends on a willingness to embrace some big changes. Sure, it can help you streamline software deployments, but it requires shaking up a company’s usual processes and routines. As we all know, change can be complicated, so before your organization transitions to an agile culture, it’s best to prepare for the likely challenges ahead.

Breaking Old Habits for Continuous Delivery

Many of the procedures we follow at work each day are second nature, and what’s familiar carries a certain comfort with it. Even when we realize a more efficient process is within reach, old habits are tough to break. For an organization that employs hundreds, any change can be a challenge.

And DevOps offers no shortage of challenges. It entails overhauling a company’s deployment process, its communication patterns, its department structures and even the roles of individuals on a team. Yet unless an organization is willing to let go of its old habits, the benefits of continuous delivery won’t ever be achieved. When acclimating people to this new procedure, it’s important to convey that the old procedure – one team throwing code over the wall to another one – will be left behind for actual collaboration in which Development, Operations and possibly other departments work together, not just in silos.

Getting Past the Resistance to Change

Encountering some reluctance to change will be natural, and a smooth transition depends on making everyone aware of why DevOps will be a big step forward. This includes communicating what the advantages will be for individuals, departments and the company as a whole. With an agile plan in place, greater efficiencies will become the norm, and a team will have the ability to spot and fix problems more quickly, so 3 a.m. phone calls to the system admin won’t happen anymore. Once managers through C-level folks are aware of these potential advantages, resistance will gradually turn into acceptance and enthusiasm.

Combating the Myths and Rumors

Many think DevOps is only for the well-known unicorn companies, such as Etsy and Facebook, or that it’s an exact science that will produce the same results for everyone. Some even believe the rumor that DevOps only works with certain operating systems. The truth is that continuous development can take place anywhere as long as the company culture is right. As detailed in the Rackspace-sponsored e-book, The DevOps Mindset: Real-World Insights from Tech Leaders, many DevOps practitioners went through their own uphill climbs to become more agile, and they have the firsthand experience to resolve many popular misconceptions.

“You don’t have to be in Silicon Valley to embrace the DevOps culture,” says James Kenigsberg of 2U. “In fact, many new firms these days have a bad habit of building shiny technology because it is shiny, which goes against the DevOps principles of communication and transparency.”

Jeff Hackert of Chef dispels the myth that DevOps is a one-size-fits-all solution. “It is not a set of proscriptions or a recipe that will give you a defined best practices outcome,” he explains. “Your best practice is the collective wisdom of the very smart people you’ve hired to solve your problems. And if what you have is a siloed or broken environment, you want to increase the communication, not impose new rules.”

Getting Automated Without a Struggle

When an organization is considering DevOps, they may find that the opportunities for efficiency not only involve employee collaboration, but technology as well. Most IT departments would jump at any chance to save time when it comes to coding procedures, and automation is a means of making this happen. By offloading these processes to an automated system, people can tackle other tasks, and businesses that release several deployments a day can greatly simplify their rollouts. For those who might not be familiar with the ins and outs of automation, receiving some guidance from a trusted IT provider can be a good first step. Finding an advisor to guide your initial automation efforts can prevent the expense of hiring a DevOps engineer with automation knowledge.

Finding a Path Around DevOps Obstacles

Every company’s DevOps strategy will be unique, and every company’s plan for overcoming challenges will be just as unique. Whatever the circumstances, communication will be crucial to any adoption, as it ensures that everyone understands why such a change is necessary. Getting acquainted with the success stories out there will dispel any myths circulating among team members, and stumbling blocks can be avoided by the understanding that familiar silos will be replaced by more efficient collaborations.

Despite its challenges, DevOps can be the catalyst for a variety of improvements, from productivity to employee morale. Those who are the most successful at being agile will not only have the right culture in place, but be able to negotiate the challenges that DevOps has in store for those ready to embrace change.

November 19, 2014
How the Heck do you Jumpstart DevOps culture?

When the thought leaders in DevOps get onto their soap boxes, it’s pretty much a guarantee that at some point or other the discourse is going to turn towards culture. But often the speeches and conversations never dive deeper than platitudes about putting emphasis on people over tools. As true as it may be, though, most newbies want to know what that really means. How exactly does an organization shift to a DevOps culture?

Answering that question should start with a little bit of translation. When speakers hit the conference circuit, generally their references to “culture” are interchangeable with “collaboration.” An organization with a healthy DevOps culture is an inherently collaborative one.

Take a hypothetical situation where an DevOps organization is setting a requirement for a certain amount of performance throughput on an application, says Matt Selheimer, chief technical evangelist for ITinvolve. Developers in a traditional organization with that requirement would probably have to scratch their heads and wonder how to write the code, and figure out what middleware components they’d need to deliver that level of throughput. But an organization running with a collaborative DevOps culture would ideally give operations folks visibility into that project much earlier in the process.

“They might be able to come and say, ‘Hey, we can solve that through upgrading the IO on the storage area network, and if we do that, that will shave two weeks off your development schedule, and we’ll be able to deliver that faster to the business,’” Selheimer says. “So it’s those moments of opportunity — those synergies that you get when you get people engaged and when you get people who have like interests following one another and engaging one another that creates opportunity.”

That’s the ideal organizations are striving for, but it still doesn’t answer that question of how to achieve that culture of cooperation. There’s no pat answer for that, but perhaps one of the catalysts most likely to shift people’s behavior and beliefs is the fostering of a collaborative work environment. And that starts with the establishment of some sort of collaborative infrastructure.

See, sometimes even culture needs a boost from better use of tools. They’re just not necessarily the ones most people talk about when they’re discussing DevOps. Instead of thinking so much about automation tools, DevOps organizations need to prioritize tools and processes that enable knowledge transfer.

“So there’s an inherent bias in the community against tools, and I’ve seen that bias be eroded on the automation side, but I haven’t quite seen that bias be eroded on the collaboration side,” Selheimer says. “It’s time to really take a look at what tools you’re using to collaborate, because I guarantee you’re using tools. You know, you’re using email, you’re using instant messaging, you’re using SharePoint, you’re using Vizio, you’re using spreadsheets, you’re using phone calls — those are tools. ”

The question is, how effective is your existing toolset at greasing the skids for easy collaboration? The point is that automation may be important, but collaboration is the horse to automation’s cart. The problem is, that too many organizations are putting automation tool sets and processes first, leaving collaboration as an afterthought. That’s dangerous, says Alan Sharp-Paul, CEO of ScriptRock, a developer of configuration management tools.

“The automation of a system is often considered an ‘end’, not a ‘means’. Bad process and bad design become codified because they get buried,” he explained in a blog post. “Taking the necessary steps to ensure collaboration on requirements is often pushed aside in the interests of simply getting it done.”

Obviously, Selheimer has a biased stake in this game—his company creates collaborative software for IT teams. But his point is valid—organizations generally need to establish better mechanisms for their IT workers to share their work, their processes, their tools and system statuses so that they can better establish a continuous delivery model. More DevOps organizations are slowly coming to that realization. It’s why we’re starting to see more announcements from companies seeking to bulk out the collaborative tool box.

For example, yesterday the folks at VictorOps released its VictorOps Transmorgrifier to help organizations bake in more existing documentation, graphs and data about relevant systems into IT alerting. And earlier this month AppDynamics announced the release of its new DevOps Virtual War Room to help consolidate application and system information and metrics. Similarly, not long before that Tasktop introduced a new product called Tasktop Data that’s meant to help act as a data repository for real-time analytics across teams, projects domains and tools. Clearly, the pool of tools is growing deeper as organizations voice the need for a better infrastructure to support cooperation across IT tribes.

But that doesn’t necessarily mean that mechanism has to be expensive or complicated. It just needs to be more cohesive than a bunch of disparate documents spread out across multiple locations.

“There’s got to be a centralized, secure, well-lit place where people can document how everything fits together,” explains Chris Clarke, senior vice president of product management for CollabNet, saying that even something like a Wiki or a meta site can help organizations start to link together documentation about processes, architecture diagrams, pointers to tools and so-on. The simpler that an organization can make it for team members to share information like that, the more likely they’ll start working closely together.

This consolidation and dissemination of knowledge is also crucial for the long-term resiliency of DevOps teams. As many more organizations seek DevOps-savvy personnel to help jump-start their continuous delivery programs, employee turnover can pose danger to those organizations whose knowledge base is concentrated around a few DevOps rock stars or isolated teams.

“Churn can be dangerous because you get enough little islands that are each trying to satisfy their own goals and you get the superheros on those island and then effectively what happens is those superheros walk out and those islands blow up,” Clarke says.

This is why effective infrastructure for sharing knowledge is especially important in 2015. According to a recent survey by Rackspace, only about 38 percent of organizations who currently use DevOps for all applications across the organization. However, approximately 20 percent more of these organizations are currently working on pilot projects. And among those who haven’t yet implemented DevOps, nearly 50 percent say they have plans to fully integrate development with operations in the near future. Effective collaboration will be critical in ensuring success as as large organizations push simple DevOps pilot projects to full-blown IT departmental transformations. On the enterprise-wide level, collaborative infrastructure lag becomes unsustainable.

“If I’m a small startup company and everybody’s all in the same offices, I have a lot more opportunity collaborate in a face-to-face manner,” he says. “But if I’m General Motors or Ford or Bank of America, and I’ve got thousands of developers and thousands of operations personnel scattered across every country and continent, I can’t do that effectively.”

November 17, 2014
Collaborate more effectively with AppDynamics DevOps Virtual War Room

DevOps enables companies to move faster—but moving faster is not a worthy goal in and of itself. Organizations have to be able to develop and deploy solutions that are effective as well as expedient. It’s important to be able to monitor and respond to events in real time to ensure everything is running optimally, and to be able to create and implement fixes as quickly as possible when it’s not.

At its AppSphere 2014 conference held in Las Vegas earlier this month, AppDynamics announced the latest release of the AppDynamics Application Intelligence platform—which is designed to help make application intelligence available so businesses can run faster, leaner, and more efficient. One unique element of the latest release is a new DevOps Virtual War Room.

An AppDynamics press release describes the DevOps Virtual War Room. “It enables IT operations, development teams and business users to see and resolve problems together in a shared virtual space. The new release also enables scheduled, auto-delivered reports, and adds a new iOS app that sends push notifications to team members when system alerts are triggered.”

Unless you can get all parties in one room, reviewing the same information, and discussing the issues and potential solutions, troubleshooting can easily devolve into chaos. Mixing various forms of communication—phone calls, emails, instant messaging, face-to-face, etc.—results in a modern day business form of the children’s game “telephone”. Statements get twisted, and bits and pieces of information get lost in translation as various parties attempt to work together.

The DevOps Virtual War Room strives to change that dynamic by giving teams—even geographically separated teams—a single virtual space where all parties can come together to collaborate. Everyone sees the same data, and the individuals can discuss options and make changes to application settings and configurations together in real-time.

Another benefit of the DevOps Virtual War Room is that it maintains a log of the communications and the changes that have been made. Valuable information about the root cause of the issue, the troubleshooting process, and the changes made to implement an effective solution are captured so that there is an audit trail to refer to if additional problems arise, as well as a historical archive that can server as a reference for future issues. The data collected from the DevOps Virtual War Room can be captured and indexed in knowledge management systems in order to streamline access in the future.

“We’ve all seen the tremendously positive impact the DevOps phenomenon has had. With our new Virtual War Room and the other tools introduced today, we are catalyzing DevOps collaboration for troubleshooting, as well as application development, in ways that no other APM platform can do,” explained AppDynamics CEO Jyoti Bansal. “Performance isn’t just about technology, as critical as that is—it’s also very much about people working together closely to get things done faster and more effectively.”

To be fair, there are a variety of other ways to accomplish a similar goal. Any number of Web conferencing or virtual whiteboard tools—including things like Skype for Business or Google Hangouts—could serve as a virtual meeting places where developers, operations, business stakeholders, and others could “meet” virtually to get things done. One aspect that makes the AppDynamics DevOps Virtual War Room different—if not better—is that it also brings the relevant data and information to the table, and presents it in a way that enables the individuals to analyze the problem and devise a solution.

November 13, 2014
Nationwide banks on DevOps to drive towards Continuous Delivery

The Financial Services industry is fast moving, highly regulated and customer-centric. The general perception in our industry is that any change in processes implies risk and delay, neither of which we can afford. But at the same time, we are faced by the mammoth task of needing to continually introduce new products and services to stay competitive and cater to the growing needs of our customers. This is unthinkable without a fast software delivery lifecycle, as our business is highly dependent on technology. The best way to accelerate software delivery in a complex, fast moving environment like ours is to adopt a DevOps approach.

As a speaker at a number of events, I have discussed the ongoing transformational journey at Nationwide and shared our approach with other enterprises who want to understand how they can follow suit. I recently had the opportunity to co-present with Hayden Lindsey from IBM at the DevOps Enterprise Summit (Oct 21 – 23) hosted by Gene Kim. You can watch a replay of our joint session Patterns for Enterprise Success: The DevOps Journey at Nationwide below.

From humble beginnings as a small auto insurer serving the farming community, Nationwide has grown into one of the world’s largest insurance and financial services companies. Like many businesses today, we saw that we needed a better way to respond to changing market requirements and regulations, to increase speed to market, and to support new channels for customer interaction. Mobile, cloud and big data technologies were further raising the pressure on us to facilitate rapid digital innovation. What we required was not just a solution, but a change in culture (the way we look at things), processes (the way we do things) and tools (the technology we use). DevOps gives us the capability to achieve all of this—and more.

The immediate challenge was to achieve these changes by leveraging our existing heterogeneous tool suite. We partnered with IBM DevOps, which has played a significant role in our success. We were already working towards agile transformation. With IBM’s support, we were able to use the DevOps approach to application development and delivery across our distributed and mainframe environments. While a DevOps approach as supported by IBM enables organizations to start anywhere based on their specific needs, we at Nationwide decided to address all four adoption paths across the software delivery lifecycle: Steer, Develop/Test, Deploy and Operate.

DevOps enabled our IT department to perform continuous integration of its code and continuous deployment into its development environment several times a day. Teams can also perform acceptance testing of customer requirements within the same iteration as development. IT managers can show their internal customers, in near-real time, what developers are producing. This capability became a game-changer as we became more agile as a business and more responsive to customers. We achieved a 50% improvement in code quality, a 70% reduction in system downtime and were able to move 58% of our teams into the top quartile in key productivity measures. I believe that, the more agile we can be from an IT perspective, the more agile the business can be in terms of going from the development of an idea to the deployment of a customer-facing product.?

The next step in our DevOps journey is to build on the gains generated by the adoption of Agile and Lean practices across the entire SDLC. This is where technologies such as TaskTop and UrbanCode come to the fore by providing the integrated continuous flow and visibility needed for a continuous delivery infrastructure. By creating this delivery highway, we are giving Business Areas the ability to deliver as fast as they need to based on their own determination of cost, risk and value. Time-consuming manual processes based on out-of-date information are replaced with automated deployment policies that use real-time information synched from systems of record. The combination of dashboard and dials gives the business the visibility and control to drive at a much faster pace than is currently possible. Lack of trust in development outcomes has historically been an impediment to speed. Our new automated approach is giving us the confidence to move faster to serve the business without compromising the quality of our deliveries.

IBM was a great partner for us here as they share our support of the open standards approach, which gives us the ability to work with a heterogeneous set of tools, across the full lifecycle, allowing us to determine holistically the best tools, the best architecture and the best approach to meet our goals of high quality, high productivity and faster delivery for our customers.

By balancing speed, cost, quality and risk, DevOps is helping us increase our capacity to innovate. That in turn helps us get to market faster and be more competitive. I mentioned during DevOps Enterprise Summit that the phrase “time is the new currency” resonated with our goals of optimizing the path from “concept to customer value”. We need to ensure we don’t let it slip through our fingers. We have long been dependent on manual processes, but in this new digital age they are only going to slow us down. This is the time to be agile, competitive and pragmatic. It’s time for all of us to break the age-old shackles and embrace the opportunity for change that DevOps represents.

About the Author

Carmen DeArdo is the Director of Application Development Tools and Technologies at Nationwide Insurance. Carmen is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across Mobile, Distributed and Mainframe and other technologies. This includes recommendations and implementation of technologies used across the development life cycle (e.g. IBM Rational tool suite, open source technologies).

November 12, 2014
Why “Enterprise DevOps” Doesn’t Make Sense
Recently, some folks have been suggesting that we need a new form of DevOps, something called “Enterprise DevOps” or “Big DevOps.”

DevOps is great, the argument goes, but large enterprises don’t operate the same way as the agile web startups where DevOps was birthed, and thus they can’t implement the same DevOps methodologies as those startups. Large enterprises have bigger teams, more inherent operational complexity, and greater governance controls (both self-imposed and externally-imposed); therefore, they need a different type of DevOps that caters to their sensibilities and not those of agile web startups.

While it’s true that large enterprises do operate in a different environment than agile web startups, none of that makes a difference from the point of view of adopting DevOps.

In short, if you’re advocating for Enterprise DevOps as something different than Plain Ol’ DevOps, then you’re missing the fundamental essence of DevOps.

Here’s why.

Early Thinking: DevOps = Tools + Culture

To best understand DevOps, we need to understand where it came from.

DevOps started as a bottom-up collection of best practices with few formal underpinnings. System administrators and developers were looking at the successes of agile development practices and trying to apply some of the same ideas to the whole development-to-operations process. Practitioners in the trenches stepped forward and said, “This is what we’re doing and it seems to work for us.”

One of the earliest and most visible examples of that was John Allspaw’s and Paul Hammond’s presentation at Velocity in 2009, titled “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr.”

[youtube http://www.youtube.com/watch?v=LdOe18KhtT4]The video is well worth your time, as is the presentation on Slideshare:

[slideshare id=1628368&doc=allspawhammondvelocity2009-090623161942-phpapp01]

Note that this presentation predates the actual creation of the name “DevOps.” Allspaw and Hammond talk about how Dev and Ops teams can work productively together, but they never say “DevOps.” Patrick DeBois would coin the term later that year when he created the first DevOpsDays.

In the presentation, Allspaw and Hammond discussed the success they were able to achieve at Flickr and provided a set of recommendations that they think are important, separated into two buckets:
1. Tools
2. Culture
In the Tools category, they provided six recommendations:
1. Automated infrastructure
2. Shared version control
3. One step build and deploy
4. Feature flags
5. Shared metrics
6. IRC and IM robots
Along the way, they mentioned that Flickr used:
- Chef, Puppet, CFEngine, and other automation tools
- SVN for version control
- Hudson for builds
- Ganglia for monitoring
Finally, in the Culture category, they listed four additional recommendations:
1. Respect
2. Trust
3. Healthy attitude about failure
4. Avoiding blame
Unfortunately, most people missed Hammond’s remark ahead of the Tools section of the presentation (timecode 7:40):

These are some of the tools that work for us. They are not necessarily going to work for everyone, and throughout we’re going to give examples of the specific tools that we use, but the important thing that we’re trying to get across is the concepts that are important.

In the years following, many self-proclaimed “DevOps engineers” (a bit of an nonsensical title if you understand that DevOps is about bringing Dev and Ops engineers together, not creating new types of engineers focused on nothing but DevOps) insisted that if you were automating your releases using Jenkins and Chef and treating the other members of your team with respect, you were “doing DevOps.”

While Allspaw’s and Hammond’s presentation was excellent (seriously, spend the time watching it) and the advice was practical and valuable, many people walked away with only a vague notion of what would later be called DevOps.

For instance, in the “One step build and deploy” recommendation, Allspaw and Hammond describe their automated system that moves code into production. Developers can click a single button to make a build live. The button is jokingly labeled “I’m feeling lucky” (in homage to Google, I’m sure).

If you’re an agile web startup, that’s par for the course.

But if you’re a large financial services company, you’re likely to think, “No! Our governance folks will never let developers push code straight to production with a single click, never mind the ‘I’m feeling lucky’ labeling. No way. Not gonna happen.”

When people suggest that we need “Enterprise DevOps,” they’re typically reacting negatively to the choices that others have made. They recognize that those solutions won’t work for them for any number of reasons, and they say, “I can’t use that. I need something different.”

But here’s the thing — DevOps isn’t about the specific implementation choices. Those are all optional. You can make a completely different set of choices than Allspaw and Hammond and still be “doing DevOps.”

Unfortunately, subsequent conferences were filled with DevOps presentations that just reiterated the “DevOps = Tools + Culture” mantra and provided similar sets of recommendations that were often drawn from agile startups rather than from large enterprises.

So, does that mean Tools and Culture aren’t important for DevOps?

No, not at all. Both Tools and Culture are critical for DevOps in the long run.

The problem that DevOps seeks to solve, however, is not a Tools problem or a Culture problem but rather something more fundamental.

In order to understand DevOps, you need to focus on the ultimate goal.

The Firm Foundation: The Toyota Production System and Lean Manufacturing

In 2013, Gene Kim, Kevin Behr, and George Spafford wrote The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.

The trio’s contribution to the DevOps movement was in recognizing that enterprises had confronted a similar set of issues previously, but in a different domain — manufacturing — and that similar solutions could be applied to software engineering and application operations. (The group took another cue from the manufacturing world, copying the novelization technique from Goldratt’s The Goal to create a fictional case study where the characters were allowed to wrestle with the fundamental DevOps truths being discussed.)

The key takeaway from The Phoenix Project is that DevOps seeks to create a smooth and increasingly speedy flow of change from development into production, with as little waste along the way as possible.

The models here were the Toyota Production System (TPS), Lean Manufacturing, and Goldratt’s Theory of Constraints.

In the world of manufacturing, we have a physical product that moves through a factory, either on a formal assembly line, or from processing step to processing step, as part of a value stream. Each step along the path is supposed to “add value” to the product until it completes its journey and exits the factory to be shipped to customers.

If you’re a manufacturing engineer, you need to do everything possible to create a smooth flow through the value stream by identifying constraints in the system and eliminating waste of time or resources.

In the software world, application artifacts and changes move through the software development and operations value stream, with each step designed to “add value” and get the changes into production.

If you’re a development or ops engineer, you need to do everything possible to smooth the flow through to production. The techniques you use are similar to a manufacturing engineer — you identify constraints in the system and go about eliminating waste of time and resources.

The Kata is the Key

Following Toyota’s success, the TPS and Lean Manufacturing were all the rage in the manufacturing world.

Consequently, many western manufacturing engineers studied Toyota’s systems in great detail. They noticed a variety of tools and solutions that Toyota used to optimize manufacturing flow, and they recommended their use to their employers and clients. Toyota’s Kanban, for instance, was heavily promoted in the manufacturing world, and later made the jump to software engineering.

Interestingly, the manufacturing world has suffered much of the same confusion that’s now happening in the DevOps world. When they tried to employ the tools and techniques they had learned from studying Toyota, things didn’t work as expected.

Mike Rother, author of Toyota Kata: Managing People for Improvement, Adaptiveness, and Superior Results, described the situation in the manufacturing world this way:

Looking back, we naturally put Toyota’s visible tools in focus first. That is where we started— the “door” through which we entered the Toyota topic. It was a step in the learning process (which will also, of course, continue after this book). Since then I went back to the research lab–several factories–to experiment further, and present what I learned in this book. The visible elements, tools, techniques, and even the principles of Toyota’s production system have been benchmarked and described many times in great detail. But just copying these visible elements does not seem to work. Why? What is missing?

Later, Rother continues:

Toyota opens its factory doors to us again and again, but I imagine Toyota’s leaders may also be shaking their heads and thinking, “Sure, come have a look. But why are you so interested in the solutions we develop for our specific problems? Why do you never study how we go about developing those solutions?” Since the future lies beyond what we can see, the solutions we employ today may not continue to be effective. The competitive advantage of an organization lies not so much in the solutions themselves—whether lean techniques, today’s profitable product, or any other— but in the ability of the organization to understand conditions and create fitting, smart solutions.

Focusing on solutions does not make an organization adaptive. For example, several years ago a friend of mine visited a Toyota factory in Japan and observed that parts were presented to production-line operators in “flow racks.” Wherever possible the different part configurations for different vehicle types were all in the flow racks. This way an operator could simply pick the appropriate part to fit the particular vehicle passing down the assembly line in front of him or her, which allows mixed-model assembly without the necessity of changing parts in the racks. Many of us have been copying this idea for several years now.

When my friend recently returned to the same factory, he found that many of the flow racks along that Toyota assembly line were gone and had been replaced with a different approach. Many of the parts for a vehicle are now put into a “kit” that travels along with the vehicle as it moves down the assembly line. When the vehicle is in an operator’s workstation, the operator only sees those parts , and she always reaches to the same position to get the part.

My friend was a little upset and asked his Toyota hosts, “So tell me, what is the right approach? Which is better , flow racks or kitting?” The Toyota hosts did not understand his question, and their response was, “When you were in our factory a few years ago we produced four different models on this assembly line. Today we produce eight different models on the same line, and keeping all those different part variations in the flow racks was no longer workable. Besides, we try to keep moving closer to a one-by-one flow. Whenever you visit us, you are simply looking at a solution we developed for a particular situation at a particular point in time.”

As we conducted benchmarking studies in the 1980’s and 90’s and tried to explain the reasons for the manufacturing performance gap between Toyota and other automobile companies, we saw at Toyota the now familiar “lean” techniques such as kanban, cellular manufacturing, short changeovers, andon lights, and so on. Many concluded— and I initially did too— that these new production techniques and the fact that Western industry was still relying on old techniques were the primary reasons for Toyota’s superior performance.

Rother concludes that the secret to the Toyota Production System and Toyota’s success was not any one particular tool or technique, but rather the Toyota Improvement Kata, which focused on continuous improvement of everything at Toyota, seeking to establish flow and eliminate waste.

Toyota Kata is a great book and if you want to understand DevOps, I encourage you to get a copy and read it for yourself. Kevin Behr recommended it to me and I was pleased to see Jez Humble reference Rother’s story, above, in his recent QCon talk on DevOps.

What is DevOps?

So, after all that, what are we left with?

What is DevOps?

Cue the trumpet fanfare…

Fundamentally, DevOps is the activity of optimizing the development-to-operations value stream by creating an increasingly smooth, fast flow of application changes from development into operations, with little waste. Optimization of the value stream takes place continuously using various continuous improvement techniques like the Toyota Kata.

“Is that it?!” I hear you cry.

Yes, that’s it.

“No way!”

Way.

“But what about Tools and Culture?! What about continuous delivery and Jenkins?! What about automation? What about mutual respect?!”

Yes, those are all in there, too. But those are the equivalents of Rother’s “kanban, cellular manufacturing, short changeovers, andon lights, and so on.” They are solutions to particular problems that you might be faced with.

If, during the course of trying to optimize your value stream, you’re faced with the problems that those tools and techniques solve, then you might try them out as solutions. Or you might not, if your situation dictates differently.

The key is to understand that simply deploying a tool or even a broad solution without thinking through your unique situation and the current state of your value stream and its current constraints is foolish. After all, the specific solutions that are applicable to an agile web startup have a high likelihood of being inapplicable to a large enterprise. And vice-versa.

Why “Enterprise DevOps” Doesn’t Make Sense

So, that brings us right back to where we started.

Enterprise DevOps doesn’t make sense because it confuses the forest for the trees. Those advocating for Enterprise DevOps make the mistake of focusing on specific tools and solutions (Jenkins, Kanban, etc.), then find them wanting for various (extremely valid) reasons in an enterprise context, and then reject DevOps in favor of Enterprise DevOps, which they suggest is somehow different.

But DevOps is about flow and continuous improvement, not about specific solutions. Flow and continuous improvement are equally applicable to a large enterprise as they are to an agile web startup. And if you miss that, you’re lost, regardless of the tools and solutions you choose.

To implement DevOps, you need to learn how to map your value stream, identify the constraints in it, and then use a continuous improvement kata to develop specific solutions for your particular problems at this point in time in your particular organization.

If you’re a large enterprise, the problems and constraints that you face will undoubtedly be different than those of an agile web startup. So, you’ll necessarily need to select different solutions.

That doesn’t mean you “aren’t doing DevOps!”

Rather, it means that you’re doing DevOps correctly.

And remember that whatever solutions you come up with to your particular problems and bottlenecks, they are likely to be temporary. As Toyota said to Rother’s colleague, solutions are only applicable to a particular problem at a particular point in time.

Epilogue: A Bitter Irony

Some of the earliest work on what would become DevOps grew out of some ideas that Andrew Shafer was kicking around in late 2008.

At the Agile 2008 Conference in Toronto, Shafer proposed a birds-of-a-feather session about “Agile Infrastructure.” Only a single person showed up — Patrick DeBois. He found an empty room. It turned out that Shafer skipped his own session because he received such poor feedback about it and thought nobody was interested. Shafer and DeBois later discussed Agile Infrastructure and many ideas that would later become DevOps in the hallways at the Agile 2008 Conference.

Shafer also presented on Agile Infrastructure at Velocity 2009:

[slideshare id=1637386&doc=velo09-agile-inf-090625020850-phpapp01]

Note slides 92 through 97. The key points from this sequence are:
- Dogma Sucks
- “Do not seek to follow in the footsteps of the wise. Seek what they sought.” This is a quote from Japanese poet, Matsuo Bashō.
- The Tao that can be spoken is not the eternal Tao.
- You are a special snowflake, just like everybody else.
Even from the start, Shafer was on to the point that a binding set of rules is self-defeating. DevOps cannot be codified in a set of rules. Because it’s bigger than that.

Note the Matsuo Bashō quote:

Do not seek to follow in the footsteps of the wise. Seek what they sought.

The wise DevOps practitioners do not seek to implement any given tool or technique out of adherence to DevOps dogma. Rather, they seek to create a smooth, quick flow of changes through the value stream. Do not try to follow in their footsteps using the exact same set of tools or techniques, but rather seek the same thing that they seek and find your own way to the same destination.

So, the bitter irony here is that we’ve been warned all along, by Shafer, Allspaw, Hammond, and others, to focus on the foundational problems, not the specific solutions they have employed. Yes, we can learn something from their experiences, but every situation is unique.

(Unfortunately, I haven’t been able to find video of Shafer’s Velocity 2009 presentation. If you find it, let me know. But you can see a later version of Shafer’s ideas, mixed with those of Paul Nasrat, in a presentation from Agile 2009, made available on InfoQ.)

Other Resources

I’ve hopefully salted this post with enough links to key presentations and resources, but there’s one more that should be included. If you’re at all interested in the history of DevOps, and you should be if you really want to understand it better, then you need to view Damon Edwards’ “The (Short) History of DevOps”on YouTube:
[youtube http://www.youtube.com/watch?v=o7-IuYS0iSE]

Connect With Me

Think I’ve left something out or put too much in, got it totally wrong or completely nailed it? Let me know! Continue the conversation by commenting on this post and following me on Twitter.

About the Author

Dave Roberts brings 25 years of technology experience to his role of Executive Advisor at BMC Software. A thought leader in agile IT and digital transformation, Dave routinely speaks at conferences around the world. Dave is a member of the Twitter #clouderati and blogs at BMC.com and Leverhawk.com. He holds nine patents and has written three books on computer programming and data communications technology.
November 10, 2014
Survey: DevOps gaining mainstream traction in IT today

New survey results released last week shows just how thoroughly DevOps has captured the mindshare of enterprises. Commissioned by Rackspace, the global Vanson Bourne survey showed that more than three-quarters of IT decision-makers are now familiar with the term DevOps and well over half of them have already implemented DevOps practices to some degree or another. And by 2017, that figure could go up by another 31 percent according to survey respondents’ plans.

“This survey shows a real appetite for the kind of development and operational processes today’s digitally-focused businesses need,” said Chris Jackson, CTO of DevOps Services at Rackspace.

Among those who have chosen the DevOps route, nearly half have already fully integrated their development team with their operations team and 38 percent say that integration is in effect for all of their applications. For the most part, the ops team has been the primary champions for these projects. Approximately 43 percent of respondents with DevOps programs in place report that IT operations have driven them and 32 percent say either the CIO or the CTO have driven the program. Just 16 percent say developers have been behind their DevOps transformation.

As expected, among respondents who have deployed DevOps, 57 percent have already experienced faster delivery of new features and 46 percent have a more stable operating environment. Additionally, 43 percent reported increased innovation and 32 percent have even reduced their IT costs.

However, it is the business upsides of respondents’ DevOps adoption that are really eye-opening. The speed and innovation seems to have directly translated to improvements in the business as a whole. Over half of companies that have implemented DevOps report an increase in customer conversion and satisfaction and 43 percent say they’ve seen an improvement in customer engagement. As a result, DevOps is stimulating topline revenue growth, with 38 percent of respondents reporting an increase in sales as a result of their revised IT approach.

“DevOps is empowering IT functions to make a real impact on the business,” Jackson said. “It is also finally an opportunity for IT to be recognized for having a central part to play in delivering great customer experiences and ultimately increasing sales.”

However, deploying DevOps has not always been easy for these organizations. Approximately 37 percent of respondents who’d embraced DevOps reported internal resistance from ops teams and 32 percent said developer resistance posed problems. Fortunately, less than a quarter of wider business stakeholders stood in the way of DevOps adoption. Nevertheless, among all IT decision-makers, 41 percent still said they had other more urgent IT priorities that kept them from the DevOps journey.

“DevOps is all about encouraging innovation and streamlining processes to help ensure development and operations are working hand in hand to deliver business objectives,” Jackson said. “While, the findings suggest that the DevOps journey is already becoming well established in many organizations, there is still a job to do in articulating the benefits to all groups involved for them to accept and drive through this change.”

November 5, 2014
Automating Away the Regulatory Compliance Myth

At the recent DevOps Enterprise Summit, we had a chance to sit down with Justin Arbuckle and discuss his concept of “compliance as code” and why it’s so important that security and compliance tests be built into DevOps practices at large enterprises. Recently, Arbuckle was chief architect at GE Capital where he introduced and sponsored DevOps at that organization – so he certainly knows about what it takes to “do DevOps” in large, highly regulated environments.

Currently, Arbuckle is the vice president of EMEA at CHEF and also chief enterprise architect there. If you watch his talk from the DevOps Enterprise Summit, Hunting the DevOps Whale, you’ll see that he is passionate about the transformative potential of DevOps in large enterprises.

At CHEF, Arbuckle is in helping large enterprises understand how to “de-risk” their migration to cloud with automation. We caught up with him at the DevOps Enterprise Summit to ask why that is so important. Here’s what he had to say.

devopsy.kinsta.cloud: When it comes to DevOps and regulatory and security concerns, many people say, “Well, just automate those tests.” What does that mean, to you? How do you “automate in” security and compliance?

Arbuckle: I think, in terms of the specific issue of regulatory compliance, a lot of what we think of as the being compliant process today is a complete myth. There is just so much detail that needs to be checked that the organization is always in a position where they have to trade off against, “It’s good enough, we’re ready to go” versus, “We’re not going to go anywhere until we’ve literally crossed every T and dotted every I.”

The reality is that a lot of the regulations, Dodd-Frank is a good example, are getting to a level of detail where they’re beginning to talk about process. They’re beginning to talk about how you design product. They’re getting very deep indeed.

So the organization that has to govern that, can’t do it based on just this idea of oversight at particular points in the life cycle. Because if you’re going to do it properly you literally slow your velocity to zero, and so you’re making a trade-off between compliance and competitiveness.

The way to break that cycle is to, and this is a big paradigm shift for regulators, a big paradigm shift for compliance people inside organizations, you have to start being able to express those compliance requirements in a form that’s codable. Using something like Chef for instance what we’re able to do is we’re able to specify a particular policy in a way that is automatable. In a way that we know is always going to be implemented every single time.

devopsy.kinsta.cloud: Would that include input from security people, compliance teams, design teams, system architects?

Arbuckle: Absolutely. Honestly, in my experience I think the number of organizations that can count fully detailed, fully implementable — and that’s the key word — security policy by their infrastructure people is countable on one hand.

Generally speaking, there are many obvious guidelines that are standards-based. But as the security organization tries to keep up with security threats, they’re obviously learning as they go. In this way policy always lags. And the only way for the organization to catch it is through this process of documentation, policy, and checks. And through it all they know that the standard is nonsense because it’s out of date by definition. So they have to create a point-in-time review. And these point in time reviews bring velocity to a halt.

devopsy.kinsta.cloud: Yes. Especially when something is wrong. They hit a red button and everything stops coming off the assembly line.

Arbuckle: That’s right. The approach that we have to get to is the idea of policy as code. But you are not going to be able to, on the first go, be able to codify everything. Very often with automation, based on my experience, is that the difficulty at large enterprises has actually less to do with the tools and more to do with knowing what you want to automate. Knowing what you actually want to put in infrastructure as code.

It’s the same thing with security teams. Ask them to go through the most important things that need to be tested in automation, and getting them to go through that process. It’s often difficult because they are not development people typically. They’re not infrastructure people, And they’re not operations people.

They’ve kind of created their own “securocracy” if you like.

What security teams have to do is understand their work is subject to source control as well. Their work is subject to variation and evolution over time and we should write tests to ensure that the following policies are being applied consistently.

DevOps: And I assume these tests are designed over time, as one goes along? For example, if there’s a new malware or regulatory beast that you haven’t seen before, you can design a test and then implement that as part of the life cycle?

Arbuckle: That’s exactly right. And the great thing about tests is they are additive. You’ve got a test that tests for the range of ports that are open on the operating system and then you add another test, maybe testing for the types of applications that are installed on the box. They are two separate tests and they are additive. So as you continue, you build up your library of tests.

And for most organizations going through the adoption of test driven development, as an example, they have to do it that way. A large application that is beginning to develop in an agile way is going to have a lot of tests. If you take a typical large core banking application, for instance, it will probably have something in the region of about 5,000 use cases. That’s thousands and thousands of tests. Let’s just agree that in no short period of time are you going to write those tests.

devopsy.kinsta.cloud: That’s an enormous catalog of tests. How do you keep them up to date and fresh? I imagine as security changes, regulatory compliance changes, you may have to tweak these tests as you go along. How do enterprises keep up?

Arbuckle: Like any requirement you have stakeholders that are responsible for those requirements. In just the same way, if you had for instance a mobile application and you have a product owner for that mobile application and that product owner is prioritizing feature sets you should also have a product owner for compliance.

And the product owner should be detailing what the set of compliance requirements are for the product.

Essentially compliance is embodied in the tests that are run against the appropriate applications. And so to manage automation and compliance as code, you need to have a product owner for compliance. And compliance as code, or automation, is the way it has to be in order for compliance to actually be true and consistent, as opposed to a myth.

November 4, 2014