devopsy.kinsta.cloud

Author: Ericka Chickowski

National Renewable Energy Laboratory Uses DevOps For Better Compliance

One of the big stumbling blocks naysayers claim DevOps can’t overcome is the perceived incompatibility of DevOps with high levels of compliance regulations. But forward-looking IT practitioners at financial institutions, healthcare organizations and government agencies alike have already started to show how untrue that perception really is.

The National Renewable Energy Laboratory (NREL) recently offered a perfect case study in this phenomenon. Not only do the key IT staffers in the trenches at NREL believe that DevOps is compatible with DevOps, but they actually used DevOps and continuous delivery principles to help the agency deliver compliance with a key piece of regulation around cloud integration within the NREL infrastructure.

The success came from a year-long project designed to garner the lab a needed update to its ‘Authority to Operate’ (ATO) designation under the Federal Risk and Authorization Management Program (FedRAMP) for the new integration of Amazon Web Services across its IT infrastructure. The project successfully doubled as a pilot program for DevOps and continuous monitoring, says Ryan Kelley, systems engineer at NREL. He explained that the agency wanted to take lessons learned from staffers in a very limited separate department who had previously dabbled with AWS and DevOps practices and apply them for a fresh approach in a broader project within NREL.

“The approach we took was, we’re not just going to try to jam all of our legacy toolsets into this and try to get them to work with the cloud,” he says. “We wanted to take a step back and say, ‘What’s the best approach for policy, what’s the best approach for toolsets and things like that for this new architecture?’”

Part of the impetus behind this fresh approach was the lingering memory of how painful it was for NREL to gain its ATO five years ago. At that time the lab used outside consultants to shepherd it through the process and it turned out to be “extremely painful, drawn out and very, very expensive,” Kelley says, explaining that the ATO governs

This time around, NREL hoped to find a way to use the expertise and resources it had in-house to streamline the process and not only comply with regulations but see a true return on its investment through a much improved infrastructure. According to Kelley, the project forced NREL to approach infrastructure almost like it was starting a new business.

“We had a fresh slate and we kind of had to build almost everything policy-wise and everything toolset-wise,” he says, explaining that in order to comply with the security requirements around achieving an ATO, NREL was most concerned about what’s called customer-responsible controls, which are often very configuration-centric. “That’s where having a centralized configuration management system pushes you towards compliance because it’s just – it gives us that central place to audit all of our configurations. Out of the gate, to get cloud underneath our authority to operate, we needed the ability to centrally manage and have our configurations approved.”

By approaching it as almost a new business, the IT team was able to make the cultural shifts necessary to achieve that through the improved teamwork and continuous delivery patterns that are the hallmark of DevOps.

“Enterprise IT tends to think that they have to deliver these big, huge projects all at once and tied up in a nice bow, whereas we’re trying to take the approach of getting a minimally viable product and then rapidly iterating on it,” he says, explaining that it took about a year to get the ATO, but that NREL is still iterating to improve the cloud integration overall.

Kelley says DevOps is providing a much needed shift for the organization due to the highly distributed nature of its programming resources. While the lab has a small cadre of IT staffers to run the backbone of IT infrastructure, the lab itself actually employs many programmers that actually work directly under what are essentially the lines of business for NREL, basically smaller sub-groups and departments working on individualized research projects. This ATO project showed that it is possible for the main IT group to interact with these smaller groups of developers to deliver quickly and effectively on projects and NREL plans to build on this success.

“You know, we’re dealing with the same problems everybody else is: complaints that we’re too slow to deliver, not responsive enough, and not working with the business enough,” he says, explaining that past pundits’ warnings for IT to ‘align itself to the business’ never before offered concrete advice on how to do that. “This was kind of one way that we did was just trying to find a project, develop a partnership and then sort of create a grassroots type of DevOps mentality. It definitely wasn’t top-down. And we’re continuing to work with these groups in this particular manner.”

Meanwhile, Kelley explains that the project also showed how continuous delivery toolsets can be instrumental in helping to with compliance demands.

“We feel that these new modern toolsets that let you treat your infrastructure as code actually gives our cybersecurity team more insight into what we’re doing because once you do that and start treating it like software, then it can be audited a lot easier,” he says. “It’s a lot easier than building each server on its own and then waiting for security to scan it and then waiting for them to get us a report back and then we deploy it.”

June 19, 2014
Security in a DevOps world: Q&A with Gartner’s Ben Tomhave

As the infosec community catches up with the spirit of the DevOps movement, many CISOs and IT security leaders still need guidance on how to make the paradigm shift. Not only are these security folk scratching their heads to figure out the additional risks posed by continuous delivery, but they’re also learning how DevOps can actually help them improve their effectiveness and within their IT ecosystem.

Gartner recently took a deep dive into this topic with a report entitled “Security In a DevOps World”. devopsy.kinsta.cloud caught up with one of the lead authors of the report, Ben Tomhave, research director for Gartner, to discuss some of his findings and to learn more about how he thinks security staff need to adjust philosophies and practices to peacefully co-exist with DevOps.

devopsy.kinsta.cloud: How do you think security people need to change their philosophy in order to get the most value out of DevOps practices?

Tomhave: Getting away from that adversarial ‘party-of-no’ attitude is very key. We have to get away from “Come to us, we’ll give you things” and start thinking of partnering, collaborating and mentoring. On the development side, security needs to be pulled in earlier in the process, providing faster feedback after code commits. And then it needs to be working with operations to provide secure foundations, really inherently secure environments that are secure by default so that nothing is changing after the code is written.

Because that’s been a huge problem, too, right? Where developers go write code in who knows what environment and then it propagates to QA, which looks a little closer and sends it to production. Then it launches and suddenly there are all these surprises, things are breaking left and right and there was no communication along the way. Everybody was working on different platforms, and ops maybe wasn’t even managing or maintaining the development platforms. We’ve got to get away from that and security is no different—it should be a part of the process. All the monitoring tools, all the hardening tools, all that stuff needs to be applied from dev through QA to production.

devopsy.kinsta.cloud: What would you say are the biggest security concerns or risks that crop up in DevOps environments? What kinds of things could go wrong if security teams aren’t embedded in the DevOps patterns?

Tomhave: I think that it ‘s the same old stuff, pushing out code that may be functionally correct but woefully insecure and all of the other gotchas, whether it be bad crypto, bad handling of identity management and insecure management of data. DevOps shops are not immune to these problems, either. One need only look at the Spotify disclosure that they had password resets because of a breach. That’s a big DevOps shop and clearly some error was made.

The good news is that a DevOps shop that’s doing lots of code pushes should be able to resolve issues relatively quickly. But that means it is incredibly imperative that they work closely together so that when there’s a break it is quickly triaged, remediated rapidly and there’s good clear communication between the incident responders–whether they be security incident responders or operations incident responders—and developers, getting the feedback back to developers in a timely fashion.

devopsy.kinsta.cloud: It’s clear from your report that you think monitoring is more important than ever to support the rapidity of the DevOps cycle. Why is that?

Tomhave: You know, we expanded that DevOps mantra of fail fast, learn fast to add recover fast in there, too. If you want to fail fast, that’s cool. And if you want to learn fast, that’s also cool. But you also need to have rapid recovery as well, because the sooner you can detect a breach or an incident of some sort or a security event, the sooner you can start interdicting and remediating and taking positive actions to clean it up.

If you do DevOps with lots of operational transparency into process, but you don’t have adequate transparency into what the project is doing with the data involved and overall security monitoring capabilities, you could have a lot of problems. Suddenly now we’re going to be rapidly iterating through features and code pushes that are expanding capabilities and then we never know whether or not that environment is being attacked or whether or not it is being adequately secured or if it is breaking down. So that transparency meme needs to expand out to security monitoring as well so that you have security operational transparency as well as process transparency.

devopsy.kinsta.cloud: I was really interested in your advocacy for a security architect to help facilitate security meshing into DevOps. What does an organization need from this role and what are the challenges of finding the right person to do the job?

Tomhave: The challenge here is what you really need is someone who is really competent with development and preferably has some actual experience that allows them to establish credibility with the development team.

But they also must be really strong security people as well. So they understand static analysis, dynamic analysis, they understand what he tools are doing, what things are looking for. They must also understand design principles from a security perspective, things like here’s why you don’t roll your own crypto, why you integrate with existing federated identity systems, why you do transport encryption and what an appropriate level of protection is for a given system.

Finding those two sets of qualities in a single person can be really challenging. It is really difficult to find them and, more importantly, once you do find that person, they tend to be very expensive, especially for smaller shops. You think of all these mobile start-ups, for example. Their feet are being held to the fire; they’re being subjected to regulator actions. We’ve seen the FTC go after some of these mobile app companies for violating privacy policies and things like that. So how do you get those security bodies in there when you may not be able to afford to hire someone?

Alternative staffing models really speak to that. They may need to find consultants through specialty shops or something along those lines to get someone to come in and be engaged either through a staff augmentation type of role or just on a per-project basis.

I think the big key is being able to do that around the design review up front and catch some of these problems early. Because no matter how much you invest in tools and how many times you scan the code base–either statically, dynamically or whatever–that’s only going to check maybe 40 percent of your code. And a lot of these functional weaknesses, you aren’t going to be able to detect them with tools. You have to have somebody look at the design and say, ‘Oh, you’ve forgotten completely to do encryption,’ or, ‘Oh, I see you’re hashing, but you haven’t salted your hash, so how are your passwords secure?’ or ‘Why are you dealing with credit card data here, this shouldn’t be done this way, you should just be interconnecting into our payment system over there. Quit reinventing the wheel.’

Those types of things need to be caught by architects early on, rather than trying to catch it using tools in the code.

devopsy.kinsta.cloud: Speaking of tools, you dedicated time in your report discussing the importance of tool chains and automation. Automation has always been important to security, but what’s changed now with DevOps?

Tomhave: Well, I think that the big thing is almost all of the testing has to be automated up front. Static code analysis should just be automatically triggered on a nightly repository of check-ins, as much as possible. Same with dynamic analysis. Dynamic app sec testing should be done automated and feedback should just be fed back directly into the issue management systems. There shouldn’t be much human touch there at all, aside from monitoring for, addressing and resolving false positives in those tools. And possibly adding some customized checks looking for specific libraries or frameworks that should be in place or should be used within the tools.

The only time where I think we want to have any sort of manual interruptions are the first up-front quick risk analysis. Deciding is this a high risk project that’s dealing with sensitive data or has uniquely high uptime requirements, or is this a lower risk project where we’re not really concerned about losses here? I think that, as well as some of the design review considerations would need some manual hands and eyes involved.

And then everything should automate until you get to the end where you say, ‘OK, this is a high risk project, we need to put extra testing into it, so now we’re going to kick off pen testing, where we need to have some manual validation in place.’ That validation not only deals with false positives but really does a deeper dive into the application. So you fuzz the interface, it looks like that broke some stuff and you say ‘Yeah, we validated and proved it and here’s the exploit. This is how it broke.’ Then feed that back in a credible fashion to developers to show there’s the spot in the code with the errors or this is the logic error that led to whatever the condition may be.

June 2, 2014
How to turn executives into DevOps champions

IT practitioners must bridge the gap of understanding about DevOps with their managers and executives if they really want continuous delivery to take root in their organizations. Next week at Camp DevOps @ Gluecon, Andi Mann, vice president in the office of the CTO for CA Technologies, will propose ways to do that in his talk, Building Top-Down Support for DevOps.

“You can only push so far from below until you need to start getting support from above,” Mann says, explaining that the practitioner audience that tends to dominate events like Gluecon are running into a ceiling with DevOps adoption because the organization-wide changes necessary require top-level support from managers and executives to carry them out. He believes passionate advocates of DevOps need to do a better job garnering that top-level support.

From practitioners and even middle managers, that will require some mind shifts and improved ways of approaching people further up the food chain. He gave devopsy.kinsta.cloud a quick preview of some of the suggestions for practitioners he’ll lay out in the talk he’s developed to help them start improving their DevOps executive pitches.

Understand management’s goals

Mann: “To start with, they need to understand their manager’s goals and even their manager’s manager’s goals. Employees often don’t think about that, they have tunnel vision and only consider their own goals or sometimes their team’s goals. Practitioners often fall into the trap of looking at something and saying ‘That’s obviously good for the business’ without really knowing if that’s the case or not.”

Think about politics

“If you’re looking to go into this major cultural change which DevOps must affect, then it needs to be in line with the organization’s goals. When you think about accelerating delivery of new capabilities, the question might be whether the rest of the organization is ready for that. So if you’re creating a new feature in a front office application, but you’ve got to train 15,000 front office staff in that application, is that something you want to be doing. You might think it’s a great thing, but your manager may say, ‘Hang on, this is just causing work for other people.’ You’ve got to understand where the boundaries are politically within an organization and figure out how to break those down.”

Bring fact-based results projections to the table

“How will you make your boss a hero? That’s glib, but you need to think of it in those terms, particularly when it comes to ROI and the business case for DevOps. Start by proving other people’s successes first to your managers and then show how your success will be achieved second. There’s survey data out there, there’s gartner data, and there’s data from vendors on what the outcome of DevOps are. It’s not enough to say to a manager, ‘I’ve got this idea, it’s a philosophy and amovement and a cultural change and it’s going to make our company better.’ Not many managers are going to spend a lot of time listening to that proposal. You need to come in with the ROI and how DevOps addresses their business challenges.”

Understand and communicate costs

Mann: “Not enough practitioners actually put together business cases for their managers to take to their executives to support the change in culture—and a big part of that is budget. Think about what training you need, whether there’s going to be additional hires to fill out skill sets. There’s cost of software—the DevOps tool chain is rapidly maturing and not all of it is free. Even the open source stuff costs money. If you want to get the most out of it you’re going to be looking at support contracts, services and training. So there’s cost involved and that is an important part of making that business case.”

As Mann will explain in his talk, these considerations need to be the building blocks of a complete and polished proposal. Bringing executives that “finished work” will more likely sway them than a few stray conversations made up of hints and gut feelings about DevOps benefits.

“Go to your boss with a finished idea. Explain ‘here’s what we need to do, this is the spend, this is the result we’re going to get in terms of dollar returns and savings, speed to market and so on,’” he says. “And then explain, ‘Here’s a project plan for how we’re going to do it,’ preferably with a paced, layered approach proposed rather than a big bang.”

May 15, 2014
DevOps and faster feedback: fewer problems, better features (part 2)

As the experts established in part one of this series , the tightening of feedback loops through DevOps gives developers a better chance to learn how well their coding choices perform in the context of production environments. This makes for higher quality code in the long-run and constantly improved skillsets among development teams.

But fast feedback loops don’t just improve the line-by-line coding work the development team puts out. It can also push through a huge boost in the value of the features they’re producing for end users.

“Feedback loops are important not just for technical feedback or feedback about things not working,” says Aater Suleman, CEO of Flux7 Labs and a professor in computer systems design and architecture at the University of Texas. “There’s also the feedback around what customers are looking form in terms of particular features and whether or not those features are doing well in the market or not.”

The tighter the feedback loop is about feature sets—and the more quickly the team can release responses to that business-related feedback–the faster the pivot time to account for market conditions. The quicker assessment of how well features are doing gives development teams and the business units they answer to a more accurate barometric reading to decide where to direct programming time. As a result, it isn’t just the quality of work that goes up, but also the business value.

“With a DevOps flow that allows developers to start seeing feedback from end customers right away, it helps them be more agile on a business level so they can be more strategic about where to spend their actual time,” Suleman says, explaining that this exposes developers to a big-picture view of their work. “You may have the most productive development environment in the world, but if a feature developers have worked on for two months is not going to be liked by the end users, it doesn’t matter how productive the developers were during that time period.”

Not only that, but it eliminates the hamster-wheel of playing catch-up with competitors’ features.
“I know of many companies which still hold to quarterly release cycles. Did you competitor just release an awesome new feature? Have fun waiting three months to catch up,” says Curtis Poe, a consultant and Agile development coach for All Around The World. ” In today’s competitive world and fast-paced business environments, do you really want to playing catch up all the time? Switch to continuous delivery and never worry about a lengthy release process blocking you from offering what your competitors offer.”
Poe says that the best DevOps teams can get the most differentiation from the crowd by understanding that they’re not just monitoring technology and code, they’re monitoring people. Namely, end users.

“Do you sell shoes online and you have a typical conversion rate of 3.2 percent? What happens if you make a release and the conversion rate drops to 2.5 percent?” he says. “Assuming that nothing technological is going wrong , monitoring business value, when appropriate, can instantly tell your designers, ‘Hey, moving that button from the left to the right side impacted our sales.’”

Additionally, that kind of business monitoring makes it more easy to understand the dependencies between technical and business functional aspects of software updates. Then the team can more easily explain to business leaders what technical obstacles stand in the way of functional improvements, and how to overcome them.

“You always want to deliver the best experience for your customers, and you want to iterate on both the product and the non-functional aspects as often and quickly as possible,” says Arnold Goldberg, vice president of customer engineering for PayPal. “Integrating new features might be the wrong thing to do if, for example, there are challenges with performance, stability or availability.”

May 12, 2014
DevOps and faster feedback: fewer problems, better features (part 1)

As organizations begin to institute DevOps patterns and continuous delivery into their IT shops, one of the most dramatic long-term returns on their investment is that it bolsters the class and consistency of work produced by existing development teams.

Organizations that evolve to DevOps don’t necessarily need to hire rock star programmers to boost code and feature quality. That’s because, in the long-run, continuous delivery actually fosters an environment that makes it possible for the developers already in place to learn more efficiently and better improve the day-to-day quality of their work. The secret ingredient to this? Faster feedback loops.

“Shorter time to feedback is the key difference between DevOps and the traditional waterfall approach,” says Aater Suleman, CEO of Flux7 Labs and a professor in computer systems design and architecture at the University of Texas. “The main challenge with waterfall was the fact that there was a very long loop for feedback, which removed the agility from the picture. It could be a month or longer before developers would actually get feedback about what they’d done.”

The longer the cycle, the more code to sift through to find problems and the longer it takes to fix issues, says Curtis Poe, a consultant and Agile development coach for All Around The World. Shorter feedback cycles make it easier to find issues quickly and attend to them as soon as possible.

“Having continuous delivery means that typically very little code needs to be reviewed for failure and as a developer, if I get a bug report for code I wrote in the last hour, I’m going to be far more efficient in fixing in than for a bug report for code I wrote last month,” Poe says.

This is true not just for out-and-out failure, but also tweaking for improved performance.

“When frequent deployments are coupled with well-instrumented monitoring metrics on all aspects of your application, small nuances in performance can be detected and teased out for relatively small amounts of code change,” says Jeff Behl, chief network architect at Logic Monitor. “Developers want quick feedback to know changing this particular small part of code can affect this aspect of the application. The sooner you learn how to optimize the application code, the better.”

Even more important, fast feedback loops are not only critical for fixing past problems, but making sure they don’t happen again in the future. The shorter the time between when a developer writes code and receives information about how well it did out in the real world, the easier it is for them to learn from those lessons and apply them to future work.

With longer release cycles, be they quarterly or monthly, there’s often a lot of energy expended trying to find the person who created an offending line of code that broke something, says BLANK. The top triage person will go through the ringer to figure out who broke the code, until they might pinpoint something like, this person made this code commit six months ago that broke the application.

“Does anyone remember what they were doing six weeks ago? Of course not. But if you were writing code at ten o’ clock and I came by at one o’ clock and told you exactly what was broken and I sent you a zip file that had everything that you needed to do to reproduce that on your desktop do you think you’re going t o be a better programmer?,” Gary Gruver, vice president of quality engineering release and IT operations at Macy’s said last fall at FlowCon, explaining that every developer wants to do a good job and is typically trying to do the best job that he or she can. “If you can’t provide them that rapid feedback , real-time in an efficiency manner, how do you expect people to get better?”

In the scenario above, the process of developing the offending code is long gone in a developer’s memory, which means it takes more work for the coder to review their work, remember the project and then mentally link that process to the infrastructure dependencies that were affected by it. With shorter cycles, devs spend less time remembering what they did and more cognitive work on understanding dependencies and the context in which the code operated. Long-term, that’s going to make for better code from the get-go later on.

“Developers who can understand how their applications work under additional loads, abnormal conditions, and degraded circumstances, will end up designing better applications and systems, and in the end play a more strategic role at the company,” says Arnold Goldberg, vice president of customer engineering for PayPal.

According to Goldberg, context in development is critical and the faster, richer feedback provided by DevOps allows developers to better understand how application improvements work in production context. And that is huge for informing best practices.

“Much of it is specific to the fabric of the existing organic infrastructure that a company has, so it needs to be a pragmatic application of it in the context of the company’s current situation,” he says.

Faster feedback loops aren’t just about making technical coding improvements, either. These more rapid assessment cycles can also greatly improve features and business value offered by developers. We’ll explain more in part two of this series.

May 6, 2014
Defeating the irrational DevOps jobs displacement fear

People are generally resistant to change, but when that change comes accompanied with the whiff of potential for job loss down the road you can count on gentle inertia to snowball in to out-and-out opposition.

Unfortunately, as early misperceptions swirl around the DevOps philosophy, many in the ops world have gotten their backs up in fear that the automation that’s tied to DevOps will endanger their jobs.

“I think it’s almost impossible to overstate the amount of angst that DevOps creates in operations,” says Gene Kim, author of The Phoenix Project. “The big fear in operations is that in order to get to lead times not in months, not to weeks, but to days, you can’t be manually configuring servers, you can’t be manually reviewing things all of the time. And so there’s a deeply held fear that these configuration management tools are going to automate everyone’s job away.”

Now, to the degree that some within IT operations ranks rely solely on large vendors to set the course of their career track, maybe there’s some jeopardy there, Kim says, but competent and capable ops staff needn’t fear the automation reaper.

“It’s a reasonable fear, but it is ridiculous at the same time because we all have so much more work that we could get done if we had the time,” agrees David Mortman chief security architect and distinguished engineer for Dell Enstratius. “Doing things manually is negative value for most organizations.”

And that negative value is having a direct impact on how IT services the business. In the mind of Shridhar Mittal, general manager of application delivery for CA Technologies, one of the big DevOps value propositions is that it helps IT from CIO on down to finally start to meet the pent up demand that the business has been clamoring for from the IT department.

“I’ve yet to see a CIO tell me that they have more supply than demand. Why do you think the business is heading out to directly work with cloud vendors and doing their own thing? It’s not because the IT said no so that 10 people could sit on the sidelines,” he says. “It’s because these projects are added onto the list of 1,500 other things they got asked to do at once.”

Mittal says that CIOs who are gung-ho for DevOps should be proactive about potential resistance from the trenches. This starts with open communication.

“The first thing is start to educate staff that this isn’t about eliminating jobs it is about reducing that gap between demand and supply,” he says.

And as IT leaders do garner more buy in, both they and operations staff will find that these workers actually become more indispensible rather than expendable through automation. DevOps allows workers that might have seemed lackluster to shine as rock stars because they’re more frequently doing higher value work.

“Once you get rid of the garbage work or manual work, which is very time consuming and error-prone, you’re going to find that your engineering team is allowed to perform better,” says Nick Galbreath, vice president of engineering at IPONWEB.

April 22, 2014
DevOps Is About Serving The Customer: Q&A with Shridhar Mittal

As a longtime proponent of continuous delivery and fast deployment schedules, Shridhar Mittal has advocated for DevOps operational patterns since before the DevOps moniker was even coined. It was a philosophy that stemmed from what he and his colleagues at what was then Interactive TKO saw happening in response to the service-oriented architecture movement: many organizations were slowing down their releases as opposed speeding them up, which was the whole “promise” of SOA, he says.

“So we started a whole company around how do you change your development processes to get speed with quality in this new world of distributed architectures,” he explains.

Since iTKO’s acquisition by CA Technologies in 2011, Mittal has used his current role as general manager of application delivery at the company to focus his efforts on evangelizing DevOps cultural shifts within the enterprise. He believes that DevOps is on the brink of revolutionizing the way IT departments function and in addition to his anecdotal evidence from frequent talks with IT leaders, he points to a recent survey CA conducted together with IDG for evidence. According to technology decision-makers, while only 7 percent of organizations have already adopted DevOps practices with enterprise-wide scope, it is gaining traction. Another 39 percent report that they’ve either implemented DevOps principles with limited scope or are in the process of adopting DevOps. And another 28 percent are either planning on implementing or are evaluating DevOps.

devopsy.kinsta.cloud recently spoke with Mittal to discuss the mainstream uptake of DevOps, the biggest transition challenges and the attitudes about IT job performance in a new era of cooperation.

devopsy.kinsta.cloud: Why do you believe that DevOps is gaining so much traction today?

Mittal: I think the whole promise of DevOps is around better customer experience. Better quality for the customer, but in a much, much faster way than you could do it before. To me, that is ultimate goal.

Ever since the whole consumerization of IT, everyone uses the web and everyone uses mobile devices, and there is no buffer in between the application and the end consumer now. Previously to that you used to have, in the call center, you had people who are using applications and the consumer never actually interacted with the application.

So if it sucked, its internal people that had to deal with that–it wasn’t the consumer. Now, you cannot afford to have your customers have a bad experience because then you are going to lose business. One of our case studies is the .com part of a very large company–one of the Fortune 100. They had done a study that showed that for every one second delay in the response in the website dropped the conversion rate of sending someone to the next step of purchasing. It dropped significantly for every second.

Think about it. If you’re doing a big marketing event, you’re sending hundreds of thousands of people to your website, and just because your website doesn’t respond properly, you lose that many more potential customers.

That’s the kind of value that we’re talking about right now. Take a simple example, say there’s the launch of a phone and the telecomm company wants to release it at the same time the manufacturer is releasing it and the systems are not connected in time. They would end up losing hundreds of millions if not billions of dollars of business because they didn’t release it at the same time, and one of the competitors actually released on time.

These are the kinds of the hundreds of millions worth of value we’re talking about. In the past, people thought about it as, maybe if I can get three percent or four percent more efficiency I can save x numbers of millions. But what I think they’ll really be looking for now is the ability to grow their business much faster.

devopsy.kinsta.cloud: How can organizations start meeting the challenge of transitioning to DevOps?

Mittal: Start small and then build up to the big There are at least three different on-ramps we talk about to get to DevOps. One is, just start with the development side of the word and decouple the development teams from each other so that they can do their things faster and still be able to deliver things on a higher quality. That’s using things like server virtualization and API testing. The second on-ramp is just improving your environment management, the release management capability. People today are spending tens or hundreds of man-hours creating and maintaining these environments then tearing them down and then bringing them up again.

So figure out how automate that process where you can take so many man-days of work and bring it down to something that can be handled by one person and can be done as a self-service in a matter of minutes as opposed to taking days weeks or months to happen.

And then the third on-ramp is to put more automated testing processes in place, for instances doing performance testing. Trying to do it in-house today is becoming more and more difficult because there’s never enough infrastructure internally to be able to handle simulating these tens of thousands and hundreds of thousands of people coming to your web site. It’s just not possible. So figure out ways to leverage cloud-based automated testing capabilities to support that.

There are many different on-ramps and people can start from one or another and then start building to much bigger things. Start small and eventually expand to put it across the enterprise.

It is also important to get the trust built between developers and operations through better visibility. The classic scenario is, the developers just want to write code, release it, then they want to go back to writing code again. Operations want to slow things down because all they care about is a stable environment. If I could get devs to have visibility and ownership and be measured on how the products are actually behaving in production and, same thing with ops, if I could give them visibility to what’s in the pipeline, what’s coming out, why is this coming out, what is the business case and how this impacts your business, they’ll all start feeling an ownership for the whole process.

devopsy.kinsta.cloud: That brings up an important question: how critical is it to change job performance metrics in both development and operations in order to make DevOps work?

Mittal: I think it’s very important. I think it’s not something that has to be done from the get-go but I think somewhere after you have done the first couple of projects and some of the people have started buying into DevOps, that’s when you have to fundamentally change the culture of the company. In my mind, that’s contracted behavior—and it’s not because people are coin-operated.

I’m not saying this in a bad way. To me, people look at the way they’re comped as a way of understanding what their direction is. What are they being asked to do?

So if they are not getting measured on quality, to them, that means “Just make sure that you just get things out as fast as possible and then if you have to roll back, then you have to roll back.” And if you’re not getting measured on constant speed, then that’s like being told by the executives that, “All I want you to do is have a stable environment; I don’t care as much about getting new stuff out there.”

So there has to be changes in the comp structure, how people are rewarded, to really change the culture of a company.

devopsy.kinsta.cloud: What other things need to be done on the line-of-business executive leadership side of the house to make DevOps work?

Mittal: I think that the business side needs to be the driver to a large extent, because the business needs to say why something is important. If supply of IT capabilities is much less than demand, that means there has to be some kind of prioritization of work. And who’s making that determination? Who is doing the prioritization? To me, without a business context, you can’t be doing direct prioritization.

I think the business has to be very involved in helping IT understand a prioritized list of what needs to get done. They don’t say how it needs to get done, they just need to say, “This is more important than this, and this is the impact to our business of that.”

April 7, 2014
DevOps Is For Horses: Stop Making Excuses For Starting

While DevOps practices may be spreading throughout organizations both large and small, often times the most visible success stories are also the most exceptional. Organizations like Netflix, Etsy and Google are brought up in conversations about DevOps time and again because they offer the most stark examples of the benefits that the model can bring to businesses. But the scale of their operations can often be intimidating to the everyday IT person.

“People will tell me, ‘Well, these are unicorns—special companies. We’re not like that. What works there will not work here,’” says Jez Humble, a principal for the consultancy ThoughtWorks and co-author of Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. “I think that’s a fallacy.”

According to Humble, two phrases that he hates the most as excuses for not adopting DevOps or continuous delivery principles are ‘That can’t work here,’ and ‘But we’ve always done it that way!’ He’s one of a growing contingent of DevOps pundits who argue that it is plenty possible for even the most run-of-the-mill organizations to engage in DevOps.

“DevOps is also for horses,” says Gene Kim, author of The Phoenix Project. “In fact, every unicorn was once a horse. Amazon in 2001, Google in 2004, Twitter in 2009 and LinkedIn in 2010—they all had huge monolithic code bases and inconsistent deploys.”

Humble agrees that these so-called unicorns “have the same problems as everyone else.” But no matter how remarkable the scale of their solutions to those problems, other organizations should be able to learn at least one very valuable lesson from their case studies.

“The critical piece is they’re always working to improve what they do, and that’s something that works everywhere,” Humble says.

That’s Nick Galbreath’s philosophy. As the former director of engineering at Etsy, Galbreath saw first hand the transformation that the organization was able to make using DevOps patterns. He’ll assure any organization that it didn’t all happen in a day. Which is why when people ask ‘How do you get started with DevOps?’ his answer is simply to start.

“If you have a sympathetic management ear, ask ‘Why don’t we try to cut deployment time in half?’” says Galbreath, who is now the vice president of engineering at IPONWEB philosophy. “Because the other way is to say ‘Deployment time takes too long, let’s do half as many.’ Wrong direction.”

Setting up a goal like cutting deployment times in half will get an organization streamlining and weeding out junk processes. It’ll also have the team working to move those processes from inside a select few peoples’ heads and into things like a batch script or even just a simple document so that they’re easily repeatable across the team and that more can be done in parallel.

“Cutting that work that adds no value in half, you’re really going to empower your group and your team to come up with innovative solutions,” he says.

In a similar vein, DevOps for normal folks could also be chunked up incrementally on a project-by-project basis or automation tool-by-automation tool basis, says Josh Corman, CTO of Sonatype.

“You don’t have to do an enterprise-wide shift hard left to DevOps,” he says. “You can do it in little projects or little small automations that free up time so you can achieve more competence.”

In addition to incrementalism, Humble suggests providing a leadership environment that encourages taking chances and learning from mistakes.

“Create safe situations so that when people make mistakes we can actually learn from them rather than engage in finger pointing,” he says.

March 26, 2014
Busting The 4 Biggest DevOps Security Myths

Last month when the IT security world convened in San Francisco for the annual RSA confab, a number of security experts tackled DevOps in talks that covered the crossover between continuous improvement and risk management. Among them was a lively panel mythbusting some of the most prevalent misconceptions about DevOps held by the security community.

Moderated by Dwayne Melancon of Tripwire, the panel was made up of Josh Corman, CTO of Sonatype, Nick Galbreath, vice president of engineering at IPONWEB, Gene Kim, author of The Phoenix Project, and David Mortman chief security architect and distinguished engineer for Dell Enstratius, all of them security professionals and very active DevOps evangelists. The banter backed-and-forthed between the participants, with four myths bubbling to the surface as particular highlights from the discussion.

Myth #1: DevOps Pace Will Leave Security In The Dust

One of the most prevalent myths about DevOps within the security community is that the rapid deploys of the approach will throw all risk management precautions to the wind and leave security practices behind as a result. Quite the contrary, panelists argued. They believe that DevOps patterns make it more possible than ever to bake security practices into IT processes more than before, when it is frequently bolted on “at the end of the caboose,” as Galbreath puts it.

“You’re not throwing out your current security system,” he said. “What you’re doing is removing a lot of the junk work that eats up so much time and keeps people underwater. You’re really enabling different teams to all improve security together.”

As Corman explained, security people say all the time that complexity is the enemy of security.

“It also happens to be the enemy of stability,” he said. “That is the reason DevOps people come willingly with open arms to the security community.”

Not to mention, Kim added, “because you can’t be highly available and have stuff coming continuously through the pipeline if it is not secure.

Myth #2: DevOps Destroys Compliance And Control Frameworks

This myth came up during the audience Q&A portion of the panel, where one audience member from the financial industry wondered at how possible it would be to institute DevOps in a highly regulated industry. According to Mortman, as seemingly chaotic and fast-and-furious as the DevOps pace can seem to the uninitiated, it is actually far more process-oriented than people realize.

“People say ‘Oh, there’s no process to this.’ But it is actually very process oriented,” he said. “It’s just not very bureaucracy-oriented. There’s not a lot of paperwork. In its very guts, it is very service-oriented and very delivery-oriented.”

As he puts it, every compliance framework in the planet says you need to log, audit and approve changes. But none of them stipulate a workflow where each change has to be approved one at a time.

“The workflow can be ‘Please approve the following list of actions that I can do without notifying you first,’” he explained, “and as long as you have an audit trail showing that that has been done properly, I have yet to meet an auditor who will not buy into that.”

Myth #3: DevOps Gives Developers Too Much Power

Many dyed-in-the-wool security pros fear DevOps because they believe it is a cowboy environment that gives developers too much power.

“I’ll be honest, when i first started observing DevOps patterns, developers doing their own deploys seemed immoral to me; irresponsible, reckless, just shouldn’t be done,” Kim said. “No sane person would want to be associated with that, right?”

The thing was, after Kim dove in and also did a benchmark study with Tripwire several years ago across 4,000 organizations, he found that when comparing groups where ops did the deploys to those where developers shepherded code through deploys, change success rates and mean time to repair were not adversely affected, and in fact, “it is often considered indispensible that developers take code all the way through the deployment pipeline and you know they are getting great outcomes.”

And from a security controls perspective, the truth is that in a DevOps situation developers “are not logging individually into each server and SSH’ing the code up, untarring it and installing it,” Mortman said. Instead these actions are being done by automated systems.

“They’re actually going to give you much better timestamps, much more accuracy, and more consistent results and visibility than a manual process being done by someone in ops,” Mortman said. “So, actually, you’re getting better change management out of that system than you are in other situations.”

Myth #4: DevOps Cuts Important Testing

One discussion point also brought up during audience Q&A was the possibility that DevOps would somehow excise important testing functionality from the IT process. The panel quickly put that one to bed.

“You can do bad testing in any model,” Corman said. “If DevOps starts eliminating any kind of testing, its a bad DevOps program. I think that’s orthogonal to some of the benefits or opportunities of DevOps.”

Kim pointed to Google as an example of how testing fits into the DevOps methodology.

“So, Google does about 5,500 code commits per day, they have about 15,000 software engineers, and 75 million hours of testing is run each day,” he said. “You can’t get that massive amount of deployment frequencies without a tremendous amount of testing.”

March 20, 2014
Partnership Between Tufin, Puppet Labs Indicates Need for Security Baked Into DevOps

A recent partnership between the IT automation developers at Puppet Labs and the security policy orchestration gurus at Tufin Technologies could start setting the pace for how orchestration of security policies can more easily be baked into DevOps practices. Announced at last month’s RSA Conference, the partnership yielded an integration between Puppet Enterprise and Tufin Orchestration Suite. This will make it possible for joint customers to streamline the configuration and provisioning of security policy changes to iptables, a host-based firewall commonly found on physical and virtual Linux servers.

“We expect Security Policy Orchestration to become a core requirement for our customers,” says Nigel Kersten, CIO at Puppet Labs. “Integrating with Tufin enables our customers to quickly and effectively address connectivity requirements across the enterprise without compromising security.”

The integration is an indication of the market’s need for better tools to help DevOps shops speed up the pace of application and network changes while maintaining a strong security and compliance posture. Today’s firewall and security policy changes are often handled in a manual, error-prone process—even at some of the largest enterprises. A recent survey commissioned by Tufin found that the vast majority of IT professionals reported having to correct 20 percent to 60 percent of security policy changes in their organizations after the fact.

“This is incredibly useful to our customers and ensures that security is woven into the server’s connectivity requirements,” says Reuven Harrison, CTO of Tufin. “As the convergence of security and network operations continues, this module enables our customers to manage change with automation that delivers tangible business value.”

According to Harrison, firewalls play a dual role in network operations, both providing security through network segmentation and connecting applications to the network across multiple touch points.

“It’s the latter that makes them relevant to DevOps folks,” he says. “Firewalls see the both application and the network, which makes them an ideal source of information for modeling applications. Load-balancers are also good for this.”

This is why the industry is starting see a convergence between these two technologies, and also why Tufin has pushed for an evolution from firewall management into the sphere of security policy orchestration.

“Security policy orchestration aids with use cases for firewalls and other network infrastructure as they relate to applications,” Harrison says. “It ensures security policy changes are automated and account for security and compliance considerations not just for network access but also for application connectivity.”

Harrison believes that while security may not necessarily be the chief selling point for evolving to DevOps practices at some firms, the movement could present opportunities to more closely fold security into a more rapid application release cycle. He believes it will be an imperative in complex cloud environments.

“As cloud infrastructure matures and gains acceptance from larger and more traditional companies, security will have to be baked into the application release process,” he says. “That means securing not just the application code but also the infrastructure on which it resides.”

As things stand, Harrison believes that security is typically split into two distinct layers. There’s the application security layer, typically dominated by code security and penetration testing and then network security that is preoccupied by zone segmentation, encryption, authentication and the like.

“Eventually these two layers, along with some emerging technologies such as virtual networks, will converge,” he says, “and DevOps will handle it end-to-end.”

March 19, 2014