devopsy.kinsta.cloud

Category: Doin’ DevOps

Booz Allen Helps the GSA Put Your DevOps to Shame

Production Docker, automated workflows, open source projects and microservices are just some of the ways that Booz Allen is helping the General Services Administration (GSA) modernize its IT operations. And frankly, after talking to Nirmal Mehta, the lead architect on its recent project, I became a little embarrassed for you DevOps-born startups.

In my circle of friends, the frustration and sarcasm used to describe government processes is a common pastime. And it’s totally fair. Even without being an insider, you can still easily approximate the magnitude of wasted money due to bad business processes and dated IT operations.

But things are turning around. Beyond the early negative press, the issues with healthcare.gov made very clear that the gap between large enterprise and government IT and the modern development shop is massive. And even more recently, the use of private sector hackers to unlock phones, and the continued efforts to recruit from and partner with Silicon Valley show that the government needs the best and the brightest technical help.

One government institution, the GSA, took the modernization challenge head-on. It selected Booz Allen Hamilton to lead the charge and build a new development platform—one that will ensure dev projects deliver results faster, at higher quality and are more systematic.

The GSA is an arm of the government responsible for managing the contracts and processes for vendors that wish to do work with the government. If you have sold to the government in any fashion, you are familiar with it. The administration oversees more than $1 trillion in transactions and more than 300,000 vendors and contractors. To manage the processes across a wide range of categories, it requires many web and mobile applications, with new ones coming on board every year.

Before working with Booz, GSA’s IT ops were about what you would expect—big, slow and risky. New projects took months to get started, with solutions that could take more than a year to complete, and they were ultimately subject to dreadfully low sub-30 percent success rates. (Watch this video and pay attention from the 8:30 point on).

The GSA came to a point where it fully recognized the hard cost of this way of operating, and the indirect cost of maintenance, tech support and unhappy customers on both the vendor and government-consumer sides. It knew something had to change.

Booz’s solution was not 100 percent unique, but its implementation was above and beyond. The firm decided to build a unified platform as a service (PaaS) that served as a portal for all new development projects. In some organizations, this is called “shared services.” But with Booz’s platform, it went beyond just application components.

The users of the solution would be development teams who were awarded projects. Once awarded, they would access the platform and create a service that consisted of applications and microservices. They would be given immediate access to a private repo via GitHub Enterprise, which would also include Docker Compose files.

Then, the teams were off and running. As soon as they made their first commit, the system—already wired up and driven by Jenkins—would pull, run tests and deploy containers to dedicated nodes in a continuous integration (CI) environment, run on FedRAMP-compliant AWS regions. Then the instances were automatically given IPs, where the development team could begin testing.

In addition to a prebaked development and deployment system, the platform provided the development team an entire DevOps stack, giving them “the entire application and lifecycle environment to build their application,” explains Mehta. That included everything from identity management and infrastructure-as-code to business intelligence (BI) and even release automation, so that after the applications were deployed, the teams had what they needed to run updates, measure and learn.

The idea is that a development team with this platform only needs to think about the logic of the application, not process. And now, at the organizational level, there is oversight, continuity and reliability—something which the GSA had never seen before. After the changes, instead of dealing with long contracts and RFP processes, the CTO could approve bids and get development up and running in days, and create solutions in months, not years.

Mehta’s team was responsible for architecting the system, putting it all together and vetting the “marketplace” of tools made available to developers. By doing so, people were able to make sure the proper security considerations were—and are—made, and that the tool configurations, Chef scripts and Docker images stay consistent.

The whole thing was built based on a set of principles. Cloud-first, secure foundation and, to my surprise, open standards and open source. The U.S. government, which fought cloud and open source for so long, is now one of its biggest champions. “I think 2015 was a huge tipping point in gov open source adoption. I think we have hit an inflection point for sure, and I’ve realized that DevOps really implies open source,” Mehta says.

If this is too hard to believe, just check out Booz Allen’s Jellyfish project (which has become part of the ManageIQ project), and this open source policy document in the GSA’s own public 18F repo, which encourages entities to use open source technologies it has developed.

This does not mean the GSA won’t spend money—that is not the driver of open source tools. The driver is cross-pollination, giving back and building better code through standards. Where warranted, such as with wanting full Jenkins support via CloudBees, the GSA will invest in technology. Jenkins is the heart of the system, and basically the home of all of the platform’s workflows and automation. Having the backing of CloudBees is important. Plus, CloudBees is a GSA vendor. It also has become one of Docker’s early customers. I can’t imagine the love it gets from those kids.

Often when I do these case studies, the solutions are too new to validate long-term success. But the platform has been up for six months now, with several development teams and beta applications deployed, which means the system works! The path to prod is still manual and belongs to a separate team, but there are goals to automate that as well.

A government agency is finally more focused on results than process. (If you have ever been through the RFP process, you know what I mean.) “The government is realizing that it is not sufficient to give the modern user, who expects technology that works [to stay with] legacy applications,” Mehta says. The platform also allows the GSA to have smaller contracts, and in theory, smaller and better development teams who don’t have to worry about development operations overhead.

When I asked Mehta if his project is the norm or just a fluke, he said “more and more companies are asking us about DevOps, not just cloud migration. We know that if they just do a cloud migration, they are going to create more problems faster.” He believes that even if DevOps is not explicitly stated, federal and local governments do care about modern IT ops—if not because they want to, but because they have to.

So, can you say you have a fully automated full-stack container-driven platform that spans multiple applications and multiple development teams? Chances are no, but you could, and you should. Mehta and team have proven that even the government can have a Ferrari of a development environment and stay on the cutting edge.

To learn more about Mehta’s project, watch his session at DockerCon2015 here. And check out this presentation from Navin Vembar.

May 13, 2016
Faster, Smarter DevOps

Whether you call it DevOps or not, if you are concerned about releasing more code faster and at a higher quality, the resulting software delivery chain and process will look and smell like DevOps. But for existing development teams, no matter what the velocity objective is, getting from here to there is not something that can be done without a plan.

Moving your release cadence from months to weeks is not just about learning agile practices and getting some automation tools. It involves people, tooling and a transition plan. I will discuss some of the benefits and approaches to getting there.

Waterfall to agile, agile to continuous integration, continuous integration to continuous deployment. Whatever your processes are, the theme is the same: Find a way to get code to users faster without sacrificing quality. But speed and quality are sometimes at opposition to each other. Going faster means things can break faster, and when we only think about DevOps as releases, it’s easy to fall into this trap.

Established development shops cannot just jump from one flow to another. Unless you start out net new, the goal is to introduce new processes without delaying releases for three months or more to do transition in lump. This often is done using a pincer approach that addresses both bottom-up tactics and top-down oversight and culture at the same time.

However, because adopting DevOps tools is so easy, the trend is to focus on tactics only and adopt from the bottom up without consideration of the entire pipeline. The outcome is release automation tools that dictate your delivery chain for you, and not the other way around.

Structured Automation

DevOps requires automation. But what is often not considered is automation that sustains and fits into the entire delivery chain. Governance, artifacts organization and inventory, metrics and security must be considered. If an organization establishes a vetting process for all new automation and how it fits into the pipeline’s orchestration, then new automation will support what exists today and in the future.

For example, many organizations driving down the DevOps path have encountered challenges when trying to incorporate practices from security or governance teams. Historically, these teams have resided outside of the Dev and Ops echo chamber and their processes were asynchronously aligned to the work being done. The challenge for many organizations is to determine the best ways to bring the people, processes and technology supporting these initiatives into the fold without slowing things down. The best organizations are finding new ways to automate policies from security and governance teams by shifting away from artisanal, asynchronous approaches to synchronous processes earlier in the life cycle.

Let’s take a look at an example of application security. A number of technology vendors in the application security arena are touting “automation” as key value point for their solutions to better fit them into a DevOps tool chain. In some instances, automation means that machines are now fully responsible for monitoring, analyzing and fixing security vulnerabilities for those applications at wire-speed. In other instances, automation implies facilitation of human-centric workflows that might represent hours or days of asynchronous analysis not fit for continuous operations. In both cases, the technologies may accomplish similar ends, but their approaches could be dramatically different.

Also, one solution might be built to support asynchronous investigations by a security professional, while the other might provide synchronous support to a developer at the design and build stages of the software development life cycle. Establishing a vetting process can help determine if the automation levels required by a team or process can truly be delivered before investments are made. It is also worth noting that layers of obscurity frequently exist within words such as “automation,” “integration,” “configuration,” and “continuous.”

There are other areas that also tend to get overlooked in building a DevOps environment. The complete story can be found here.

May 12, 2016
The Beauty of Inheritance Architectures

Spoiler alert: This article is not about inheriting your great-grandparent’s estate. It is rather about how embedding the principles of inheritance into the architecture of your chosen DevOps toolset can yield exponential results. And yes, I realize that you, the customer, have little to do with what a vendor builds its tooling architecture upon. However, you, the customer, can vote with your feet, to search out products that offer features such as inheritance, rewarding vendors that have it and punishing those that do not. The simple question to address here is, Why should the customer care?

Like anything that can grow, sustainability on scale is a key concern. How much will it cost me when I am maintaining 6,000 applications on this particular DevOps toolset, as opposed to the six applications I started with? You can also infer maintaining a higher number of technologies as having the same need. For example, if I need to maintain three different types of databases, two different middleware, four different hardware platforms on “x” different versions of operating systems … you get the idea. The larger the application portfolio (in sheer numbers of apps), or the more technically complex the portfolio (in sheer numbers of technology components in an app), eventually will drive you to worry about how easy it is to maintain these applications within your chosen DevOps toolset.

The capabilities of your chosen set of DevOps tooling will govern the costs of both the initial adoption of your services as well as the ongoing maintenance of the application portfolios you serve. It is hard to sell the value of DevOps to the business when it begins to see a one-for-one increase in DevOps engineering staff for every new application or technology venture under consideration. However, one of the chief culprits in driving large DevOps support staff organizations boils down to the lack of an inheritance architecture in the tooling that was originally selected.

Setting a Common Language

So before we discuss how inheritance will benefit us, first let’s agree on a common vernacular of what an “application” is. When you say the word “app” to someone, you get a wide variety of responses on what that means—from something that sits on your phone to a cluster of things that sit on the mainframe. Let’s begin by examining what an “app” is by the effort it takes to onboard and then maintain it on our DevOps sets of services (regardless of platform).

Consider for a moment the hierarchies by which we could describe software constructs. It is relatively easy to discuss how software works in terms of hierarchy. First, at the lowest layer, we could describe having multiple technology components (or the lowest level of a particular piece of technology that must be addressed in a build). Next, assembling these components together forms the picture of what a single application looks like. Finally, assembling multiple applications that have a common symmetry or common purpose together form a system or series of apps.

An imperfect but quick example could be the Microsoft Org Chart component of its Office application(s). For purposes of example we will argue that Org Chart, Clip Art Manager and Equation Editor are all components of the application Microsoft Excel. To build a new version of the application Excel, we would have to deal with each of these individual components that, once assembled in our build, would create a new version of Excel. When you combine more of the apps—Excel, Word, Outlook and PowerPoint—you get the system, or series, called MS Office. (And, yes, I know this is a less than ideal example, but it is meant only to illustrate a point.)

Assessing the Work Effort

The reason for discussing software in this way allows us to begin to discuss the level of effort to automate it. By addressing the work effort to automate each component (by type), we can begin to estimate how long it will take to complete “the automation of software.” For example, if my homegrown application has a source code component, a configuration file component, a database component and a middleware component, then I have four components I must automate (or include in the build and deploy process) to assemble a new version of my application.

This gives us a “least common denominator” language for assessing how much effort it takes to automate a given app. I have seen most common applications have only two to three components. But I have also seen several apps that, due to their unique architecture and construction, have 30 to 50 components a piece. Obviously, it is easier to automate an application with only three components than one with 50 components. The other factor we need to apply on the ease of automation is whether I have encountered this particular component technology before in my prior automation efforts.

Let’s say, for example, that the application I am estimating contains Java component(s). If I have encountered Java automation prior to this, I should have constructed a template for Java type component automation that I can use for a situation just like this one. In this situation a familiar technology component is far easier to automate than a new one (or, rather, new to me). However, if this is the first time I have ever seen Java, then it may take me a bit longer to construct the template I want for this app, and for future apps I encounter, that also use this technology type.

The Beauty of Inheritance

Here is where inheritance begins to play its critical role. If my DevOps-build construction tooling includes inheritance, then I could construct a “master” template or framework for each type of technology as I encounter it. A master template for SQL databases or Oracle or DB2, as an example. Or, I could make a master template for WebSphere or SharePoint types of technologies—one master for each kind of programming language we have in my shop. You get the idea.

Having done this allows me to have a limited number of master templates, in which I can maintain all the critical steps and processes my company wants me to do for a given technology type. If different business organizations or development organizations need a variation, they can create a “child” of my master and vary it according to their own needs. As long as the linkage is maintained to my parent/master template, I can make the big, overarching changes driven by corporate mandate, and the business unit can continue to vary items to their unique needs.

Having inheritance in this instance allows me to roll out changes to an entire organization, such as, for example, a security patch or process, for a given technology by simply adjusting the master template of that component. Inheritance automatically flows my change down to all the “children” of my master and possibly to all their “children,” should the need arise. The alternative is having to go through the specific automation of each application that has the targeted technology component and updating it manually, hoping I don’t miss one, and hoping the steps I may need to adjust do not break the old automation instead of alter it the way I needed.

Imagine having a WebSphere subject matter expert able to adjust every client application that uses your corporate installation of WebSphere by simply adjusting one master template. Or having a network guru, or database expert, be able to do the same for all applications that use their particular technology component all at once. Inheritance preserves the real-world need of having limited alterations of the same technology type within the same company, flowing down changes from parent to child.

Having a one-size-fits-all is simply not practical. Using copy/paste requires repeating updates to each impacted template or framework manually. But having the ability to create offshoots from a main, and not lose the connection to the main, allows corporatewide alteration from a central point of control. The costs of maintenance or administration are formidable; inheritance can reduce them. This allows a much higher ratio of applications or technology components supported by a much lower number of engineering staff.

A Wider Impact for Inheritance

Inheritance is keenly useful in release orchestration software as well. Most vendors offer the cut-and-paste approach: you copy the last the release you did, alter it slightly and use it again. That might work for some organizations. But having an inheritance feature creates entirely new dynamics on how a release is constructed for a given line of business and how corporate governance can be implemented with minimal effort. In this case, a corporate master release template can be constructed, and child releases can inherit from the master all the main governance required without risk of omission.

I realize inheritance architectures might not be at the top of your list when evaluating DevOps tooling solutions. But I would submit it is a strategist’s job (if you need one, let me know, I am looking 🙂 ) to point out how long-term sustainability can be greatly benefited by this architectural feature—even if the vendor would love nothing more than to steer you away from it. If you already have committed to a tooling solution, you may want to bring this up at your next vendor features performance meeting. If not, you should at least consider what having it means to your internal cost structure, and what not having it will add to the true cost of ownership of any proposed solution.

To continue the conversation, feel free to contact me.

May 11, 2016
5 Prerequisites for a Successful DevOps Initiative

There are certain tools and attitudes that are essential to any successful DevOps transformation.

The manner in which a DevOps transformation is performed will have huge implications on the level of agility the company is able to achieve. However, before that point is even reached, some vital foundations must be laid. It is from that foundation that any innovation occurs. With that in mind, each of these elements are equally important, and should be regarded as such.

Consensus

Your first achievement on the road to successful DevOps should be an agreement on what DevOps means to your company and how it should be applied to your needs. Consensus on a strategy is almost as important as the strategy itself and must be in place before a detailed road map is constructed. Remember that DevOps is subjective and its implementation should vary from company to company. Consensus hopefully will be easy, but if not, it must be built by persuading doubters of the need for certain principles and practices selected.

This is the first step on a cultural journey; the collaboration between Dev and Ops staff and their consensus on a plan could make or break the DevOps idea. The concepts behind the initiative need to be agreed on from all corners of IT, as well as the business and any other key stakeholders such as suppliers. It is important the cons and pros of switching to DevOps are made clear for a balanced proposition. The argument for a shakeup on this scale needs to be compelling and justified for it to go ahead.

Consensus also involves handling expectations. To keep everyone happy there must be metrics and approximate timelines set in place so everyone knows what to expect and when, to avoid conflict further down the line. Only once consensus has been achieved and the basics are put in place can a more detailed road map be built.

Flexibility

A thorough road map is the skeleton that allows its muscular teams to flex independently toward a common goal. However, flexibility without allegiance to the agreed principles can cause chaos. The DNA that runs through all teams stays the same but applications vary, and so teams need to work autonomously if they are to cater for the specific needs and nature of the app they are working on.

While the road map gives a structure and handles expectations, DevOps culture needs to be tied to business outcomes and not formal internal milestones. Each DevOps team needs to be trusted to interpret the road map as it suits the specific project they are working on. Agile methodologies must be used, of course, but their interpretation is devolved to the teams themselves. This license and trust is key in allowing DevOps teams to react quickly and mold around the needs of the individual application.

Teams need freedom in terms of what tools they use and to an extent, how they use them. The methodology used to achieve the agreed core principles should have an element of flexibility. Tool-agnosticism is a must when choosing an automation solution for effective DevOps. Teams are happier and work faster when allowed to choose the tools that suits them best, making them more successful.

Automation

As C. Little said, “DevOps isn’t about automation, just as astronomy isn’t about telescopes.” An automated continuous delivery pipeline should underpin any DevOps initiative, adding consistency and reliability to flexible delivery methodologies. Tribal knowledge from Dev and Ops is gathered to create a pipeline that is the sum of the knowledge of its contributors. The pipeline is repeatable, and over time is made more efficient by rooting out inefficient manual processes. Modelling into a workflow helps all involved visualize where a team is in the app life cycle and what next steps need to be taken next and by who.

Trust in application release automation can be a big deciding factor in how companies achieve successful DevOps practice. Sometimes there is a culture within a company of wanting to have manual control over certain processes that inevitably slow things down. Automating any and every manual task possible is essential to speed up DevOps. If a manual intervention is needed, the relevant member of staff can be notified by the automation solution itself, which automates out the remainder of the task for which manual intervention is not required. Over time, manual interventions will disappear altogether.

The ability to give DevOps teams freedom in their tool selection keeps staff productive but adds yet more complexity and leads to a highly heterogeneous workplace. It’s important your automation solution has approved integrations to all tools DevOps teams might choose and also releases free integration to new tools quickly, so you can stay on top of the latest trends and continue to be successful.

Cooperation

Unless you’re a startup with heavy investment, you’ll likely be changing the structure and workload of your existing staff rather than hiring a new team suited perfectly to their individual roles. The success of DevOps depends on the ability to make positive change in the way team members collaborate.

Resistance to change may occur due to staff being accustomed to working in one particular way for a long time. However, this could also be down to personality type. As it is Ops and Infrastructure staff who face the biggest changes and are used to limiting rather than encouraging risk, often they can be the most resistant to change. Often devlopers do not have to change the way they work to the same extent, so fewer feathers are ruffled.

I have seen big corporations that do not have compliance from everyone, so their DevOps is limited in scope and unfortunately they will never reach full agility. Sometimes you must adjust your teams to ensure positivity to DevOps and receptivity change in general.

Rethink Architecture

I’m not saying DevOps cannot be implemented in legacy systems, but there are elements of legacy systems that make it hard to instill full automation and continuous delivery. What is important is that the organization is open to a certain degree of architectural change. Monolithic apps can certainly slow down a DevOps initiative—often it’s mainframe apps that are not built with a service-orientated architecture—so it’s important to know how to adapt them to fit your strategy.

A full architectural revolution is not needed to be successful. Obviously this is where cloud-native startups have an advantage as they can assemble their infrastructure from scratch, but there is no need to rip and replace existing systems for the sake of it and waste money. Certain elements of even the most monolithic systems can be modernized—for example, Siebel deployments can be fully automated so operations staff are free to concentrate on more demanding aspects of the cultural change DevOps involves.

DevOps applications can interact with legacy systems if we use the right techniques. Make sure your automation solution has integrations with the likes of SAP, Oracle and Siebel, and so enables legacy systems to be enablers on your digital journey rather than obstacles that will stand in your way as you seek to reach full DevOps and end-to-end continuous delivery.

May 10, 2016
Continuous Delivery: The Atlassian Way

Sonatype’s Mark Miller (@TSWAlliance) caught up with Ian Buchanan (@devpartisan) for our 2016 DevOps Leadership Series. Ian discussed his experiences at Atlassian, including continuous delivery, ChatOps, and use of tools such as Bamboo, Nexus, Puppet and Datadog.

https://www.youtube.com/watch?v=MgRaYKHGqeQ

Buchanan: I’m Ian Buchanan. I’m a Developer Partisan at Atlassian, which is developer advocacy for our developer tools.

Miller: Ian, most people know Atlassian from solutions like JIRA and Bitbucket.

Buchanan: Yeah, I focus mostly on Bitbucket and Bamboo. Bamboo’s our continuous integration and deployment tool.

Miller: Good. One of the subjects we talked a little about at dinner last night is how Atlassian is using Nexus. Can you give us some background on that?

Buchanan: Yeah, that’s right. Well, we’re a big Java shop. We have been for a very long time and have a lot of Maven dependencies for external reasons; we also store a lot of our own internally developed libraries in our Nexus repository.

_________________

We’re a big Java shop. We have been for a very long time … we store a lot of our own internally developed libraries in our Nexus repository.

_________________

Miller: As far as a binary repository, are you using it for anything other than components?

Buchanan: Not that I’m aware of. I know that we’ve recently started to get into Docker quite a bit and Nexus is a good fit for that as well. I don’t know the extent which we’re using Nexus for that purpose yet, as Docker’s still quite new for us.

Miller: What does your continuous integration pipeline look like?

Buchanan: It certainly varies by product in the details. Of course, we have our own continuous integration product, Bamboo. Bamboo is the most pervasive tool we have.

For the Java stuff it’s pretty simple pipeline where libraries get built and published into Nexus so that they can be used downstream. We also have some interesting, newer cloud products that are built with Python. A lot of the deployments, whether they are Java or Python stuff, are handled by Bamboo’s deployment projects. They pull artifacts from Nexus and put them into production.

_________________

A lot of the deployments, whether they are Java or Python stuff, are handled by Bamboo’s deployment projects. They pull artifacts from Nexus and put them into production.

_________________

Miller: Are you guys using continuous delivery in house for your workflow?

Buchanan: We’re in an interesting position that I think I see a lot of companies in. We have some behind-the-firewall products (our server products) and then we also have cloud products. In the cloud side, continuous delivery is used when we want to go very fast. But we also have to balance that with the approach we take for our server products so that they don’t fork terribly. We have continuous delivery up until there’s a product delivered. At that point, there is another kind of pick up—where the cloud products engage with other things like Puppet and Ansible to configure the environments with products that other people can pick up and run on premises.

Miller: I was talking to your team in San Francisco. How are you guys using ChatOps with HipChat?

Buchanan: I’ve been talking about ChatOps for quite some time. We do a lot of very interesting things with ChatOps. They play a very important part of our continuous delivery pipeline in that we publish build results in there. People can see when pull requests are ready and check those out. We get to see what’s happening from production as well. We have integration with Datadog; it tells us some of the monitoring things that are going on. It’s not just information coming into the chat room. There are also commands that folks can issue to make the continuous delivery pipeline move along. There are certain stages where you can type a command to our ChatOps and they will perform the necessary actions. Much of the deployment and change management happens in a ChatOps context.

_________________

There are certain stages where you can type a command to our ChatOps and they will perform the necessary actions. Much of the deployment and change management happens in a ChatOps context.

_________________

Miller: Anything coming up in the future you guys are working on that’d be fun?

Buchanan: Well, a lot of the innovation really happens at our quarterly “Ship It” events. That’s where our developers spend 24 hours doing whatever kind of innovation interests them. A lot of time is spent scratching an itch they have. Some of the things that I saw there were about having more information coming back upstream, not just into the chat rooms, but into JIRA where more of the longer-term tracking is going on. They’re experimenting a lot with putting more information into JIRA.

Miller: Nice. Final question, if you were going to be a superhero, would it be Dev, Sec or Ops?

Buchanan: I’ve played Dev roles and Ops roles and I almost feel like I know those much more. But from a lot of the sessions at DevNexus, I have to feel like Sec is the superhero. They’re the superheroes because they solve some unsolvable problems in a lot of ways. But I don’t feel ready to play that role at the moment, myself.

Miller: It’s interesting. I agree with you that they are unsung heroes. Most of the time they just get hammered for what they’re doing.

Buchanan: That’s true. Yesterday I was in a session about securing REST endpoints. The speaker went through problems with BASIC off , DIGEST off and J2EE—all of these things which we use in our products. At the end we’re left with, “What is the answer?”

Well, the answer is that for each of these, there are different problems. So we have to identify the right context. We have to understand where some things are broken, know that those things aren’t perfect, and design for that imperfection. That’s a mindset that we have to carry through dev, ops and sec. I hope more security folks step up and start telling those stories.

May 6, 2016
Trust, You Must: 8 Tips for Adopting CI/CD

We have done continuous deployment (CD) for four years, and enable many of our customers to do continuous deployment with quality assurance (QA). We’ve helped tons of companies adopt and optimize their CD setups. We’ve found that continuous deployment is all about trust — augmented with processes and infrastructure.

To that point, here are some common tips for adopting continuous integration (CI)/CD that I find myself sharing frequently:

Use a CI Tool

This sounds obvious, but you’d be surprised at how many companies try and roll their own. Don’t. There’s no upside for your company—and infinite downside. We love Circle, but there are plenty of other great options. Most are at least somewhat prescriptive when it comes to workflow, which is important when you’re getting started.

Do Unit Testing

It’s always worth reminding everyone that continuous deployment is simply continuous integration applied to deployments. To do continuous integration, you need good unit test coverage. If you haven’t got a solid base of unit testing and continuous integration already, you’re not ready for continuous deployment.

Have Solid Production Monitoring

Bugs and regressions will get shipped; it’s inevitable. Put a system in place, by looking at the data in production, to tell you when a regression or bug was shipped. Tie this into automated rollbacks, so that if a significant performance or key performance indicator (KPI) regression (i.e., signups per minute) appears, your CD system rolls back to the most recent stable version while your team investigates.

Trust Your Team

We talked about the infrastructure requirements, but if you don’t have trust, then continuous deployment isn’t going to work for you. Trusting your team is about giving your developers the freedom to make mistakes, deploy as they see fit and check each others’ code. It is possible to combine continuous deployment with a culture of hierarchical permissions, but we have yet to see that truly work.

Remove Red Tape from the Code Review Process

Your team should review code changes, typically through a pull request, which goes hand in hand with trusting your team. Developers should be trusted to choose the most qualified and insightful person to review their code. One of the most common mistakes among teams trying to adopt continuous deployment is that they only let one or two “senior” developers review the rest of the team’s code, “bottlenecking” all releases. Don’t make this mistake.

Involve Developers Closely in Production Operations

The most common mistake we notice among companies that don’t make a successful transition to continuous deployment is that their development organizations are siloed. Dev and Ops should be joined at the hip. Have your developers deeply involved in the setup and definition of your CD infrastructure, and half the battle is already won.

Shift Left: Test Early and Often

Your team needs constant feedback. Think of your QA objective as getting the right feedback in the right hands at the right time. When it comes to deployments, this means knowing what’s working and what’s broken as the breaking change is made. The sooner you uncover the bug after it’s created, the easier it is to fix. The best companies doing continuous deployment have comprehensive unit test and integration test coverage that runs before any code hits the staging server.

Treat Staging as Your Holy Place

The final pre-production environment is usually known as ‘staging.’ A common mistake is to have developers ship new code direct to staging, while figuring out if the new code is ready to go live. This is a terrible idea. As soon as more than one developer is trying to ship their changes, your Slack channel will blow up with cussing and recriminations. Isolate the pre-production environments to each project, and enforce the “staging as ready to go live” philosophy. This way, integration testing can happen at a self-defined pace, without anyone else’s release being blocked.

About the Author / Fred Stevens-Smith

Fred Stevens-Smith is Co-founder and CEO of Rainforest QA, a quality assurance-as-a-service startup backed by Storm Ventures, Rincon, Andreessen Horowitz, Morado and Crosslink.

May 5, 2016
5 Key Aspects of High Fidelity DevOps

As much as we talk about how we should have shared goals spanning Dev and Ops, it’s not nearly as easy as it sounds. To fuel a DevOps culture, we have to build robust tooling, also. That means investing up front in five key areas: abstraction, composability, automation, orchestration, and idempotency. Together, these concepts allow sharing work at every level of the pipeline. Unfortunately, it’s tempting to optimize work at one level and miss the true system bottlenecks.

Creating production-like fidelity for developers is essential: We need it for scale, security and upgrades. It’s not just about sharing effort; it’s about empathy and collaboration.

But even with growing acceptance of DevOps as a cultural movement, I believe deployment disparities are a big unsolved problem. When developers have vastly different working environments from operators, it creates a “fidelity gap” that makes it difficult for the teams to collaborate.

Before we talk about the costs and solutions, let me first share a story from back when I was a bright-eyed OpenStack enthusiast:

I was at the fourth OpenStack Summit in Boston. My team at Dell had created open deployment automation, Crowbar, that could turn raw servers and install a fully working OpenStack cluster in about 30 minutes. I don’t want to oversell it; there were plenty of rough edges and preconditions. However, the automation worked and OpenStack came up like a kudzu seedling. In fact, we’d actually brought a half-rack of servers to the summit for live production demos.

On my way back up to the demo rack, I was riding an elevator with one of the founding developers of the project. He was positively glowing over a new project that he’d created called “Devstack” and wanted to show off the cool logo they’d designed. He gushed, “It’s awesome! It runs all of OpenStack in a single python environment on my laptop! You don’t even need Vagrant anymore—it’s a super fast and easy for developers to get OpenStack running because all the configuration is in Python.” I watched as he typed a command, text scrolled and welcomed me to the OpenStack API. “See, now I’m running OpenStack!”

At the time, it seemed like a harmless toy, since a single-node cloud deployment is useless to users, testers and operators. I pointed out that we’d gotten our multinode automation working in virtual machines (VMs) now so it would be easy to use even without dedicated hardware. He asked what we’d written it in. “Mainly Chef with some Rails UI,” I responded, thinking we’d made some solid 2011 DevOps choices. He was less enthusiastic, because we were not using the developer-preferred Python. Even so, I encouraged him to come by the rack for a demo, but without any anticipation.

We left the elevator on a friendly wave of OpenStack synergy. He sprinted into the developer lounge and I trudged back to the user and vendor showcase. Even without words to describe it, I saw that the project had started diverging between developers and operators. Developer collaboration drove a need for something fast to set up that ran on a single laptop. More importantly, Devstack stripped away all that messy multinode networking and DevOps tooling that made it hard to configure and run OpenStack. In what turned out to be a one-way journey, there was now a universal developer environment.

Unfortunately, that developer environment configuration did not translate into production.

In fact, OpenStack deployments are notoriously challenging and there is an overabundance of divergent approaches. Even with so many efforts, few have managed basic operational milestones such as upgrades and high-availability configurations. I believe a wide fidelity gap is a key, if sad, part of this story.

We don’t have to live with fidelity gaps!

In fact, DevOps exists to close those gaps by bringing developers and operators to a shared space. You are already well-versed in the need to treat development as a pipeline with automation to keep it repeatable.

If we know the challenge, why do we allow these gaps? It’s because fidelity can have too high a cost in terms of complexity, system overhead and time. These costs multiply the faster we try to iterate. That creates a conflict between the organizational benefits of agility and fidelity; sadly, few teams have been willing to burden developers with fidelity or operators with change frequency.

How Can We Make Fidelity Cost-Effective in Initial and Ongoing Costs?

There are five components to consider: abstraction, composability, automation, orchestration and idempotency/repeatability.

Platform abstraction

Platform abstraction hides the differences between environments and tools that make developer and production environments different. For true fidelity, we use “platform” in a very broad way. A cloud API or server is a platform, but so is an operating system or container system. In practice, there are many possible and acceptable abstraction layers.

Teams need to invest in ways that allow the same automation to work on multiple platforms in the same way. That requires modifying the automation to be resilient in different environments and accept configurations instead of making assumptions. These changes are good hygiene, anyway, and make your automation more robust and portable.

The simplest way to add abstraction is externalize environmental configuration. Since it’s difficult for operational systems to detect the broader system configuration, we need to inject that information into your deployments. Unfortunately, that leads to the “massive pre-deploy inventory file” antipattern and our next component.

Composability

Composability goes beyond making automation modular. It means that we can flow information, à la functional programming, through the discrete parts of the install.

In a good composable design, each unit has defined inputs, outputs, preconditions and behaviors. That allows us to chain them together to accomplish work. Our systems should be able to connect outputs from earlier modules to inputs from later ones. This eliminates the “pre-configuration file of doom” because we can build up system knowledge as the work is done. Since the functions are well-known, we can determine if we’ve completed all the needed setup.

Substitution is even more important than having a chained configuration flow. Since modules are isolated actions and known parameters, we can create alternate modules based on user- or infrastructure-driven requirements. For example, choosing CentOS vs. Ubuntu changes package management; however, downstream applications only care that their packages are installed. We need to make it acceptable to detect and adjust modules based on situational awareness without breaking up our downstream components.

This type of change tolerance is essential to fidelity.

Automation

You may assume this is obvious; however, we all have a tendency to overlook manual steps or setup in our daily work. Common statements such as, “It takes a day to build my dev environment,” or, “We always run this pre-config script the systems are up,” indicate that we are not as automated as we want to admit.

Fidelity gaps open when teams have different processes.

We don’t expect developers to use hundreds of physical servers; however, we can expect them to isolate components in ways that are similar to operations. That adds overhead to their work that requires automation. We don’t expect operators to reset and redeploy systems 10 times a day; however, we can expect them to use repeatable configurations that do not require manual intervention.

Remember: It does not have to be the same automation in all cases! Composability allows us to have equivalent but different paths to the same state.

Sequence/Orchestration

There is a downside to composability. If you’ve replaced your fragile monolithic automation with robust composable modules, then you’re going to have to add some complexity to chain them together. It an unavoidable tradeoff because the composite system needs glue to hold together. That glue is orchestration.

Truthfully, adding orchestration is not a downside. It is a necessity for scale. Any multisystem configuration process requires orchestration to be sustained. The nature of scale applications is that we have to coordinate activities horizontally across the system during deployment, scaling and upgrade. That allows us to create trust relationships between nodes, set up networks before trying to use them and coordinate servers and clients while we upgrade APIs. Wait loops and iterative convergence are difficult to troubleshoot and more difficult to predict.

There is a concrete benefit when you combine composition and orchestration: Small work units create a system that is very transparent and parallelization in operation. Those are key values in scale ops.

Idempotency / Repeatability

The last component is one of the most elusive: idempotency. The objective is to allow multiple runs of modules in our automation system to leave the system in a consistent state. That allows us to repeat execution of a component without harming or undoing work that we’ve already done.

The concept is not simply skipping work we think we completed! If we rerun an action, then the action should exit in the requested state every time. If the state is already good, then do nothing. If the state is not configured, then create it. If the state is not good, then fix it. Basically, we want to be confident that we can achieve our end goal every time but never make unneeded changes.

This is not a simple task! The nature of configuration is to change systems, and we usually assume that we are going from unconfigured or misconfigured to correctly configured. For idempotent scripts, we have to guard against going from correctly configured to broken.

This may seem like a lot of work for developers who usually start from clean systems, but it’s life-saving for operators when a timing error causes a unit to fail.

Hey, You’re Adding a Lot of Work to my Deployments!

Following these five principles does add some overhead; however, all of the changes are good hygiene anyway. They make your automation more robust and portable. More importantly, they reduce friction between your development, test and ops processes.

Deployment fidelity is a real benefit that helps drive success. Hopefully, these five principles have made it a more concrete and obtainable goal.

May 4, 2016
Defining What DevOps is Not

It’s not often that my emails turn into full-blown conversations on DevOps. But when it happens, it’s a great time to take advantage of the insights. John Purrier of Automic and I recently discussed DevOps sustainability. Both of us think that sustaining a modern delivery chain and adopting it are two very different things. However, most only consider the latter. And because of that, in two years (or much sooner), they will find themselves in the exact same place they started—in an environment that is unable to adapt, rooted in habit and suddenly the old guy on the block.

(more…)

April 29, 2016
DevOps Days London: The Human Factor

Day 2 of DevOpsDays London 2016 kicked off with a presentation from the legendary Gene Kim. The theme that dominated today’s proceedings was the “human factor,” (fitting, as we finished Day 1 with #humanops) nicely started by Gene’s quote from Nathan Shimek: “We need DevOps to make our work humane.”

The second day delved beyond what makes people happy, posing questions regarding what it is to be human, and why automation is not everything when it comes to DevOps. As Gene said, “DevOps transcends the technology stack we are working with.”

DevOps aims to supplant pain with joy and help people find autonomy, mastery, purpose and deep satisfaction in their working lives in technology. Werner Vogel, CTO at Amazon, is often quoted as saying: “You build it, you run it.” But Gene took this further as he explained how a developer had described his most satisfying working experience as coding something and deploying it and monitoring it and fixing it, too. He also touched on the power of shifting left, using Test Driven Development—and its demand that we write the test script in before we even start to code—as an example. That’s building quality in. That’s cross-skilling and cooperation.

It resonated with the audience when Gene said there was one single question that, when answered, tells us everything we want to know about an organization’s DevOps maturity: “To what degree do you fear deployments?”

We came back to this question later in Jeromy Carriere‘s talk about his experiences of scaling at Google. Going the gym for the first time is painful—we have to build muscle memory, he said. We need to practice these processes until they are like breathing.

Gene made a book recommendation: “The High Velocity Edge,” by Steven J Spear. When we do DevOps we are aiming to transform into a dynamic learning organization. We should not expect but be required to talk about problems. No fear, no blame. Everyone in an organization should be trained to run a blameless post-mortem—or a healthy retrospective, if you prefer. As Claire Agutter put it: “Improving daily work is more important than doing daily work.”

Gene then outlined the DevOps Enterprise Summit London event coming up at the end of June and the extraordinary stories of experiences people share about their DevOps journeys during DOES events. He said: “Often the transformation leader was putting themselves in personal jeopardy.”

I can testify to that. It takes courage, optimism, conviction and tremendous amounts of energy to be a change agent.

The human theme continued into the panel discussion as the COO of Technology at Barclays explained how its command and control structure, waterfall-ruled development and volume of changes resulted in total gridlock. He said that the big challenge is trying to re-engineer the organization to behave differently: “Changing HOW we do our work is the hardest thing.”

Changing human behavior is very challenging indeed; as Joanne said, measurements have to change and we have to focus on customer value. Gareth Rushgrove also majored on the human angle in his talk on DevOps and microservices. He said: “Microservices are as much about people as technology,” and, “Which comes first: the software or the team?”

Essentially, DevOps and microservices help us to make lots of small bets—spreadbetting, if you like—spreading risk and accelerating feedback. And we need small teams to be able to do microservices and move away from monolithic applications.

One particular Ignite talk stood out from the rest, and won the hearts and minds of the audience: @coldclimate from Server Density. Through a personal story of a back injury brought on by prioritizing work over looking after himself, the #humanops message: “You are not immutable, you are not highly available!” struck home hard.

So there we have it. My key takeaway from DevOpsDays London 2016? DevOps is less about technology and automation and more about happy, healthy humans.

April 28, 2016
Reasons Why Developers Should Test More

When I got started as a professional software developer, I thought awesomeness in development was measured by the lines of code (LOC) per day and how quickly code got to QA for testing. Coding a few thousand lines overnight gave one instant bragging rights, regardless of how well the code worked (or didn’t) in the morning. The more senior guys always acted like they knew better, suggesting that it’s not about the quantity but the quality. But I was too young and arrogant to listen. And, of course, I had to learn it the hard way. My seemingly amazing overnight code would show cracks once under the microscope of the QA team.

True today as it was then, most mistakes are tiny and easy to fix, but even small problems require a tedious set of steps from when a QA engineer finds the mistakes to when the developer fixes them and sends them back to QA for verification. Some bugs are the size of the Mariana trench and require weeks of work and thousands of additional LOC. And, even new code, once resubjected to QA, can generate bugs, therefore repeating the cycle without an end in sight. Of course, by this time projects are late and developers are tired, frustrated and burnt out.

After this happened to me too many times, I realized that maybe I was wrong. Maybe the experienced guys were on to something with this quality thing. They never seemed too tired, and their projects were in much better shape. I decided that for my next feature, delivering quality instead of LOC would be my priority.

I’m sure you think you know what’s coming next. I’m going tell you that I tested my stuff from that day on and it worked perfectly—no more boring bug-fixing. You think I’m going to tell you, you should do it too, right? Well, no. I actually spent a lot of time in my second attempt instrumenting my code better so that I could debug it faster as the need arose. I also spent a lot of time writing my own test tools because I didn’t have one. But I kept moving forward and learned the hard way. Here is what I took away from those moments:

Develop More, Bug-Fix Less

There are two common reasons for developers’ involvement in a project. Developing a new feature or enhancement is about creativity, as it gives one the opportunity to design and implement new things. And then there’s the challenge of bug-fixing, which is all about maintaining existing functionality within older, already-made things. As a developer needs to do both, often at the same time, minimizing bug-fixing time is important to have more time in which to actually be creative and develop new functionality. And the best way to minimize bug-fixing is to test code more, as it’s developed, to avoid the death spiral I describe above. Good initial quality will mean QA will find fewer bugs and it is very likely that the criticality of the bugs found will be greatly reduced. As such, fixes no longer would be a struggle on short notice; shortcuts that generate more bugs don’t have to be used; and developers can focus on new features as opposed to fixing bugs. In all, the reasons are to develop more, and bug-fix less to be more productive.

Be More Reliable

Bugs also are nasty in that they tend to appear really, really late in the game. The Murphy’s Law on bugs should be something like, “The hardest bug is found one hour before the release is pushed to production.” This typically happens because everyone is busy peeling the onion, finding and fixing bug after bug until they get the “system” stable enough for QA to crush it with those large-scale system test cases that expose that one nasty bug. It is an artifact of waterfall development but also happens in agile development, as no agile team can afford running long duration tests in a sprint. So what happens then? Who ends up in crunch mode? The developer. To avoid this, developers need to find and fix the easier bugs sooner, as they write the code. That way, the onion is half peeled once it reaches the QA team. The nasty bug will be found much sooner—if the developer did not find it and fix it already. In short, developers can be more reliable by avoiding last minute bug-fixing. And we all know, reliability in execution is fundamental for fostering trust with management. A trusted developer is the one that gets to work on important, meaningful projects.

Be More Flexible

While in an ideal world the design of a feature is not changed frequently, in real life the exact opposite is true. At some point, even regularly, developers will have to modify the code as the scope of the features keeps changing. Massive code changes such as refactoring, if not done right, can send developers right back to a long, teeth-grinding, bug-fixing period. Avoiding it by testing early, as they change the code, offers a significant advantage—flexibility. Developers would be able to accommodate required changes more quickly.

Agile Development is Critical

Let’s do a quick recap: By simply testing code as it is written, developers can spend less time fixing bugs and more time developing new features, to become more productive, more reliable and trustworthy and even more flexible in implementing changes. This takes us to our final reason for early testing: Agile development is no longer nice to have, but business-critical. All of the attributes described above are quintessential for succeeding in an agile team. Developers are expected to be flexible, reliable and build a lot in short time periods. The days of trial and error, of learning the hard way, are gone. Today, we need to go from design to development to testing to going live in days, not months. Quality is what differentiates products. Developers need to do all of this to be successful.

About the Author/Gabriel Chiriacescu

Gabriel Chiriacescu is the director of Software Development at Ixia, with more than 13 years of experience in embedded software development. He is currently leading the product management effort for the new Ixia Developer solution, an agile test tool designed specifically for developers.

At Ixia, Gabriel has led multiple engineering teams responsible for developing market-leading test solutions in a variety of technological areas, including wireless, broadband access, data center, triple play, authentication and security.

A native of Bucharest, Romania, Gabriel holds a BS in computer science from the Polytechnic University of Bucharest. After living several years in Los Angeles, Gabriel is now based in North Carolina and enjoys discussing CICD, SOA design patterns and the evolution of software development.

April 27, 2016