devopsy.kinsta.cloud

Tag: continuous delivery

Carving through the hype – take the DevOps maturity assessment

As anyone who spends time with their family over the holidays can attest, the notion of maturity is a highly subjective concept.

Whether it’s sizing up the next generation’s advancement or defaulting to time-honored inter-sibling behaviors, annual get-togethers offer tangible proof of what’s new and those things that likely won’t ever change.

In the domain of DevOps – an umbrella concept that encompasses a massive array of processes and supporting capabilities – gauging the maturity of your organization’s standing is also a highly interpretive task.

Given that “DevOps” means something different to nearly every organization, based on its unique makeup, methodologies, tools, and most importantly, business goals, attempting to measure progress against any outside model may seem an effort of limited value.

That said, as an increasing number of organizations seek to embrace DevOps, and growing recognition that well-worn stories of leading-edge “unicorns” don’t relate closely to most people’s real-world expectations, many teams are left wondering where their own efforts stand, especially in comparison to their peers.

To help acquit this pursuit, and allow organizations to scope their progress, CA recently launched the second iteration of its online DevOps Assessment.

Split into three short surveys covering core limbs of the DevOps tree – “Develop and Test”, “Release and Deploy”, and “Manage and Monitor” – this free tool offers detailed insight into how your organization’s maturity stacks up to that of other practitioners.

For instance, in the domain of development, how well do your teams fare in defining and managing requirements, and then presenting them to others in a structured manner? Related to release cycles, are your teams truly meeting business-driven expectations for more frequent application deployments, shorter lead times, and higher success rates? Or, when it comes to applications monitoring, are you able to distinguish the most critical and recurrent problems and how they truly impact business services?

These are perhaps some of the most fundamental questions that one could apply to measurement of DevOps adoption in general, but further investigation attempts to take a closer look at more finite, underlying issues.

Are your development teams accurately replicating realistic conditions from production systems to create virtual services and testing scenarios? Are your application release deployments fully automated across development, test and production environments? Can your teams monitor complete business services that link user experience, transactions and applications to infrastructure to pinpoint any root cause issues?

The best thing about the survey is that it immediately provides visibility into how your answers compare to those of previous respondents. There’s no fuzzy math or editorial interpretation (though we may highlight trends in a report at some point), merely instant validation, and the option to create a customized report card charting where your efforts stand in relation to those of your peers.

If you’re wondering precisely just how to achieve your long-term DevOps goals comparative to where you stand today, it might even be worth leveraging some of the involved questions for the purpose of evolving process and methodology.

The only remaining questions are: Do you think that you’re ahead of the curve? Are you afraid to find out just how far you’re behind the crowd? And most importantly, what do you have to lose?

Take the survey today and get a real-world fix on your overall level of DevOps maturity. Guaranteed it’s a far more informative activity than debating family and friends over whose turn it is to carve the turkey, or wash the dishes.

December 3, 2015
To Be Continuous: Continuous Delivery and Mobile Development

In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk with Kevin Henrikson about about succeeding with Continuous Delivery and mobile development. This podcast is brought to you by Heavybit, a 9-month program for developer startups.

Kevin is partner director of engineering at Microsoft where he manages the Outlook teams.

Edith is CEO and co-founder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

Paul is founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

[soundcloud url=”https://api.soundcloud.com/tracks/235452076?secret_token=s-a8Rxo” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

Paul: All right, Kevin. So, what’s your favorite thing about Continuous Delivery?

Kevin: Good question. So, Acompli started kind of the approach, when we first started was just a couple people. And our deal was like, “Hey, you’re trying to get this thing to work, send emails.” So it was like, keep checking it in, get closer and closer to kind of the definition of it was working.

And then as we started to experiment farther out, we said, you know, “What’s the right sprint cadence?” And we’d come from kind of traditional enterprise companies, where, you know you ship every six weeks or every four weeks and we’re like, oh, we can go faster.

People are doing this agile thing. It’s like two week sprints, and we tried two weeks and realized that most of the work came in on Thursday afternoon or Friday kind of thing, ’cause the end of the second week and so what if we just do one week then most of that work will come in, you know, the first Friday. And sure enough, it worked, right? So 80% of the work still comes in on the last day or kind of completes on the last day or gets to the point of ready. And so, what we liked is that it gave us a much smaller kind of chunk of deliverable, right.

So, like the train keeps moving but the size of what’s on the actual boxcars is smaller and so if you have a problem, it’s easier to roll back, easier to fix.

Or if you have a problem, you can just patch it quickly ’cause you’re used to moving quickly.

Edith: Wow. Okay.

Kevin: So unlike in a traditional world where you ship something every six weeks, you’re like oh, we made a mistake, we need to go patch this. Like, the patch felt like an exception and you’re like, your muscle memory for patching was not there, right? So you’re like do we do it now? Is it worth it?

And then it’s like well, if you’re five weeks in and you discover that thing, it’s like well, the six-week thing is out there where with, if you’re shipping every week, you’re like oh let’s patch it. We shipped yesterday? We just click the button again, and it builds and it goes. So you’ve built the muscle memory to move quickly, right. And so I think, what I like the most, kind of going back to the specific question, is just the fact that it lets you get good things done quickly, but also correct mistakes quickly.

Edith: Wow. So this is a good time for you to introduce yourself.

Kevin: Cool. So, Kevin Henrikson. I currently manage the Outlook, kind of non-Windows teams at Microsoft. So Outlook for iOS, Android, and Mac. Previously, I was VP of Engineering at Acompli, which was a mobile startup, focused on iOS and Android.

Edith: Yeah so you had a lot of good thoughts right then. One of the things I wanted you to talk more about is why do you think 70-90% of the work is always on the last day?

Kevin: Yeah, I think people are kind of deadline driven. I think generally they want to work until the last minute and I think software development ends up being this thing where you’re never done. And so the only way to define done is to actually draw a line and say this is done and then still some things miss that line. And so the tighter you give the window for things to get done, the more often they’ll hit that window because you’ve kind of broke things into smaller chunks.

And also, because software’s kind of this long-lasting thing that never is kind of done done that you end up with, if the chunks are small, people are able to estimate and think is that going to take me a day? Very rarely does somebody think something’s going to take them a day that takes months.

But very often, like if you think something will take months, it may take many, many months. Because it’s hard to guess, there’s so many variables, so many things are going to change in the environment, turn what you’re doing before you get there.

So, I think the short answer is if you start with a smaller window and more discrete small tasks, those tasks are easier to estimate, they’re easier to consume, they’re easier if you get disturbed, like oh I’m going to go down and get a soda, or somebody asks me a random question ’cause some fire comes up, I can kind of make it up because I was only expecting to spend half my day working on that anyways.

So, I think that kind of works out in our favor.

Paul: So, there’s an implication with what you said there, that the tasks sort of grew because it was two weeks,, rather than–

Kevin: Yes.

Paul: So, I guess the phrasing is, when it was two weeks, was it that a one-week task had grown to two weeks? Or was it that things got cut when it got shrunk down to one week, and you ended up with technical debt, or something along those lines?

Kevin: I think it was a couple combinations. I think if you feel like you have two weeks to work on a task, you’ll spend more time polishing or potentially doing things that aren’t on the critical path to getting it done.

And I think that’s natural, right? Like, maybe it is you’re paying off too much technical debt, right? In the start-up world, you have to have that lever. And even just in general software development, like, you have to have a lever. We’re not launching the space shuttle here. There’s room for some wiggle room and imperfection.

And I think breaking it into small pieces lets people focus on, “Hey, I’m only gonna do this” and “I have this much time to do it.” Where you say, “Hey, it’s got two weeks,” you may be deep into the second week and saying, “You know, I just don’t like my approach. I’m gonna change it.” And you basically end up throwing away a week’s of work, whereas, if you’re thinking that “Hey, the deadline’s coming in a week,” you’re gonna make that kind of epiphany, like, “Oh, I made a mistake,” or “I’ve taken the wrong path” much quicker.

So, I think it allows you to break your decision-making and optimize on getting it done, because the window is kind of quickly approaching, being a one-week spread versus two.

Edith: This might sound like Seven-Minute Abs, but why not two-day sprints?

Kevin: Yeah, so I think in two days we– or one-day sprints, right, how do you continue to make it faster? So, for us, there was sort of one artificial gate, which was the fact that you really can’t ship quicker than about a week with Apple.

And even then, it sometimes slips by, because, you know, four or five days, and then during a peak season, if there’s lots of apps, like a Christmas holiday, or around a platform update, you’ll see a spike in the review times. So, Apple kind of, and we run iOS app first before we port it to Android. So that ended up driving us towards a little bit to a week. Cause it was like, if you shipped on Monday, the app would get in the store some time end of that week. And that was a natural cadence to start the next one.

So, it’s really hard without pushing the emergency release button, which Apple only lets you do once or twice a year, to ship more than once a week. And then, we kind of let the rest of the team fall behind that. If you’re only shipping the app once a week, let’s keep the new chunks of new work, or big bug fixes to that path on our service.

The way our architecture is, and we can talk about that more, we spend as much, you know, 77% of the logic of the app is down deep in the service, because that we can update all the time. So, when we push a release and don’t wanna fix something, we fix from the service the same day.

Edith: Interesting.

Kevin: So, that allows us to have a much more agile approach, because the service is constantly being updated, but the app is kind of gated in one-week chunks.

Paul:

So, what you’re saying is that the biggest impediment to your shipping frequently is Apple.

Kevin: The platform, yeah. The platform, basically, Apple’s– And specifically, it’s the fact that Apple has a review process, which is great. That Apple review process generally keeps the bar of apps better and puts some good discipline inside of the wider range of development, people that are coming to the Apple platform.

But, it also, when you wanna move quickly, you can’t. The platform itself kind of enforces this four to seven-day lag in shipping.

Paul: So, you’re now running teams that are shipping on non-Apple platforms. Did you guys make any changes to your release cadence as a result of that?

Kevin: Yeah, so I think on Android, for example, we still kind of keep to the one-week junk of we plan a sprint. Build for a week. Release to our beta group or dog freed users, and then release to production. But Android gives you this advantage where you can do percent-based rollout.

So, with Android, we end up releasing usually two or three times a week. So, you’ll release kind of The Release on Monday, at like 1-2%, kind of walk that out three, five, 10%. Usually at five or 10%, you can pick some new signal of crash or behavior that you wanna adjust, make a hot fix, push another release, start at 10%, 20%. Oh, you see another one you wanna tweak at 20%. And finally, from 20 or 30, you end up going all the way forward.

Paul: Do those Android users help you on the other platforms, as well? Are they finding crashes that also exist on the other platforms?

Kevin: Today we’re 100% native on both, so there’s no shared code between iOS and Android from the mobile app perspective. Obviously from the service, it’s 100% shared code. So, what we’ve actually done in the past, is when we launch a new feature that we think is either controversial, or has a high chance of risk, for example, when we launched iMap support.

We launched generic iMap support on Android first, knowing that we could patch it quickly, and adjust if things went wrong, simply because there was no way to test the millions of random iMap servers in the world and get any kind of focus. When we talked to Gmail or Yahoo, or– Single point.

Paul: That’s really interesting, actually, the idea that because the rollout, or the sort of safety features are only supported on Android, that people will end up building features there first.

Edith: How deliberate did you have to be about putting stuff in the service, versus in the apps? Was this the lesson that you learned?

Kevin: So, we had always thought of the service as the place where we always wanted to do as much of the logic that could break us– You just don’t wanna do it twice. You don’t wanna implement active sync twice. You don’t wanna implement an iMap stat twice.

And it also allowed us to pick a protocol between our cloud service and the mobile app. It was optimized for mobile. So, think of the API, the shape of the API looks closer to the screen of the app. There’s a request for “Get my inbox,” and then make a change. Where, if you think of iMap, or the mirror out of email protocols, that we talk today, they don’t have that very linear type of approach of “Give me this screen. Give me that screen.”

They’re thought of in a way where you’re going to sync all the mail down to your desktop client, and then manipulate it locally, and then sync changes back up in some either connected or disconnected state. So, the service was basically set to put complex logic, but also to allow us to get leverage, ’cause everything we could build in the service was one less thing we’d have to do twice on the client, because we picked a path that we didn’t support. We didn’t basically have a lot of shared clip or clients, essentially none, just the API.

Paul: And it’s the same service that sits behind all the different clients?

Kevin: Yes, so we have a single cloud service that basically powers both the iPhone client and the Android clients.

Edith: It’s interesting, so at TripIt we had a lot of similar issues. Like, we were at one time one of the biggest, a lot of our users are Gmail.

Kevin: Okay, awesome.

Edith: People would sync their Gmail with us, and their calendars. Did you find there was a difference between the you or the app experience that Andoid and iOS wanted? How much did you bifurcate that?

Kevin:

Sure, so, a lot of people ask is like, “Hey, did you launch a feature first on Android or iOS, “or do you always have to make them “come out identical?” My answer to that is pretty simple. Most normal people carry one phone. They’re either an Android user or an iPhone user.

So, they don’t know that you released it first on Android or iPhone, because they don’t carry both phones. So, we think of that, from a feature point of view, that when they’re done, then we ship it. Now, if we’re trying to line a press event around something, we’ll wanna make sure they’re both there before we do the press event. One may come out a day or two before the other. But, generally speaking, we don’t try to, you know, legislate you have to do it this way, ’cause it works on iPhone.

In fact, we actually took a different approach, where we said, We wrote it one way on iPhone and then we went to Android. It wasn’t just like, “Oh go port all the screens over.” It was like, “What would this screen look like for an Android user?”

So, we want our apps on the mobile device, and most mobile users wanna look at an app and feel like it fits in with the rest of their phone. So, the way that an Android navigation works, the way that people use the back button, or not, is very different on iPhone and Android. The fact that there’s widgets, and the way the widgets are exposed on Android, and the way that those people want to integrate deeper into the platform.

Those kind of things we let go with the platform. Our view is write it in native code, ’cause it allows us to take advantage, and move quickly to the platform-specific features. But it also lets us tailor the experience, and the UI and UX to whatever that user base wants.

Paul: It’s interesting that’s, that you talk about it that way, because in the old days, kind of ten years ago, when people were making multi-platform applications, there was a tendency of people to try to make it all match the same, the Firefox is an example here, or Chrome one of those tools that people use on all platforms, was designed to look the same on every machine, even though people didn’t have that machine.

I think one of the nice things that’s happened with the mobile ecosystem is people have thrown away that concept, and they’ve made it feel actually native for each of the platforms.

Kevin: Yeah, and I think that also goes to feature set, right? Like, you would think that the traditional app is like, “Oh, if it works this way on Mac, “or on Windows, or on the web, you would expect kind of 100% feature parity between all those platforms. And today on mobile, we go and select where you actually use it. There’s features that exist in mobile only.

Because mobile has all this additional context of your location and where you are, and what you’re doing. Are you running? Are you driving? There’s other sensors in the devices today that allow you to build specific features that are richer. So, we always thought of a commonality.

So, rather than sending more emails, send better email, right. Don’t say, “I’m running late.” Say, “I’m running late and I’m here,” and send a quick tap of a button to put the map and show you that, “Oh, I’m 20 minutes away,” or, “I’m on the train,” or, “I’m at the train station,” so that people can understand you don’t have to type out how many minutes it’s gonna take. You don’t know. Or, you walk in, or you drive in–

Edith: Plus, people lie all the time. Like, “I’m on my way” has a pretty loose meaning of anywhere from “I might leave my house in 30 minutes” to–

Kevin: And depending on the context, right? What that person’s thinking, so yeah, good point.

For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.

December 1, 2015
A 6-pack for building a business case for CD

There are all sorts of compelling business benefits awaiting those who adopt Continuous Delivery. It has the potential to reduce operating costs, reduce defects and redundant work, improve efficiency, and result in a much faster delivery speed. By releasing applications to market faster, you boost your chances of success and gain a competitive advantage. But how do you get from the theory to the practice?

If you really want to scale up and realize the benefits of CD, then you’re going to need money to retool, retrain, and change the existing culture. In order to secure the budget you need, all of the key players in your organization have to buy into the change. A strong business case is essential. Here are six key areas to look at when you build yours.

Production incidents

A natural place to start with building a business case for CD is with production incidents, because they’re very visible. When you perform a root cause analysis on an incident, you’ll often find that the software worked fine in the development environment. By standardizing the environment configurations you can reduce the number of incidents, free up resources from the provisioning process, and reduce wasted time.

It’s possible to calculate the cost of data center downtime for your company, and you can use root cause analysis to find out where provisioning errors were to blame. From that, you can extrapolate the potential savings made by reducing these outages with standardized environment configuration. You also have to consider the reduction in wasted time and effort, resources that can now be productively employed.

Deployment failures

Take a look at your past deployment failures and work out the average cost to the business of rolling back the deployment, diagnosing the cause of failure, and remediating. By automating deployment you can cut these costs completely.

Manual testing

To employ an API-based testing model that can be integrated into your delivery pipeline will pose an upfront cost, but that cost per build decreases with each new build. With manual testing, the cost rises with each new build. Automated testing gives you better coverage, earlier. If you can find defects earlier they tend to be quicker and easier to fix, which reduces the potential knock-on impact in terms of production incidents, and, ultimately, reduces costs. You’re also cutting manual labor costs and overhead.

Manual build process

If someone on your team is writing scripts to automate your builds, there’s a labor cost involved and they may cause a potential bottleneck. If you can switch to a centralized delivery pipeline where the build is automated and which development teams can customize for their needs, you can reduce the time and effort required. You also free up the developer who would have written the scripts to work on improving the product instead. At the same time, you’ll boost oversight, control, and reliability in the build process.

Better analytics

A lot of development time is wasted on building features that don’t add value. According to research by Forrester, only around a third of the ideas implemented result in a measurable improvement in the customer experience, a third have no obvious effect, and the final third actually have a negative impact. By building analytics into your code, and constructing a strong feedback loop, you can find out what really improves the customer experience and focus your efforts there. The earlier you can get feedback, the easier it is to prioritize and maximize the value you’re adding.

Focus on the big picture

Most business cases might focus on some of the benefits we mentioned above, and that’s because they’re easier to measure. But it’s the intangibles, such as improved competitiveness, goodwill, and revenue, which are harder to quantify, but which have the greatest impact.

You want to present the potential cost savings, but this is about more than cost cutting. The long term benefits filter through as you ramp Continuous Delivery up. Faster feedback loops and a greater understanding of your customer are laudable aims for any business. You’ll have to delve into specifics to generate the persuasive numbers you need, but it’s important that you keep those intangible benefits front and center as part of the business case you’re constructing.

About the Author/Andrew Phillips

Andrew Phillips heads up product management at XebiaLabs, a provider of continuous delivery automation. Andrew is an evangelist and thought leader in the DevOps, Cloud and Continuous Delivery space. He sits on the management team and drives product direction, positioning and planning.

November 10, 2015
Continuous Delivery versus Continuous Deploy
Getting a solid grasp on the meaning of DevOps requires a clear understanding of the terms used to describe it. Over the last several years, we have seen a paradigm shift in the way software is developed, moving away from a waterfall approach, with monolithic releases, to an agile approach with incremental releases. This shift has changed the way in which the Application Lifecycle (ALM) is managed. Focus around what we normally would refer to as ALM is now more accurately described as Continuous Delivery. Continuous Delivery is not a set of tools, it is an emphasis on people (culture), tools and processes. The change from ALM to Continuous Delivery includes three very basic principles:
- The software is always deployable which means you prioritize keeping the software deployable over adding new features. To do this requires smaller, less risky changes released overtime instead of all at once.
- A feedback loop of the build, test and deploy steps expose the readiness of the software allowing anyone from development, test or production to determine that an immediate release of the software to any environment can be approved.
- Automation of the build, test and deploy process is critical in order for the process to be repeatable and to support a higher frequency of builds, deploys and test cycles.
So like ALM, there is an awareness of a lifecycle (Dev, Test, Prod). But unlike ALM the lifecycle should not be seen as linear. With Continuous Delivery, a software update may move from Development directly to Production without having entered the Test phase. Because Continuous Delivery principles support keeping the software in a deployable state, with a visible feedback loop to determine the readiness of the software, software can be confidently moved into any environment at any time.

This change between a linear, waterfall approach, to an non-linear agile approach is far more realistic in the way organizations manage their software. Over the course of my 20+ years’ experience working with both large complex enterprises and small software teams, the need to be able to perform a release, into any environment at the ‘drop of a hat’ has been consistent and problematic. The principles behind Continuous Delivery defines a more realistic approach to software development.

Moving from a linear application lifecycle approach to Continuous Delivery includes making changes at many levels. First, developers must begin embracing Agile practices, managing smaller, incremental releases. Because Agile increases the frequency of builds, testing and deployments, DevOps tools must be in place to support the automation of these tasks. In the waterfall approach, teams had the luxury of time to get build scripts to work, execute test manually, and write a deploy script for every environment. Continuous Delivery does not offer that luxury therefore automation of these tasks becomes critical. This is why Continuous Build, Continuous Deploy and Continuous Test tools are implemented. Continuous Build addresses the frequent building of code changes, updating binaries upon a check-into version control, and reporting on what artifacts were impacted providing feedback into the readiness of the build.

Continuous Deployment automates the movement of those binaries to any endpoint in any environment and supports server configuration management, all performed on an incremental basis with feedback showing the inventory and the changes. Continuous Test automates testing so developers and/or testers can run through critical functional test, and code analysis to provide feedback on the readiness of those binaries.

To maximize on automation, these steps are chained together using the workflow engine of the Continuous Integration Server. The workflow can be triggered upon a source code check-in, kicking off a compile/link (build), running code analysis, deploying to an endpoint and running test. In Continuous Delivery, each environment may have their own Continuous Integration workflow that performs the steps needed. For example, a Production CI workflow may just perform activities related to moving the code to hundreds or thousands of endpoints, with database updated, but not execute a build or test.

Not all is perfect in the Continuous Delivery approach. Automation continues to be a bottleneck. As with most paradigm shifts, changes in behavior and adoption is a major hurdle. In most companies, the use of manual scripts to perform builds, deployments, test automation and manage infrastructure are still called under the covers of the Continuous Integration workflow. These are the same scripts that were used in the waterfall approach, except now there is not the luxury of time to fix the scripts or update them to adjust to software changes. To support Continuous Delivery, the build, deploy, test activities must be elastic, automatically adjusting to code changes, database updates and server configurations.

To create a more dynamic Continuous Delivery process, DevOps tools are moving away from the need for coding one-off scripts and instead moving to a model driven approach with Playbooks and Plug-ins. Many developers still hold on to these scripts with the argument that they do not want to be at the mercy of a ‘vendor’ or ‘tool’ to manage these low level tasks. As Continuous Delivery pushes the frequency of releases, this final paradigm shift, moving away from the old approach to one-off scripting, will need to move into full automation. Continuous Build will be supported by Build Automation, Continuous Deploy will be supported by Application Release Automation and Continuous Test will be supported by Testing Automation and Virtualization. The automation tool chain will be orchestrated via a Continuous Integration Server and the entire process will become a reliable and fast Continuous Delivery process.

About the Author/Tracy Ragan

Ms. Ragan has had extensive experience in the development and implementation of DevOps for large organizations. She began her consulting career in 1989 on Wall Street specializing in build, testing and release management for the distributed platform. It was during her consulting experiences that Ms. Ragan recognized the lack of standardized development procedures on open systems that was common on the mainframe. In the years leading to the creation of OpenMake Software she worked with development teams in implementing a community driven standardized build and deploy process that enabled frequent builds and releases, automated around version control. Her knowledge and experience contributed to the creation of OpenMake Meister, the first commercial Build Automation solution. Ms. Ragan served on the Eclipse Foundation Board of Directors as an Add-in Provider Representative for 5 years. She has been published on numerous occasions and regularly speaks at conferences including CA World where she presented for 15 consecutive years. She holds a BS Degree in Business Administration, Computer Technology from California State University, Pomona.
October 30, 2015
Continuous Deployment is king of the DevOps hill

One of the promises of DevOps is continuous deployment. DevOps is, first and foremost, a matter of shifting the traditional business culture more than the tools and technologies that enable it. Once that culture exists, though, one of the defining aspects of DevOps is automation and continuity.

Earlier this year I wrote about the seeming ubiquity of “continuous” solutions throughout DevOps:

If DevOps is the buzzword du jour for tech, then continuous is the buzzword du jour of DevOps. Everything is continuous. Continuous development and continuous testing lead to continuous deployment and continuous delivery, which requires continuous support. Continuous monitoring produces continuous integration and continuous change. Continuous security results in continuous incident response…or vice versa. To top it off, all of the continuous activities continuously feed each other to drive more continuousness in some sort of DevOps Mobius strip of continuity.

Taking a closer look at all of that continuous continuousness, however, continuous deployment stands out as one of the prevailing goals of effective DevOps. Continuous integration, continuous delivery and the other continuous tools and processes are all in some way related to facilitating the flow of continuous deployment or monitoring and managing it after the fact—which ultimately feeds back to the beginning of the loop and back to continuous deployment.

Chris Riley tried to muddle through the confusion of continuous integration, continuous delivery and continuous deployment. In his explanation of continuous deployment, he stressed the impact of this symbiotic loop that surrounds it: “Continuous deployment has additional costs, as it relies on instrumentation, to ensure that new functionality does not result in bugs and also on infrastructure that allows effortlessly backing out new features when a defect has not been caught by automated tests.”

I defer to Riley on most things, but I don’t agree with his summation that everyone should do continuous integration, some should do continuous delivery, but only the rare few should strive for continuous deployment.

I guess it depends on what perspective you’re considering it from. From the standpoint of ease or simplicity and the likelihood that an organization can execute it effectively I can see where Riley is coming from. From the point of view of the goal of embracing DevOps, though, I would reverse his order and say that everyone should do continuous deployment, most should implement continuous delivery to facilitate it, and some (or maybe most as well) should adopt continuous integration to drive continuous delivery.

The primary objective of DevOps, though, is to deploy apps. Everything else plays a supporting role.

That said, there is a difference between embracing DevOps with the goal of developing and deploying apps more efficiently and effectively, and adopting continuous deployment tools that automatically pass all changes that pass automated testing and are verified as deployable by continuous delivery tools into production.

A blog post from Puppet Labs explains, “There are business cases in which IT must wait for a feature to go live, making continuous deployment impractical. While application feature toggles solve many of those cases, they don’t work in every case. The point is to decide whether continuous deployment is right for your company based on business needs — not on IT limitations.”

There are many acronyms and various interpretations out there for DevOps, so my opinion certainly isn’t gospel on the subject. What do you think? Should continuous deployment be the primary goal and centerpiece of DevOps automation? If not, which of the many “continuous” solutions would you suggest as the central focus?

October 15, 2015
Digital Disrupters – Building a DevOps Leadership Profile
While the advancement of DevOps methodology remains fairly nascent for many organizations, an ever-expanding class of leading practitioners are evolving a wide range of related best practices.

Beyond the well-worn concept of “unicorns” – the scant few organizations that stand as mythical examples of DevOps nirvana – this broader set of early adopters have begun to pull back the curtain on their efforts, lending further detail to the strategies that have bred this current model for success.

Last week, industry analysts Freeform Dynamics published valuable research that highlights the ongoing work of such DevOps innovators, dubbed as “Digital Disrupters” for their accomplishments in pursuing larger digital transformation. The report “Exploiting the Software Advantage” – sponsored by CA Technologies, outlines a critical set of initiatives crucial to navigating the DevOps landscape.

What are the specific hallmarks of those organizations leading the charge? Where are others struggling to catch-up? Many of the Freeform Dynamics survey results, based on 1442 respondents, reaffirm widespread DevOps observations, while other findings serve to shed new light, including:
- Everyone builds software: Perception is, in fact, reality and to deeply engrain digital transformation, Digital Disruptors are increasingly identifying with this mantra. While 95% of leaders have committed to becoming more app-centric, only 51% of mainstream organizations have adopted this persona. Similarly, 84% of Disruptors directly identify as “software companies”, regardless of industry, compared to 40% among the masses.
- Transformation is required: This point may seem obvious, but it certainly bears repeating. A whopping 92% of leaders have actively moved to break down traditional silos between Dev and Ops, with 51% of those organizations having already completed this work. By comparison, only 18% of mainstream organizations claim to have done the same, supported by 59% who at least share this more collaborative DevOps vision.
- Commitment is critical: This may also seem like a no-brainer, but the numbers bear out this requirement. Some 77% of Disruptors have agile development methodologies in place, with 70% having “broadly” implemented DevOps activities. For mainstream organizations, these metrics rate at only 31% and 21%, respectively; leaders are going all-in and appreciating the benefits, and those that follow will need to do the same.
- Experimentation is omnipresent: To garner the benefits of continuous delivery, embracing trial and error is a must. Among Disruptors, 93% recognized the need to “experiment and fail quickly on the road to success”, compared to only 58% of mainstream organizations. Some 95% of leaders have already shifted from fixed product cycles to continuous innovation, while only 56% of the mainstream have made this leap.
- APIs are everywhere: Ubiquitous integration, both in-house and in connecting to third parties, has only increased emphasis on the API. Disruptors are resoundingly focused on API management, with 68% using APIs to integrate third party services into their applications, and 66% to offer external access to systems and data. Mainstream peers compare at only 26% and 24%, respectively, denoting a sizable API management gap.
These are only a handful of the most dramatic and obvious areas where DevOps advocates have clearly re-architected their approach, separating themselves from those organizations that have yet to go all-in; it doesn’t stop there.

As highlighted in the research, other areas of differentiation and dichotomy include cross-functional alignment, emphasis on digital channels, use of digital to drive core business efficiency, and even funding of digital investments.

Beyond these key philosophical milestones that litter the pathway to DevOps success, the survey also highlights the vast array of strategic goals that these organizations have sought to accomplish. The results spotlight the massive range of significant business benefits that practitioners believe DevOps can deliver.

Leading use cases include:
- Improved customer satisfaction (49%)
- Better workforce productivity (44%)
- Improved customer experience (41%)
- Improved competitive differentiation (38%)
- Better agility / speed-to-market (37%)
- Opening up of new markets (35%)
- Creation of new business models (30%)
- Creation of new routes to market (26%)
If one were to take those outcomes out of context, unrelated to the impact of DevOps, it’s fair to say that they read like a generic list of leading business initiatives – further underlining the movement’s overall relevance.

That’s not to say this is all accomplished easily. Report respondents detail hard won victories that required significant time and investment to achieve all this DevOps-driven advancement. After all, the notion of “disruption” doesn’t exactly evoke images of ease or comfort.

Click here to read the report for yourself – there’s a lot more to be considered. Namely, how well your organization measures up in affecting digital transformation in its own right, and where it can take steps to move forward.
October 14, 2015
To Be Continuous: Shipping Velocity and Team Structure

In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk about shipping velocity and it’s affect on team structure.

Edith is CEO and cofounder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

Paul is the CEO and founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

[soundcloud url=”https://api.soundcloud.com/tracks/226489524?secret_token=s-Xkrrb” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

Paul: Okay, so one of the things we wanted to talk about this week was the idea of shipping velocity and how it’s affected by team structure, sort of miscellaneous factors that aren’t part of continuous delivery.

Edith: Yeah, I mean, can you explain a little bit more about what you mean by velocity first?

Paul: Sure, so velocity is a sort of nebulous term that they use a lot in product management and that sort of thing to indicate how quickly, or the sort of through-push with which engineers and product teams are able to put through code.

One thing that isn’t particularly clear, and there’s this amazing blog post by a guy called Phil Calcado who worked at SoundCloud, and he talked about the idea that even though that they had a very high velocity, they weren’t able to ship features very, very quickly, so often it would take 60 days to ship a feature even though no one got blocked.

The idea of shipping quickly, is a combination of velocity and latency, and it depends on your kind of business goals, whether you need to ship correctly or you just need to keep the through-put up.

Edith: Well, do you think part of the issue with SoundCloud might have been that they were trying to do too large features? I mean, that’s somebody’s first reaction when they hear it takes 60 days.

Paul: So in the blog post what he said is first someone would spec out something, and then they’d put onto the next person’s queue, and that would be maybe a designer who’d drop wire frames, and then the wire frames would end up on an engineer’s queue to implement it, and in between someone doing work and someone else doing work, there would be, sometimes, you know, 10 or 11 days of just waiting for it to get to the front of that person’s backlog.

So it wasn’t so much, you know, are you biting off more than you can chew, it’s just the process wasn’t particularly set up to be able to ship things very quickly.

When he got in there, the first thing he did is he said, “Oh, you’re not doing continuous delivery,” or “You aren’t doing continuous deployment,” so he turned on continuous deployment and got that set up, and it was a 66-day process before he finished, and when he got continuous deployment set up it was a 65-day process.

Edith: So all that work to save a single day.

Paul: Right. Well, it was more about, the discussion was about how it wasn’t really about, continuous delivery. It wasn’t really about the time that people were spending on certain things, the vast majority of it was the time that was spent waiting on it.

Edith: Yeah, that takes me back a lot to the whole idea of just in time manufacturing and lean.

The whole idea of lean was to reduce waste, and one area of waste is time.

Like if you have people on a factory line who are waiting for the next part, or you know, stuff queuing up, then you’re being very wasteful.

Paul: I think this is how people, I think that that’s the thing that the backlog is intended to fix, that you’re never waiting for things because there’s always something in the backlog that needs to get done.

But what isn’t considered as part of that is something is sitting in the backlog, for five days, 10 days, two weeks or whatever waiting for someone to address it, then the speed with which you can get a feature out increases dramatically, or decreases dramatically, I guess.

Edith: Well, that’s the whole idea behind the Toyota factory line is that they, the just in time delivery meant that there was always a part right then, not before and not after.

Paul: Right, okay. But right when they’re ready. But you couldn’t say, “I need a car now.” I think it’s not really optimized around the idea of how quickly something goes from one end to the other.

It’s more optimized around how quick, or how much, how little waste that you can have in the system itself, which means, in this case, how much developer, engineer, or designer engineer and project manager time is being spent doing nothing.

And I think you can certainly reduce that time to very, very low without necessarily optimizing the speed that which it takes to assemble a car, which is something that you need, really, to get to market quickly, and that sort of thing.

Edith: Yeah, and it’s a trade off. You know, I went to a talk that the former CTO of Yammer gave, Adam Pisoni, and he talked about how on projects they always had an extra engineer. Like if they thought a project would take four engineers, they would actually put five on it.

Paul: Right, right.

Edith: For precisely that reason, that they never wanted anybody to be sitting around blocked. So they would rather take the cost of an extra engineer, just so there was always somebody to do the next task.

Paul: Right. And so there’s an interesting thing about how teams are structured in that. In the Yammer example, they have teams, and teams do a project, or do a particular feature, and I presume that they’re cross-functional, there’s a couple of designers, couple of engineers, whatever.

A different way to structure it is to have teams that are aligned on a particular, by what they do, and you have the engineers who are sitting at the front of the front-end queue, or who work on the front-end rather than work on a particular project which has components of front-end and back-end.

And you end up with a very different structure, or it’s obviously a very different structure, but you end up with a different ability to ship, depending on how you’re, depending on what you’re trying to do, and what’s the most important thing for your organization, also how your organization is staffed.

So the backlog works very well if you’re short-staffed in any particular area, and so you spend a lot of time prioritizing so that the people, maybe you’re short on design time, so that the design time is spent really, really well, but it doesn’t necessarily allow you to turn around a feature from start to finish and get it out to your customers incredibly quickly.

Edith: Yeah, and that’s a decision that different organizations make. Some organizations say, “We’d rather have 100% utilization, even if it means our features take longer.”

Paul: Right. My feeling is that most organizations don’t actually make that decision consciously.

Edith: It’s implicit.

Paul: Right. Oh. So, what do you mean by that?

Edith: Oh, well so I worked at a lot of big enterprises, and my saying was always,

The lack of a decision sometimes is a decision.

Like, we’re not going hire an extra designer to unblock a lot of people, so therefore our feature’s are always just gonna be gated by that.

Paul: So when you say it’s implicit, you mean that, I guess you kinda mean the same thing as I do, that people aren’t really thinking about the advantages or the disadvantages of the structure of particular teams.

Edith: Yeah, that people are just kinda, they say, “Oh, everybody looks busy, and why does a feature take 65 days?”

Paul: So one of the structures that I really liked, there was a talk at Heavybit, and I think it’s on the Heavybit website by Peter van Hardenberg from Heroku.

Edith: Oh, he’s so great.

Paul: So in the talk he talked about that at Heroku they have product teams that are around specific areas. And the product teams have two engineers and a PM, and maybe a designer and that’s kind of roughly it.

Teams have like two to five people, and they’re focused on a particular area, so they do the prioritization within their areas, they do the support and the ops within their specific features, and then they’re able to trade-off, or they’re very, very close to the problem.

They have then project managers who are working to protect them from other areas, or bleeding into other areas, but also working to solve their particular problems, rather than necessarily one backlog for the entire, maybe the entire platform or the entire web, or something along those lines.

Edith: So basically, little small start-ups within a bigger company.

Paul: Yes, and I think actually that’s how they structured it at the start. I remember them saying that there was a, they would actually have presentations to the board, and that each of the teams was structured, the project manager was like a CEO, and they actually like gave presentations to the board, and I think that didn’t necessarily last very long, but I think that was the initial idea behind it.

Edith: Yeah, we did something very similar at TripIt. We had a, you know, we would have a new project we wanted to do. We were trying to do TripIt offers, and we had a project team just for that, and that’s what they did.

And the idea was that they could focus. I’ve also heard of people who do stuff completely different. Like I talked to a guy who ran an agile shop and every project had to be a week long, and nobody had any ownership whatsoever.

So he set it at, and not even like ownership in the way we would think of it. He said our front-end people were back-end, people were mobile, also.

Thank you for listening, the entire Heavybit podcast can be found here.

October 6, 2015
Re-thinking DevOps tools for Continuous Deployment success
Gartner predicts that 25% of Global 2000 companies will be actively implementing DevOps methodologies in their organizations by 2016. As DevOps goes mainstream, it may be time to re-think some of the implementation techniques that have worked in small companies and in non-production uses. The common assumptions that are often applied to make a continuous integration process work are generally not valid in production real-world datacenters. Common assumptions include:
- Dedicated resources are available for separate dev and test environments
- Pools of virtual machines are sufficient for executing regression and QA tests
- Testing environments mimic production environments
- Tool chaining is sufficient to implement DevOps workflows
Let’s consider each of the above assumptions and how they impact success in real production environments.

Dedicated resources for separate dev and test environments

In any large organization, it is prohibitively expensive to maintain separate and dedicated infrastructure for development and test environments. In our experience, any organization with more than five or so development teams will need to have a better way to manage and share development, test and QA infrastructure. This means that they will not be able to just trigger regression tests directly from build completions. They will need to first ensure that they have access to the resources needed for test execution. We believe that DevOps tools need to also incorporate reservation and scheduling capability for shared and constrained physical and virtual infrastructure.

Pools of virtual machines for sufficient for executing tests

DevOps tools are mainly focused on automating the process of developing, testing and running applications in homogeneous pools of virtual or physical servers. In these environments, it is possible for the DevOps tool to depend on the virtualization environment, PaaS tool, or OS for the server pool (such as CoreOS) to provide execution and deployment. Unfortunately, delegating the scheduling (or lack thereof) to the virtualization or OS tool does not work in large production environments and is certainly not going to provide enough control for regression testing that needs to mimic real world datacenter infrastructure. This can become a severe limitation of DevOps tools. We believe that a more complete DevOps toolset must incorporate workflow orchestration that allows the developers to have control over how applications are executed onto the target virtual environment in each step of the DevOps process.

Testing environments mimic production environments

Global 2000 datacenters are not homogeneous. They are typically hybrid infrastructures where applications are deployed across a combination of traditional, virtual and public cloud computing. In addition, production network topologies are typically complex and require configuration to support applications. Ignoring this complexity in development and testing will result in applications that fail to transition to production successfully. Most continuous integration tools are not capable of controlling deployment into this more complex environment. This explains why most DevOps projects today stop at “continuous integration” and never get all the way to “continuous deployment” into production. DevOps tools need to have the capability of controlling not just the deployment of applications, but the configuration of the hybrid infrastructure on which those applications will run – and this needs to result in test infrastructure configurations that really do mimic production infrastructure.

Tool chaining is sufficient to implement DevOps workflows

The prevalent solution to a continuous deployment process is to chain DevOps tools together end to end from development to deployment. This creates problems because each tool must have a point to point integration with the tools on either side of it in the process flow. By the time an IT shop strings them all together, there is often no complete solution. Eventually they will find that the next tool in the chain has only one (or zero) options because it doesn’t have the specific integrations needed for all of the tools already selected.

More critically though, simply chaining tools together ignores the three key problems described above. A true end-to-end solution should not just chain development, test, QA and deployment tools together, but should also incorporate:
- Reservation and scheduling of resources prior to initiating the next step in the DevOps process;
- Workflow orchestration capability that controls placement of applications onto an appropriate mix of virtual or physical servers; and
- Automation capability for configuring the target infrastructure and network topology before running applications.
We believe that we will see customers augmenting their existing DevOps tools as they move into full continuous deployment, with additional software that can provide each of the above capabilities.

About the Author/Joan Wrabetz

Joan Wrabetz is the CTO for QualiSystems. Most recently, she was theVP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.
October 2, 2015
Native Integration of APM Driving DevOps Maturity

With applications performance monitoring established as one of the fundamental cornerstones of today’s DevOps methodologies, it’s easy to overlook the reality that effective instrumentation of such tools remains extremely difficult. Specifically, the biggest challenge remains creating performance metrics that address both IT and larger business initiatives.

There’s no question that APM is one of the most widely adopted and deeply embedded elements in today’s DevOps world, serving as the critical source of measurement that enables more transparent and agile applications. Yet, despite the fact that performance monitoring solutions, and their respective providers, are considered to be among the most mature on the DevOps landscape, customers still struggle to optimize the systems to meet their specific requirements.

However, as the market continues to advance, specifically via direct integration of widely-adopted APM tools with other strategic applications management systems, such as e-commerce platforms, organizations are finding a faster path to success, and bigger-picture DevOps benefits. One such partnership, announced in late August, saw market leader New Relic expanding integration with e-commerce platform provider Magento, owned by eBay.

Jon Azrielant, director of marketing at online retailer Jewelry.com, a Magento customer, said that his company initially ran into significant obstacles in adapting New Relic’s APM capabilities to its processes, despite the comparative strength of the tools.

“We had a really difficult time implementing New Relic initially, it was a painful process,” Azrielant said.

While the APM tools provided Jewelry.com with the data necessary to generate vital alerts, along with more agile sysadmin and operations monitoring, the more silo-based, fragmented approach that it employed could not empower the retailer to circumvent traditional walls between apps support and service support teams, Azrielant said.

Leveraging the extension of New Relic’s Software Analytics Cloud within Magento, the company has created more advanced instrumentation to inform critical business decisions. Among the most helpful of these measurements are those relating directly to customer experiences, while correlation of such metrics with Jewelry.com’s bottom line has also advanced executive decision-making, he said.

“The critical result of this integration is more complete visibility across applications and business performance, importantly tied to the actual business ramifications,” said Doug McIver, director of product for Magento. “In the past, merchants had to do a lot of difficult integration to use these systems together, so obviously this is a hugely valuable capability for us to be able to offer, especially to customers with highly performant site requirements.”

Moreover, Magento claims that it is growing its own customer base by tying APM more closely to its platform. Likewise, New Relic officials concede that to help advance crucial DevOps workflows, it must partner with such platforms to deliver APM data within the context of business requirements.

“If we’re talking about how to win in world where we’re iterating the end users’ experience on a regular basis, we have to put performance metrics within context of the UI, and the business results; working from the same data sets and getting people on the same page is incredibly important,” said Abner Germanow, senior director of solutions marketing at New Relic.

While pure performance stats are “near and dear” to New Relic for obvious reasons, the solutions provider is also attempting to advance a broader array of information that drives closer collaboration through new “iteration and experimentation”. New Relic is also focused on further developing the specific types of information most critical to key constituents such as e-commerce platform users. The need to create a common language for those DevOps efforts is amplified by the fact that development and operations officials are often unsure which metrics are critical to the business, and vice versa.

“Instrumenting code has always been a challenge, and we’ve worked very hard to ease that pain across a lot of different platforms; there’s always another language where work can be done, and we look to our customers to help lead us into the next phase of instrumentation,” Germanow said. “We’re seeing a lot of pressure around the need for constant change and iteration in industries where it’s truly a matter of survival, such as e-commerce, publishing, digital governance and finance – places where the industry is behaving very differently now than it has over the last 50 years.”

This work, and the subsequent APM integrations offered via platform providers such as Magento, have allowed end user organizations to remain focused on customer experience and sales, the ultimate goal of DevOps, said Jewelry.com’s Azrielant.

“In terms of how this relates to the DevOps revolution, this is the advancement that is bringing everything together and allowing everyone to look at the same numbers, to share common APIs, to streamline reporting and event response, and define everything in uniform terms,” Azrielant said. “Ostensibly, the most basic tenet of DevOps centers around the unification of these disciplines, and you can’t support that without the unification of this information.”

September 29, 2015
To Be Continuous: Evolution of Continuous Delivery

In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar discuss the evolution of continuous delivery. This podcast is brought to you by Heavybit, a 9-month program for developer startups.

Edith is CEO and cofounder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

Paul is the CEO and founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

[soundcloud url=”https://api.soundcloud.com/tracks/224359194?secret_token=s-rlyTD” params=”color=ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

Edith: So Paul, where did continuous delivery come from?

Paul: This is one of those things that evolved over quite a long time but I think that that major catalyst of continuous delivery was the fact that we moved to SasS and that SasS now exists.

Edith: So what was it like before there was continuous delivery? What was it like when you were starting out?

Paul: The first company that I really worked for was a games company and I think that a lot of them are kind of in the same place now. You would release software on a disk that shipped to customers via stores and you had to have a gold master at some point. We were a startup so it was quite advanced.

We had a daily build server and we actually were selling middleware so we shipped software to those companies but those companies would ship at the end of the day, they would ship a disk that went out once and there was no updates at the time.

It was 2003, 2004, something like that. There wasn’t any way to automatically update on customers’ Playstation 2, unlike even games now have a little bit of continuous delivery but back then, that was, you got your chance to ship it and that was it.

Edith: Yeah, I just had a brutal flashback to one of my first summer jobs, which was installing Windows 3.1. This is back when you actually had a set of 35 disks so my job as a summer intern was to go to every computer in the office and just literally sit there. Disk in, disk out, disk in, disk out, and if something went wrong on the install, I had to start over at the beginning.

It was a two hour long process just to get everybody up to 3.1 so in those days, there was no change of a fix pack. I think everybody was going to be on this version until probably they got another sucker summer intern in there.

Paul: So if you imagine people and their, people installing, even modern or relatively recent version of Windows where you have to think about the service pack one or service pack two or what level are you patched to, that was basically the old way of doing software.

The first time that I remember realizing that there was a different way to do it was in a Joel on Software blog. One of the major things that they could do is that there was a bug, they could fix it and in the cloud, it would be fixed.

Edith:

Yeah, I think it’s good though to realize that the reason that there was so much emphasis back then on making sure that a build was correct was because there was such heartache and pain if it was wrong.

Paul: Yeah, it made sense to hold up the release for new features or for something like that if it was promised to a customer because then, there was no alternative. It would be three, or six or twelve months later when they got the next opportunity to go ahead.

I think frankly, building software in the cloud and having the possibility of doing a release at any time you like has actually just completely revolutionized how people make software.

Edith: I completely agree. So to go back to the old days, all the emphasis was on extremely high quality and extremely stable.

Paul: Right, right.

Edith: Because you basically had once chance. You were lobbing this thing out into the world.

Paul: And even things where they had quarterly release cycles or something like that, it was still, it felt a lot like one chance. I worked at Mozilla and I was there on the switch from Firefox 3.0 to Firefox 4.0. Firefox 4.0 took 18 months to commit and there was things that were ready to be released the first week of that release cycle and that had to be held up for all these unrelated releases.

I think there was problems with the JavaScript engine and there was memory bloat and that sort of thing that kept pushing it back. And Firefox, during the process, or during the time that I was there, switched its process and it switched to what they call a release train model where basically, they ship a new version every six weeks and it doesn’t matter what’s shipped or what’s not shipped.

They’ll hold it up for total life, or build breaking, it just doesn’t work at all but basically everything else would make it go ahead. The cadence and the velocity that they’re able to get since that happened has been massive.

Edith: Yeah, so that was the first evolution. It used to be let’s try to cram everything in and then you’d get this vicious cycle where it’s like well let’s just put this one more thing in and then you’d have to add on all the QA time.

Edith: So that was the other reason why releases would take so long is you had so much QA.

Paul: Interesting.

Edith: I talked yesterday to a guy who had worked at Microsoft and he said, you know, quality had to be rock solid for Microsoft going back to my disk example.

Paul: Right, and these are people who have unit testing, automated tests as part of their builds.

Edith: But even beyond that, you have to have a person actually try it and then you have to actually go out and try it on all these different systems.

Paul: On all the different systems, right.

Paul: So one of the things that made it possible for the browsers to do, to move to the release train model is that they had the alpha channel and the canary channel, or whatever it is they called it. In Firefox, it’s the Aurora channel.

That’s the super, actually that’s not even the super beta stuff, they even have a nightly channel. They have a million people using the nightly and that tells them whether things are broken and how they are in fact and in production. I guess it’s kind of the first evolution of feature flags.

Edith: Yeah, I mean let’s talk about that for a second.

Paul: Right, we probably need to, aren’t we going to define things in this one?

Edith: Yes, this is the definitional episode, Paul.

Paul: Right, let’s start with what’s a feature flag because I think we’ve already talked about six things that we haven’t defined but what’s a feature flag?

Edith: A feature flag is basically the ability to hide code but yet have it public. So it used to be that everything that shipped was visible. So a feature flag is just basically an if then that wraps a piece of code and says it might be shipped but nobody can see it until the software team decides that it’s visible.

Paul: Right, and so some examples of that could be in a browser that you change a preference and the preference is use the new version of this or another example might be in, that a feature goes out in Facebook that’s shown to 1% of customers in Northern California who are in this demographic. Yeah, and it’s very liberating.

Edith:

The very liberating thing about a feature flag is that if something’s not working, you don’t have to do a redeploy.

So to go back to the old dark days where you had all these disks out in the world, the feature flag is useless because you’re going to ship all these disks out to people and they’re going to install it. A feature flag is useful when people are consuming stuff as SaaS.

Paul: The way that I think about a feature flag is that it separates the active shipping code from the act of actually turning on that code.

Edith: Yeah, deployment from roll out.

Paul: Right.

Edith: Yeah, so it’s deployed, it’s available, but it’s not visible.

Paul: Right, so now that we know what they are, what were we actually talking about?

Edith: We’re working our way through the evolution of software so we got up to release trains. You should talk a little about how continuous integration fits into that.

Paul: So continuous integration, to define it, it’s basically the act of having an automated build server that runs every day or at some sort of cadence and tests that your software works.

So CircleCI, my company, makes continuous integration as a service and what happens there is whenever you push to GitHub, we’ll automatically check your code out, we’ll build it, we’ll run your tests on it, and we can also deploy it to your customers, which is called continuous deployment or continuous delivery, depending on exactly how you think about the world.

Edith: You make this seem so simple but it’s really revolutionary. I remember the days without this.

Paul: Sure, sure. I remember in my first job, we had a daily build server and so we would find out every day whether or not the code actually compiled, even though we could get it all compiling on our own machines or some modules of it or whatever, compiling on our own machines, there’d be all sorts of build artifacts and dirtiness in the code that meant that this code didn’t actually work when it was ready to be shipped to customers.

Edith: Yeah, or even worse, I remember it used to be a thing when you broke the build, it was very shameful. You would have to sing a song or you’d have to do something embarrassing and that just doesn’t really happen anymore.

Paul: Right, right. My co-founder used to forget to add new files to Git so it constantly wouldn’t work although it was perfectly working on his machine but there’d be a file missing somewhere.

Edith: The classic excuse of, “Oh well it worked on my machine.”

Paul: Exactly, yeah.

Edith: Yeah, I didn’t know about builds my first summer job. At my first summer job, I was programming in Visual Basic.

Paul: Okay.

Edith: I was basically making a war game. It was for a defense contractor and I was trying to justify a space based laser versus an airplane based laser and if you had all these missiles coming over from Russia. Okay, this is a long time ago.

Paul: Yeah, yeah.

Edith: How many could you shoot down? The thing worked perfectly locally and then at the end of the summer, they wanted it off my machine. I was just like, “Oh.”

Paul: Okay, you’d need to put it on a plane or in space or one of those things?

Edith: No, they wanted my actual files off my machine.

Paul: Right, yes, in order to put it in space or on a plane?

Edith: No, it was just a simulation. I wasn’t actually controlling missiles when I was 19. Though that would have been cool.

Paul: I had this absolutely terrible release process at this place that I worked. It was O2, which is a really large phone company in the UK, in Ireland, and I think they got bought by someone recently but whatever. They were launching this feature called i-mode and i-mode is this massive thing in Japan that completely failed to take off anywhere else.

Edith: Are you saying it was literally big in Japan?

Paul: Literally, it was big in Japan. And so they tried to launch this in Ireland. Everybody in the whole world knew that this was going to be a failure, but that was how the process worked anyway. And the release process was they had all this code somewhere that was going to be deployed and attached in servers and when they did that, there’s a manual. I was part of the manual team that manually validated with handsets, clicking through handsets manually, and vouched that this stuff actually worked.

There was no automation. The entire release process had to be repeated dozens and dozens of times as they fixed bugs and they reconfigured. It was, it was just a complete cluster.

Paul:

When I moved to doing software that had any sort of continuous release cycle, it was liberating because you actually know the path from what you’re doing on a day to day basis to it arriving at customers. Everyone, every developer in the place can know what that path is and everyone knows how code acutally gets shipped. You don’t have to keep it in a continuous state of you know, oh maybe one day I’ll ship this.

Edith: Yeah, I mean I think to go back when you had these longer and longer and longer release trains or even pre-release train, people got very relaxed.

Because it just was like oh, I’m just going to sit in the corner and code away and sometime in the distant future, it might ship.

Paul: Right, and then as you’re coming towards the end of the quarter, everyone is hammering things out as quickly as possible and maybe there’s a few things that shipped early in the quarter that never got cleaned up or never got quite ready.

Edith: What do you mean by shipped right now?

Paul: So I remember sometimes where you’d merge something into master because you needed to get it tested but it wasn’t quite production ready. It’s out with the beta testers or something like that, but in order for it to actually ship to hundreds of millions of people, you need to remove whatever hacky thing you put in place.

Edith:

I think one of the things that has driven continuous delivery is just the rise of test-driven development.

Paul: That’s interesting. Yes, I mean, I think it’s obvious. You end up in a situation where when you’re trying to move to continuous delivery, you’re saying I’m afraid and we talked about the fear last week.

In order to get past that fear, you need to have automated testing and I guess in the old days, there was manual release cycles and you can’t do manual release cycles 10, 20 times a day. It’s just not possible.

Edith: You can’t depend on having a fleet, like to go back to the Microsoft example, you can’t rely on having two QA engineers for every developer who will, you know, you just fling out code and they’ll find everything.

Paul: Even with two QA developers, you can’t have two QA developers when you push code every five minutes.

Edith: Yeah, they’ll never keep up.

Paul: Exactly and then they’ll never identify the change that actually caused the bug and so on.

Edith: Yeah, I think the things we’ve talked about so far, SaaS has unlocked continuous delivery, test-driven development has unlocked it. What else do you think is really a foundational thing?

Paul: I guess the other one is kind of this change in customer expectations. Customers, as a fact of some people doing continuous delivery, everyone else has had to do it so once you start to experience it a little bit, and you start to wonder why do other sites go down for six hours on a Sunday night, this doesn’t make any sense. Why do I have to download new versions of this software versus clicking the update version or why do I even have to think about this?

Edith: Yeah.

Paul: I think for the installable software market or the desktop software market, Chrome was that sort of killer app that everyone sort of realized that this has completely changed the game.

Edith: Yeah, it’s completely changed the game.

I think even now asking people to download a plugin seems too onerous. They don’t want to do it. They just want stuff to work.

Paul: It’s almost tied to a conversion funnel. If you expect people to install 31 disks to upgrade to the latest version of Windows, people are just not going to do it. You’re not going to achieve the conversion rate that you’re expecting. There is too many alternatives to your app that are out there.

Thank you for listening, the entire Heavybit podcast can be found here.

September 22, 2015