devopsy.kinsta.cloud

Tag: continuous integration

The continuous continuousness of DevOps

Defining DevOps is getting easier, but it’s still an elusive goal. Part of the challenge is that DevOps is more of a culture than a specific “thing” and how that culture exists is subjective and varies from one organization to the next. DevOps is what you want or need DevOps to be to some extent, but one thing seems to be consistent no matter how you define DevOps—it’s continuous.

If DevOps is the buzzword du jour for tech, then continuous is the buzzword du jour of DevOps. Everything is continuous. Continuous development and continuous testing lead to continuous deployment and continuous delivery, which requires continuous support. Continuous monitoring produces continuous integration and continuous change. Continuous security results in continuous incident response…or vice versa. To top it off, all of the continuous activities continuously feed each other to drive more continuousness in some sort of DevOps Mobius strip of continuity.

What does it all mean? I mean, aside from being a catchy buzzword and making that new DevOps tool or service sound sexy and impressive?

In a nutshell it means there’s no end. You don’t really reach a point where everyone can shut down their PCs and grab a beer to celebrate the completion of the project. The project—whatever it is—is more of a living, evolving thing than a deliverable.

There’s also no beginning—or at least no new beginning. All of the continuous activities cycle back over each other like a Spirograph drawing. Have you ever tried to find the beginning or end of a Spirograph drawing? Good luck.

DevOps is the Alpha and the Omega, the beginning and the end, the first and the last.

When you strip out the buzzwords what it really means is working in a more effective and efficient way. I’ve worked on software development projects the old-fashioned way. No, not Agile—Waterfall. The cascading tasks feed each other in what appears to be a logical progression. It makes sense to define requirements, design the software, implement it, test or verify it, and then kick back and go into maintenance mode…on paper at least.

In the real world new requirements are discovered when you’re ready to implement, and testing results in new design elements. Those new requirements need to be designed and the new design element needs to be implemented and it all needs to be tested and maintained which inevitably creates new requirements, design elements, and so on.

Waterfall was already DevOps—it was just dysfunctional because each “continuous” loop was treated as a unique and separate project. Items that were identified after the design phase were just accumulated on a list to be incorporated into the next phase or iteration. The net result was producing a somewhat stable product that you already knew was flawed, but that you’d get to fixing eventually.

Agile development accelerated some of the feedback loops of Waterfall and resulted in faster refinements to projects, but it still had issues. The primary issue is that developers can’t go faster by themselves. Developers are just one cog in the machine and producing software faster just creates a bottleneck if operations isn’t prepared and the infrastructure isn’t ready to handle it.

Chris Riley explained the challenge of Agile. “However, the Ops department was inherently left behind with deployments piling up faster than they could be released and customers never actually received the value they demanded. This trend ultimately gave rise to DevOps, a new-age version of Agile methodology encompassing the Dev and Ops segment to enable service delivery agility and quality.”

Another way of looking at it is that DevOps is an evolution of Agile. I wrote recently, “DevOps is just a natural extension of Agile. It takes the guiding principles and best practices that Agile brought to developers and applies them on a much broader scale across the organization.”

So now everything is continuous. There is no top of the waterfall that cascades through multiple layers until you reach the bottom of the waterfall. Now there’s a continuous culture of continuity. It may sound confusing when viewed through all of the various buzzwords, but when you step back it makes sense. It makes sense because that’s really the way it worked—or should have worked—all along.

DevOps gets processes and policies out of the way to enable people to get things done more efficiently. For some reason we’ve decided that every element should be labeled continuous—which actually makes it sound a little frenetic or overwhelming. Maybe we should use “efficient” instead. That can be our new buzzword.

April 22, 2015
Continuous Testing System Stability
The effort to speed up continuous testing will be wasted if the underlying system that continuous testing depends on is not stable. In my prior blog
Continuous Test Results Analysis – at the Speed of DevOps, I discussed the importance and practices for speeding up continuous testing results analysis. However, unless the continuous test system that the test results are derived from is rock solid, the results will at best be unreliable and at worst – results may not be generated at all!

One of the first rules of DevOps is to “keep going and don’t stop”. Stopping is the equivalent of pushing the red button on a factory assembly line effectively causing the entire line to shut down. Then, panic ensues because every second that the line is down affects all of production leading to increasing costs not to mention restarting the line is substantial also.

Many techniques are employed to minimize the need to stop testing despite reported product test failures. However if the underlying continuous testing system itself is not stable, then the entire DevOps system including continuous testing will immediately be interrupted until the system recovers. Development team members are looking for results, so the time savings for speeding up the availability and the total work associated with results analysis is huge for every minute of results analysis time saved on each CI/CT cycle. When aggregated over multiple code branches, a few minutes saved per test result can be man-years of time saved over a year!

So what can be done to ensure the continuous test system itself is stable and produces reliable results even in the face of inevitable occasional system failures? After all, there is no such thing as a truly perpetual machine that never ever breaks down. Below are some suggestions in a checklist format that have proven useful in high performance DevOps deployments.
1. Build a fast-response multi-discipline team: There are a wide variety of ways any system of interconnected computers and applications can fail whether they are virtualized, internal, or running on a cloud used remotely from a data center in the cloud. The following list of suggestions for the team will assure fast recovery when the system does experience failure.
  • Assign a continuous testing system architect with responsibility to design system stability SLAs, design high-reliability system architectures and define sufficient team workflows suitable for disaster prevention and recovery consistent with recovery time SLAs.
  • While the team members may be in-or-out sourced, full or part-time, there needs to be clearly responsible first responders and 2nd line escalation team members with definitive workflows that take care of failure events 24/7 for every hardware, software, tools, network and firewall component that the system depends on.
  • Assign product team leads to plan changes needed for new products and services.
  • 24/7 availability system designs are pricey to build, expand, and maintain, so finance and accounting team members need to be part of the team to keep clear, current cost data relevant to the continuous testing system.
  • A senior budget manager approves expenses needed for stability without delay.
2. Anticipate failures – prepare, prepare, prepare!: Don’t wait for a failure before testing system recovery procedures! Documented disaster prevention and recovery plans or high-availability tools won’t save the continuous test environment if the team is not ready to react during a failure event.
  • Run failure simulations for each defined failure scenario at a frequency that matches the expected probability of each failure.
  • Cross-train team members on system recovery best practices so the skills will be ready when most needed – ensure no one person has essential unique recovery skills.
3. Choose test tools designed with high stability system administration capabilities: Look for the following features designed into in test tools to ensure the continuous test environment can be configured and managed for high reliability.
  • Tools are implemented as autonomous services, controlled through RESTful APIs, that can be separately invoked and managed
  • Test system service monitoring is built-in including up-time, wait-time and usage metrics for each use case of each test system service.
  • Diagnostic capabilities are built-in to each service. When a failure occurs the failure log includes helpful descriptions including causal information.
4. Engineer the system for stability: Typical continuous testing systems have many “moving parts” and many of them have to work in concert for the system to perform properly. It is common engineering knowledge that the reliability of a system is determined by the reliability and configuration of its parts. Here are some suggestions to improve the overall system reliability.
  • Choose powerful servers and virtual machines (VMs) with ample processor speed, fast I/O, large and fast memory and data storage. This is important to minimize the chance of system failures due to possible overloading of these resources.
  • Deploy multiple servers and VMs for each system component with enough redundancy so that no one component failure will interrupt the entire continuous testing system.
5. Configure the test workflow to avoid false negatives: There is no point in reporting product test failures to users if the reason for the failure was an underlying continuous testing system failure instead of a failure of the product itself. Such “false negatives” are a big waste of time. Here are some suggestions to reduce the chance of false-negative test reports.
  • Assign an application to continuously verify, in-real time, that all the system components and interconnections are operating and report these results to the continuous testing report aggregation tool. These results are called CT system environment test results.
  • Use the CT system environment test results within the continuous test results analysis tool to determine if any test failure coincides with an environment failure. Mark each verdict “inconclusive” instead of “fail” if it does and include a description of the environment failure in the verdict log.
6. Maintain the system to assure long-term stability: Every system needs maintenance at some point. Hardware components wear out and software components become obsolete. Each time a system component is replaced or upgraded to new versions there is a risk of instability. Here are some recommendations to reduce the risk of introducing instabilities to the continuous testing system when changes are implemented.
  • Create a maintenance environment that mirrors the test and production environments.
  • Define a test suite that will exercise all system components and topologies.
  • Test changes in the maintenance environment before deployment.
  • Occasionally run failure/recovery simulations in each production environment.
The above is a partial list of suggestions for continuous test system stability that have been proven to yield good results for DevOps. At Spirent we think testing has a bright future in DevOps. You can read more about our views at Spirent.com/solutions/devops

What do you think of these suggestions and do you have others that should be mentioned?
April 1, 2015
Continuous Testing – Accelerated !
Except for cases where test duration is the point of the test, as indicated in my prior blog “Isn’t Continuous Testing a Longevity Test Anti-pattern?”, faster continuous testing cycles are preferred because failing early and often is a DevOps goal. There are multiple reasons for this, but it should be evident that faster tests reduce the wait time to get results and since there are typically many development team members looking for results the developer time savings multiplier effect is huge for every minute of test time saved on each CI/CT cycle. When aggregated over multiple code branches a few minutes saved per test can be man-years of time saved over a year!

So what can be done to accelerate continuous testing? Frankly a thorough answer to this topic could fill a book. Below are some suggestions in a checklist format intended to be useful for DevOps .
1. Organize the team for speed: When team members and the organization are explicitly selected for CT speed the entire DevOps process will be accelerated.
  • A test architect familiar with the principles of best practices test environments and continuous test design is critical to design and tune continuous testing for speed. The architect establishes the CT strategy and speed goal relative to the coverage goal.
  • A test analysis team reviews test results so no time is wasted on results analysis.
  • Developers and test members are trained on best practices test design and their test scripts are peer-reviewed to ensure they comply with the strategy.
2. Select powerful test resources: Powerful, dedicated, scalable resources will accelerate all test processes and shorten the time to results.
  • Select the fastest processors, low latency storage and I/O to initialize and execute both the test systems and system-under-test entities quickly.
  • Dedicate resources that have longer initialization times so they can be pre-initialized.
  • Virtualize resources that can be scaled horizontally without sacrificing speed.
  • If affordable, arrange enough resources to run ALL tests in parallel.
3. Choose test tools that are designed for fast test execution: Look for the following features when you are evaluating test tools. Not only will this save you a lot of headaches in the long term, it will give you the flexibility to design tests that integrate with other DevOps tools.
  • Restful APIs that do not require wait states between commands and responses
  • Priority queuing of test cases
  • Creation of test topologies that can be pre-loaded
  • Ability to spawn out tests in parallel to run on any number of machines
  • The ability to park a test that needs to wait for an event and return to it after proceeding with subsequent tests
  • Caching features that enable initialization and results data to be processed for one test while the tool proceeds to execute other tests
  • Multiple devices, session types and results data processing utilities implemented as an embedded part of the tool and not requiring slow speed scripting
  • The ability to render commonly used test utility functions in a fast execution language.
4. Organize test schedules and test suites to fail early: The sequence and approach for scheduling of test cases greatly affects the speed of testing. If possible the scheduler should launch all tests in parallel as soon as the build is ready. Otherwise priority schemes are needed.
  • Organize the test scheduler and test suites to avoid any unnecessary tests.
  • Cache initialization and topologies data and resources ahead of the next test execution.
  • Check the test environment before running tests, abort if invalid conditions are detected.
  • Organize tests so the most important ones are run first.
  • Set a failure escape threshold and abort the test if exceeded.
  • Eliminate dependencies between tests so that each one can run standalone or in parallel. Queue the next test while a test is running.
  • Use smart gray box testing methods to limit test selection to the most important tests according to the software change and skip tests that are low risk.
  • Run longer tests in parallel with shorter ones.
5.Design test cases to fail early: The test cases themselves need to be designed with the strategy to get test results as fast as possible.
• Design each test case with a single unique test purpose and make it self-contained so it may be scheduled to run standalone or in parallel with any other test case.
• If a test case has multiple verdicts arrange the highest priority verdicts first and close the test at the first failure.
• Prune or refactors tests that have a low failure history.
• Shorten dead-script timers, and make any wait timers conditional so they can expire as soon as the condition is met.
• Reduce the number of test iterations to fit a time budget.
• Reduce communication wait times by using the fastest protocols. Optimize the most frequently used utilities.
1. Arrange test results reduce to minimize analysis wait times: Arrange the test results to be consumable as soon as the tests are run so the system does not bottleneck.
  • Report test results as the tests are run and accumulate the results across parallel test targets.
  • Establish thresholds for aggregated test results and abort the tests and release test resources to be used by other tests if the thresholds are exceeded for any run.
  • Keep track of test timings and adjust the test schedules according the average and peak run times for each test case.
  • Separate test result report processes to persistent resources so no test process is waiting to finish a report and the test process can proceed to the next test in parallel with test results.
  • Correlate the test results to the changes and use trends analysis to help speed up diagnostics and as an input to test selection such that the most failing test cases are given higher weight in future test runs.
The above is a partial list of suggestions for accelerating continuous testing that have been proven to yield good results for accelerating continuous testing and DevOps.

At Spirent we think testing has a bright future in DevOps. You can read more about our views at Spirent.com/solutions/devops

What do you think of these suggestions and do you have others that should be mentioned?
March 10, 2015
“I want to do Continuous Deployment”

“We want to do continuous deployment” – i’ve heard this many times, but have yet to find a scenario where it actually fit. Why? Because you are confused. And rightly so. How many two letter acronyms can be thrown at you in one movement. DevOps takes the confusion further and has two of the acronyms identical, but meaning very different things. The CDs, which are not the round shiny discs in this case. This straight forward post hopefully helps clear up some of the confusion.

(more…)

March 3, 2015
CI and CD Across the Enterprise with Jenkins – CloudBees
We are happy to announce the next devopsy.kinsta.cloud Webinar, in partnership with CloudBees :

CI and CD Across the Enterprise with Jenkins

Delivering value to the business faster thanks to Continuous Delivery and DevOps is the new mantra of IT organizations. In this webinar, CloudBees will discuss how Jenkins, the most popular open source Continuous Integration tool, allows DevOps teams to implement Continuous Delivery.

You will learn how to:
- Orchestrate Continuous Delivery pipelines with the new workflow feature,
- Scale Jenkins horizontally in your organization using Jenkins Operations Center by CloudBees,
- Implement end to end traceability with Jenkins and Puppet and Chef.
Date: November 19th

Time: 1:00pm eastern, 10:00am pacific
```
REGISTRATION LINK: (Limited registration, so don't wait!) 
```
About the Speaker:

Cyrille Le Clerc is Director of Product Management at CloudBees with more than 15 years experience in Java. Prior to CloudBees, Cyrille was Chief Technology Officer at Xebia. As an early adopter of the “you build it you run it” model, he put it in place for a number of high volume web sites. He naturally embraced DevOps culture. He is also very active in the Java community, creator of the open source project “embedded jmxtrans” and a speaker at various conferences.
November 3, 2014
Why Continuous Integration Doesn’t Work

Continuous integration is easy. Download Jenkins, install, create a job, click the button, and get a nice email saying that your build is broken (I assume your build is automated). Then, fix broken tests (I assume you have tests), and get a much better looking email saying that your build is clean. Then, tweet about it, claiming that your team is using continuous integration.

I’ve seen it multiple times in multiple projects. The start is always bright and smooth. Problems start later when the build gets broken again and we simply don’t have time to fix it. We’re simply busy doing something else and a few broken unit tests shouldn’t be a distraction for us. After all, we all know that unit testing is not for a team working with deadlines, right?

Wrong. Continuous integration can and must work. The latest build should always be clean. Always.

What is Continuous Integration?

Nowadays, software development is done in teams. We develop in feature branches and isolate changes while they are in development. Then, we merge branches into master (Git usage is assumed). After every merge, we test the entire product, executing all available unit and integration tests. This is what is called continuous integration, abbreviated as CI.

Sometimes, some tests fail. When this happens, we say that our “build is broken”. Such a failure is a positive side effect of quality control because it raises a red flag immediately after an error gets into master. It is a well-known practice, when fixing that error becomes a top priority for its author and the entire team. The error should be fixed right after a red flag is raised by the continuous integration server.

# Why Continuous Integration Doesn’t Work?

CI is great, but the bigger the team (and the code base), the more often builds get broken. And, the longer it takes to fix them. I’ve seen many examples where a hard working team starts to ignore red flags, raised by Jenkins, after a few weeks or trying to keep up.

The team simply becomes incapable of fixing all errors in time. Mostly because the business has other priorities. Product owners do not understand the importance of a “clean build” and technical leaders can’t buy time for fixing unit tests. Moreover, the code that broke them was already in master and, in most cases, has been already deployed to production and delivered to end-users. What’s the urgency of fixing some tests if business value was already delivered?

In the end, most development teams don’t take continuous integration alerts seriously. Jenkins or Travis are just fancy tools for them that play no role in the entire development and delivery pipeline. No matter what continuous integration server says, we still deliver new features to our end-users. We’ll fix our build later. And it’s only logical.

# What Is a Solution?

The solution I propose is simple — prohibit anyone from merging anything into master and create scripts that anyone can call. The script will merge, test, and commit. The script will not make any exceptions. If any branch is breaking at even one unit test, the entire branch will be rejected.

In other words, we should raise that red flag before the code gets into master. We should put the blame for broken tests on the shoulders of its author.

Say, I’m developing a feature in my own branch. I finished the development and broke a few tests, accidentally. It happens, we all make mistakes. I can’t merge my changes into master. Git simply rejects my push, because I don’t have the appropriate permissions. All I can do is call a magic script, asking it to merge my branch. The script will try to merge, but before pushing into master, it will run all tests. And if any of them break, my branch will be rejected. My changes won’t be merged. Now it’s my responsibility — to fix them and call the script again.

In the beginning, this approach slows down the development, because everybody has to start writing cleaner code. At the end, though, this method pays off big time.

In my projects I automate this merging operation using www.rultor.com, a free hosted DevOps service. Its usage is explained in this article: Rultor.com, a Merging Bot. I would be glad to answer your questions, just shoot me an email.

September 26, 2014
Waterfall and Agile, the effect of automation and continuous integration
Agile’s A Silly Idea?

I was a young developer working at a large enterprise when I first heard about some of the principles that later became part of the Agile process back in the mid- to late-90s. I was skeptical about what I heard, to say the least. After all, what we today call waterfall had been the norm for a long time; software got built and then made stable in the end. Yes, it required a lot of wailing and gnashing of teeth, but wasn’t that the way it was supposed to be?

The idea of pair programming at first seemed like a joke to me when I first heard it: talk about a waste of time and resources, and the perfect way to allow a mediocre developer to skate by and let someone else do all the hard work. Just the perfect thing for big companies: let the dead weight glom on to the ones who were productive.

Even the names, titles like Scrum and Extreme Programming sounded like a joke. Ooh, you’re an Extreme Programmer… let me kiss your feet! Ah, the good ole’ days as a young developer, with a huge ego, showing everyone else how to do it. But I digress. Anyway, I was of course skeptical, having the first discussions around Agile handed down to me from on high, from some nameless technology group that couldn’t possibly understand our world, or understand that we were building high quality software saving our company millions of dollars a month in automation.

I was never put into the XP training programs that were being instituted: I ended up leaving that company before I was exposed to it. So, I went from that job to a start up product company, StillSecure, where we also fully embraced waterfall, but a less documentation-heavy version of it, and we didn’t have business analysts. And, rather than 18 month schedules, we were working on 3-4 month schedules. It was waterfall-lite, and it did work a lot better (even if to meet our deadlines, we had to cut scope or take a hit on quality). We came a lot closer to our goal on the 3-4 month schedules than we ever did on an 18 month one.

Sometimes Agile Just Doesn’t Work

At one point, though, we did attempt some 12-18 month projects, and failed miserably: the complexity got the best of us, and it was too easy to allow scope creep to kill us. In the end, because of the complexity of test automation on our product, we got to where we had to settle into a 6-7 month release schedule: we had to build enough into a release to make it worthwhile for all the testing that had to go into it. We did automate some testing, but all the really time-consuming work involved network equipment, such as switches, VPN concentrators, and routers in the lab, and testing different scenarios with different network configurations. Something that even now cannot be fully automated, or at least not at a reasonable cost.

Agile without Automation is Decidedly Not Agile

Along the way, we started building a new product, a SaaS-based firewall management system focused on the cloud. Since we were starting with a clean slate and a very small team, we decided to try to be more Agile-like. We moved to a two-week iteration model, but released about every six weeks to our customers. In hindsight, two things prevented us from realizing the best benefits of Agile:
1. We were completely focused on building new features, and invested no time in test automation.
2. We had a really strong QA lead that was very effective, and quick enough to keep the developers on their toes fixing bugs for a week or two following each iteration.
Okay, so #2 was more an excuse, not really something that prevented us. However, when you’ve got someone like that, it makes it difficult to say: we’re going to slow down and build test automation so that we can build features faster. However, that’s a short-sighted view: sure, you can get away with it early on, but complexity builds and builds to the point where a single person can’t keep up with the testing demands, and then you’ve got to make the hard choice: hire another QA person, or go back and automate as much testing as possible. You take the hit one way or another: but building test automation as you go lets you establish a credible cadence and ensures that you can survive things like a code refactor without investing a lot of time to prove out that you didn’t break anything. That means a lot as your code base gets large: it’s the difference between having to build your product on an increasingly shaky foundation, or allowing you to quickly and cheaply improve your foundation as you go.

I mentioned in my last post my frustration with being expected to predict the future months or even years in advance, and being held accountable for my predictions despite any number of factors changing compared to the original plan. Using shorter iterations did indeed allow us to better predict what we would get done in any given time period, but my management still expected that I would meet a long term roadmap, on a set schedule. That part didn’t fit well with Agile, it was still waterfall-thinking with an Agile wrapper. And, we failed miserably when tracking against those longer-term expectations. In the end, that was the primary reason that I could get behind Agile methodologies: the understanding that methodologies which require and expect a software project to unfold perfectly, without changes over the course of months or even years, is a methodology built to fail.

To compensate for this fact, competent waterfall project managers create a strict change management process to track all the reasons why a software project plan doesn’t come to fruition as everyone expected up front. This prevents surprises for management, and makes sure that everyone knows which decisions were made, and why. Without that list of excuses, any long term project would be considered an abject failure: even small projects experience relatively large numbers of change requests during the course of a long development cycle, and more often than not, in my experience, those changes increase the effort required to complete the project, because they often account for complexity that was not known at the time the system was designed.

Agile admits that things change over time, sometimes drastically, and builds that into the process. It’s easier on the team, the managers, and the leadership of the company: no one gets a false date to commit to a customer, and yet the team can continually make measurable, reliable progress toward a long term goal. And, because you change less between each test cycle (with continuous integration, you run tests each night, or prior to each build), you don’t end up with a hot mess of code wherein you have to spend weeks or months just to get to stability before you can fix all the other new bugs you just introduced.

But is Agile actually more productive?

The following chart shows an estimate of the ongoing code stability during a typical 6 month waterfall project with no continuous integration versus an Agile project. The dip from weeks 2-7 are the coding phase of a waterfall project, where the code is “on the operating table” and completely unstable, having no integration testing done on it. On the Agile side, the code may fall somewhat in terms of overall stability, but continuous integration testing helps prevent massive drops in stability that then requires massive effort to recover from.

This graph is entirely anecdotal, based on my experiences with Agile and Waterfall. What I’ve seen is that over the course of a Waterfall project with no continuous integration, the stability of the code essentially hits zero (the system does not work at all, and may not even compile cleanly) very quickly without continuous integration, and only slowly returns to a high level of stability as system integration testing proceeds. In a project that leverages continuous integration, it’s impossible for this deep dip in stability to occur, because a continuous integration environment would break with this kind of instability. However, some dips do occur, as a result of missing test cases that cannot be caught by the continuous integration environment. Even with the best efforts of developers, it’s impossible to catch every potential case, so stability drops slightly from the maximum in this environment, and fluctuates. This graph changes completely for a waterfall project that leverages continuous integration: it ends up looking more like Agile as far as code stability goes. Does that then provide some evidence, at least, that Agile methodologies are valuable only in so far as how much automation of the testing process they do? That leaves us with two controversial open questions:
1. Which methodology ends up as the more productive one, generating more features for customers per unit time?
2. And, which methodology provides better time to market?
Those aren’t the same question: time to market provides you with something to put in front of customers as early as possible. Generating more features in a given time lets you build more functionality in less time overall: helping you keep up with or pass competitors. Some might argue that waterfall is best on #1, and Agile best on #2. I’ve seen some waterfall teams build deep functionality very quickly, at very high quality.

It’s the Automation, Not the Process

Whichever meets your goals better, one thing is certain: automating things that can be automated is always a good idea. In fact, that mantra has built the entire industrial age. In my opinion, waterfall methodologies can take advantage of continuous integration and test driven development, and yet still retain its very thoughtful, well-documented, very architecture-focused view of the world. The transition to DevOps, whether from an Agile or a waterfall methodology, is pretty easy from there: invest in automation up front, and you’ll go faster over the long term. Next time, I’ll talk about my experiences with DevOps at JumpCloud, and what it enables us to do that we couldn’t do otherwise, and the implications that has for reliability and security.
May 14, 2014
DevOps Lessons Learned – The “DevOps Engineer”

Back in 2008 I was part of a startup that had success and sold to large company three years later. During the startup years, there was three of us in IT plus an offshore team that did our web development. The three core resources owned everything from the API down while the offshore team owned the front end. Since we were small, we each owned our own part of the architecture and managed the deployments and operations of the system with few issues. Once we were purchased, our world changed drastically.

Our team quickly ramped up to over 20 people while my original two architects soon left for greener pastures. Now we had a lot more people involved in the process and none were domain experts. Things quickly started spiraling out of control as server sprawl started creeping in and deployments were becoming increasing challenging and error prone. We hired our first “DevOps” person to help us reign all of this back in. Then things got really ugly. For those who have followed my DevOps writings and my rant called “No you are not a DevOps Engineer“, what I am about to describe is the basis for my position on DevOps.

DevOps as a Role: Fail Pattern

We never intended to create a DevOps silo or position. We just wanted someone with expertise with cloud and automation since everyone else on the team was a developer and many of them were new to both the cloud and the platform. Unintentionally, this person became labelled as DevOps and he was soon joined with three other people which became an official DevOps team. We now were the proud owner of a new silo which became the new bottleneck.

The DevOps team had their hands full to regain control over the environment. Permissions had to be pulled back to prevent server sprawl from continuing. Development teams now had to request environments from the DevOps team. When we were a startup, this was easy and manageable. As the team got larger and more concurrent projects were being run, this became challenging. The DevOps guys were so busy keeping up with the demand that they were not always in tune with the goals and the deliverables of the different projects. The app dev teams were so focused on “agile” that they were not including the DevOps guys early enough in the projects.

The DevOps team became drastically overworked. They were monitoring and operating the systems, they were being pulled from their deliverables to provision environments, and they were trying to put controls around a platform that experienced sudden growth from a 3 person team of founders to a 20+ person team of new people. To make matters worse, we knew what we wanted to do to fix these problems but there were not enough hours in the day to get there. We had mapped out an entire continuous integration and delivery strategy, but we just could not get enough time to focus on it.

DevOps started to get a bad rap from the app dev team because there was often delays in getting environments provisioned in time. We experienced environmental issues between dev, QA, and stage because of the lack of timely communication of the environment requirements from the app dev team and because the different app dev teams were not being consistent (e.g. different version of Python and different libraries).

All of this led to frustration from the business, especially from the business people who came from the startup and was used to frequent deployments with high quality.

Lesson Learned

Eventually the team turned this around. Although they still have an official DevOps team, the team works much closer with the app dev teams. A high level of standardization is now in place for the environments. The teams share operational requirements early in the project life-cycle and developers assist in the environment creation. The lesson learned here is that creating a DevOps role or organization is not the correct approach. Creating a new silo just creates new bottlenecks. DevOps is not a role, it is a way of getting quality, reliable software to market faster. The sooner an organization realizes this the sooner they can get to a successful state.

March 27, 2014