Category: Blogs

Agile’s A Silly Idea?

I was a young developer working at a large enterprise when I first heard about some of the principles that later became part of the Agile process back in the mid- to late-90s.  I was skeptical about what I heard, to say the least. After all, what we today call waterfall had been the norm for a long time; software got built and then made stable in the end. Yes, it required a lot of wailing and gnashing of teeth, but wasn’t that the way it was supposed to be?

The idea of pair programming at first seemed like a joke to me when I first heard it: talk about a waste of time and resources, and the perfect way to allow a mediocre developer to skate by and let someone else do all the hard work. Just the perfect thing for big companies: let the dead weight glom on to the ones who were productive.

Even the names, titles like Scrum and Extreme Programming sounded like a joke. Ooh, you’re an Extreme Programmer… let me kiss your feet! Ah, the good ole’ days as a young developer, with a huge ego, showing everyone else how to do it. But I digress. Anyway, I was of course skeptical, having the first discussions around Agile handed down to me from on high, from some nameless technology group that couldn’t possibly understand our world, or understand that we were building high quality software saving our company millions of dollars a month in automation.

I was never put into the XP training programs that were being instituted: I ended up leaving that company before I was exposed to it. So, I went from that job to a start up product company, StillSecure, where we also fully embraced waterfall, but a less documentation-heavy version of it, and we didn’t have business analysts. And, rather than 18 month schedules, we were working on 3-4 month schedules. It was waterfall-lite, and it did work a lot better (even if to meet our deadlines, we had to cut scope or take a hit on quality). We came a lot closer to our goal on the 3-4 month schedules than we ever did on an 18 month one.

Sometimes Agile Just Doesn’t Work

At one point, though, we did attempt some 12-18 month projects, and failed miserably: the complexity got the best of us, and it was too easy to allow scope creep to kill us. In the end, because of the complexity of test automation on our product, we got to where we had to settle into a 6-7 month release schedule: we had to build enough into a release to make it worthwhile for all the testing that had to go into it. We did automate some testing, but all the really time-consuming work involved network equipment, such as switches, VPN concentrators, and routers in the lab, and testing different scenarios with different network configurations. Something that even now cannot be fully automated, or at least not at a reasonable cost.

Agile without Automation is Decidedly Not Agile

Along the way, we started building a new product, a SaaS-based firewall management system focused on the cloud. Since we were starting with a clean slate and a very small team, we decided to try to be more Agile-like. We moved to a two-week iteration model, but released about every six weeks to our customers. In hindsight, two things prevented us from realizing the best benefits of Agile:

1. We were completely focused on building new features, and invested no time in test automation.
2. We had a really strong QA lead that was very effective, and quick enough to keep the developers on their toes fixing bugs for a week or two following each iteration.

Okay, so #2 was more an excuse, not really something that prevented us. However, when you’ve got someone like that, it makes it difficult to say: we’re going to slow down and build test automation so that we can build features faster. However, that’s a short-sighted view: sure, you can get away with it early on, but complexity builds and builds to the point where a single person can’t keep up with the testing demands, and then you’ve got to make the hard choice: hire another QA person, or go back and automate as much testing as possible. You take the hit one way or another: but building test automation as you go lets you establish a credible cadence and ensures that you can survive things like a code refactor without investing a lot of time to prove out that you didn’t break anything. That means a lot as your code base gets large: it’s the difference between having to build your product on an increasingly shaky foundation, or allowing you to quickly and cheaply improve your foundation as you go.

I mentioned in my last post my frustration with being expected to predict the future months or even years in advance, and being held accountable for my predictions despite any number of factors changing compared to the original plan. Using shorter iterations did indeed allow us to better predict what we would get done in any given time period, but my management still expected that I would meet a long term roadmap, on a set schedule. That part didn’t fit well with Agile, it was still waterfall-thinking with an Agile wrapper. And, we failed miserably when tracking against those longer-term expectations. In the end, that was the primary reason that I could get behind Agile methodologies: the understanding that methodologies which require and expect a software project to unfold perfectly, without changes over the course of months or even years, is a methodology built to fail.

To compensate for this fact, competent waterfall project managers create a strict change management process to track all the reasons why a software project plan doesn’t come to fruition as everyone expected up front. This prevents surprises for management, and makes sure that everyone knows which decisions were made, and why. Without that list of excuses, any long term project would be considered an abject failure: even small projects experience relatively large numbers of change requests during the course of a long development cycle, and more often than not, in my experience, those changes increase the effort required to complete the project, because they often account for complexity that was not known at the time the system was designed.

Agile admits that things change over time, sometimes drastically, and builds that into the process. It’s easier on the team, the managers, and the leadership of the company: no one gets a false date to commit to a customer, and yet the team can continually make measurable, reliable progress toward a long term goal. And, because you change less between each test cycle (with continuous integration, you run tests each night, or prior to each build), you don’t end up with a hot mess of code wherein you have to spend weeks or months just to get to stability before you can fix all the other new bugs you just introduced.

But is Agile actually more productive?

The following chart shows an estimate of the ongoing code stability during a typical 6 month waterfall project with no continuous integration versus an Agile project. The dip from weeks 2-7 are the coding phase of a waterfall project, where the code is “on the operating table” and completely unstable, having no integration testing done on it. On the Agile side, the code may fall somewhat in terms of overall stability, but continuous integration testing helps prevent massive drops in stability that then requires massive effort to recover from.

This graph is entirely anecdotal, based on my experiences with Agile and Waterfall. What I’ve seen is that over the course of a Waterfall project with no continuous integration, the stability of the code essentially hits zero (the system does not work at all, and may not even compile cleanly) very quickly without continuous integration, and only slowly returns to a high level of stability as system integration testing proceeds. In a project that leverages continuous integration, it’s impossible for this deep dip in stability to occur, because a continuous integration environment would break with this kind of instability. However, some dips do occur, as a result of missing test cases that cannot be caught by the continuous integration environment. Even with the best efforts of developers, it’s impossible to catch every potential case, so stability drops slightly from the maximum in this environment, and fluctuates. This graph changes completely for a waterfall project that leverages continuous integration: it ends up looking more like Agile as far as code stability goes. Does that then provide some evidence, at least, that Agile methodologies are valuable only in so far as how much automation of the testing process they do? That leaves us with two controversial open questions:

1. Which methodology ends up as the more productive one, generating more features for customers per unit time?
2. And, which methodology provides better time to market?

Those aren’t the same question: time to market provides you with something to put in front of customers as early as possible. Generating more features in a given time lets you build more functionality in less time overall: helping you keep up with or pass competitors. Some might argue that waterfall is best on #1, and Agile best on #2. I’ve seen some waterfall teams build deep functionality very quickly, at very high quality.

It’s the Automation, Not the Process

Whichever meets your goals better, one thing is certain: automating things that can be automated is always a good idea. In fact, that mantra has built the entire industrial age. In my opinion, waterfall methodologies can take advantage of continuous integration and test driven development, and yet still retain its very thoughtful, well-documented, very architecture-focused view of the world. The transition to DevOps, whether from an Agile or a waterfall methodology, is pretty easy from there: invest in automation up front, and you’ll go faster over the long term. Next time, I’ll talk about my experiences with DevOps at JumpCloud, and what it enables us to do that we couldn’t do otherwise, and the implications that has for reliability and security.