devopsy.kinsta.cloud

Category: Blogs

The ROI of Enterprise DevOps

Ever since the 1996 film Jerry Maguire, I can never think about any technology development without hearing Cuba Gooding Jr as professional footballer Rod Tidwell screaming out, “Show me the money!” So I was not surprised when the CIO of a large enterprise recently asked me the enterprise exec version of that famous line:

Q. Where is the ROI in Enterprise DevOps?

Fortunately there is plenty of evidence to show enterprise executives exactly where to look when trying to determine the ROI of DevOps.

“Show me the money!”

Here on devopsy.kinsta.cloud, Rajat Bhargava has posted on talking to your CFO about DevOps, outlining four key areas of financial impact: generating cash, reducing expenses, enabling growth, long-term stability. I also had some ideas about financial aspects of getting executive support for DevOps in a great article by Erica Chickowski. And site editor Alan Shimel has posted some excellent insight on vendor-sponsored research from IDC on the cost-benefit of downtime avoidance.

“Show me the money!”

Not enough? Well, we also have several years of research on ROI factors associated with DevOps adoption in both small businesses and large enterprises. The company I work for, CA Technologies, has commissioned independent third party research into enterprise DevOps adoption since 2013; Puppetlabs has run similar research [PDF] with good data on all sized organizations, but especially mid-sized businesses.

You can see in the chart below the areas where benefit accrues through the adoption of DevOps (taken from CA Technologies/Vanson Bourne research released in November 2014):

Still need more? No problem! There is also good data on how much of a return you can expect from DevOps. Again, the following chart is taken from the same CA Technologies/Vanson Bourne research:

Figure 2 Respondents report seeing double-digit improvements in key technology and business areas. Total varies by benefit. Figure 1 & 2 Source: Vanson Bourne/CA Technologies 2014

“Show me the money!”

Okay, I get it – you still want more, and you want a straight answer – a single, simple heuristic to work with. In talking with many enterprise CxOs, faster application delivery is a pressing goal, so just this one factor can act as a key ROI driver. At the Continuous Delivery (CoDe) conference in Copenhagen in October, Jan Bosch (Professor of Software Engineering at Chalmers University of Technology) showed very elegantly how faster delivery acts to reduce (i.e. shorten) the ROI period.

His theory is based on the premise that for many (most? all?) businesses, time to revenue is dictated substantially by product release cycles. Maximum revenue accrues around product release, then tails off before spiking again for the next new product release. For example, a smartphone manufacturer will get revenue in 6-month peak and trough cycle as they release a new device twice a year; a software vendor will get a revenue peak with a new game release, which tails off until new DLC is released or a new game hits the stores.

Because DevOps can accelerate business cycles, Jan theorized, business can bring in revenue faster, which in turn makes a DevOps-based business more profitable over the mid- to long-term. This is especially critical in the modern application economy where almost every business is a software business.

So, for example, if DevOps allows a business to release twice as fast, they can effectively double their annual revenue, as seen below.

Figure 3 Proportional revenue acceleration from doubling product release frequency with DevOps

The impact is exaggerated for this illustration, but based on actual data (see above), a 20% increase in annual revenue would be on par with enterprise averages.

“Show me the … costs!?”

Of course, any discussion of ROI must realistically reflect costs too. Adopting DevOps in a large enterprise is not free – it requires investment in people, process, and technology. On the people side, even just a small cultural change often requires new activities and other behavioral shaping. Process change requires training and often impacts short-term revenues. Technology investment includes procurement of automation and measurement solutions – key components of a CAMS (Culture, Automation, Measurement, and Sharing) approach to DevOps, especially in a larger enterprise.

Many of the investment factors can be seen in the following chart, again using CA Technologies/Vanson Bourne research:

Figure 4 Which of the following is your organization likely to invest in over the next year as part of your implementation of a DevOps methodology? Total = 499 ITDMs who have already or plan to implement DevOps in the next two years. Source: Vanson Bourne/CA Technologies 2014

“Show me … my own ROI!”

These ROI factors are of course a combination of averages, research data points, and anecdotes. As with all generalizations, when it comes to a specific enterprise your mileage may vary. However, this is certainly a good starting point when building a business case to prove the expected ROI for your organization.

You may find other areas for ROI too – upstream in the PMO, DBA, or downstream in NetSec, training, documentation. If you have additional data points on finding the ROI for DevOps, I would love to hear about it. Add your ideas here in the comments, or connect with me and/or devopsy.kinsta.cloud on Twitter – at @AndiMann and @DevOpsDotCom respectively.

March 9, 2015
Webinar: Application Release Automation at Scale

Organizations of all shapes and sizes depend on feature-packed application releases to keep end users productive and happy. As application complexity grows, managing deployments manually becomes more difficult and slows down the pace of business.

Join Amy DeMartine, Sr. Analyst at Forrester research and Mike Hamrick, Software Configuration Management Lead at Lockheed Martin, servicing CMS Centers for Medicare & Medicaid Services, as they discuss:

1) How Application Release Automation (ARA) solutions eliminate error-prone manual work, decrease risk and increase uptime

2) How Infrastructure and Operations teams in large organizations overcome modern service delivery challenges

3) Best-practices and tools for overcoming the challenges of ARA at scale

Event recorded Thursday April 2^nd, 2015

Click here to register to see it On Demand

Video

Slides

[seoslides embed_id=”a3d2f6e4e229″ script_src=”https://devopsy.kinsta.cloud/embed-script/electric-cloud-webinar/6046/” overview_src=”https://devopsy.kinsta.cloud/slides/electric-cloud-webinar/” site_src=”https://devopsy.kinsta.cloud” site_title=”devopsy.kinsta.cloud” title=”Electric Cloud Webinar” /]

March 8, 2015
Are VMs becoming El Caminos?

In 2010, it looked as if virtualization had won. We expected all servers to virtualize workloads and the primary question was which cloud infrastructure manager would dominate. Now in 2015, the picture is not as clear. I’m seeing a trend that threatens the “virtualize all things” battle cry.

Really, it’s two intersecting trends: metal is getting cheaper and easier while container orchestration is advancing on rockets. If metal can truck around the heavy stable workloads while containers zip around like sports cars, that leaves VMs as a strange hybrid in the middle.

What’s the middle? It’s the El Camino, that notorious discontinued half car, half pick-up truck.

The explosion of interest in containerized workloads (I know, they’ve been around for a long time but Docker made them sexy somehow) has been creating secondary wave of container orchestration. Five years ago, I called that Platform as a Service (PaaS) but this new generation looks more like a CI/CD pipeline plus DevOps platform than our original PaaS concepts. These emerging pipelines obfuscate the operational environment differently than virtualized infratructure (let’s call it IaaS). The platforms do not care about servers or application tiers, their symantec is about connecting services together. It’s a different deployment paradigm that’s more about SOA than resource reservation.

On the other side, we’ve been working hard to make physical ops more automated using the same DevOps tool chains. To complicate matters, the physics of silicon has meant that we’ve gone from scale up to scale out. Modern applications are so massive that they are going to exceed any single system so economics drives us to lots and lots of small, inexpensive servers. If you factor in the operational complexity and cost of hypervisors/clouds, an small actual dedicated server is a cost effective substitute for a comprable virtual machine.

I’ll repeat that: a small dedicated server is a cost effective substitute for a comparable virtual machine.

I am not speaking against virtualize servers or clouds. They have a critical role in data center operations; however, I hear from operators who are rethinking the idea that all servers will be virtualized and moving towards a more heterogeneous view of their data center. Once where they have a fleet of trucks, sports cars and El Caminos.

Of course, I’d be disingenuous if I neglected to point out that trucks are used to transport cars too. At some point, everything is metal.

March 6, 2015
Reverse Executive Sponsorship: Gaining grassroots DevOps momentum

As IT practitioners investigate the possibilities around DevOps and Agile transformations, one of the things they’re likely to hear over and over again is that these type of organizational tectonic shifts require executive sponsorship to truly succeed. But in the real world, garnering that support is hardly as easy as just picking up a phone and asking. Last week at IBM InterConnect in a panel featuring a number large enterprises, DevOps practitioners talked about the challenges of building DevOps momentum when the executives don’t initially buy into the prospect.

“We’ve done this in three different organizations, and I’ve never had executive support—not until I’m considered ‘safe,’ and then they give it,” said Chris Nowak, senior vice president of DevOps for Bank of America. “We’ve done it grass roots every single time. You have to build your own legitimacy.”

John Karnes at Silverpop shared a similar experience. Prior to the marketing firm’s acquisition by IBM, his team started its DevOps journey.

“We had to build our own excitement our own momentum around what we were doing,” explained Karnes, who is director of production operations and performance engineering for Silverpop.

As Nowack put it, DevOps people are internal salespeople and the product is faster delivery and more capacity for innovation. He believes that in order to sell the executives, technology departments need to collect metrics early and often.

“You want to be able to prove what you’re doing, so from the beginning if you can baseline, get some metrics out there,” he says, explaining that his team was eventually able to show as much as ten-times operational speed as a result of its DevOps patterns.

Karnes said that one of the keys to gaining momentum was communicating and playing up milestones throughout the organization. Being in the marketing world, one of biggest metrics for the business was uptime. Before its transformation, Silverpop was running three to four releases a year with about 8 hours of downtime each. Nowadays, that’s down to about 20 minutes of downtime per year. Through internal campaigning, he was able to show along the way what a difference DevOps was making for the business.

“As we started getting releases more consistent, with fewer errors and faster, we’d have an internal email distribution list that we’d send out all of our release notifications to and I’d say ‘Hey we set a new record!’” he explained. “After about a year or so, we’d have executives come in and say, ‘Hey, how’s this release going to go? Are we going to break a new record?’”

In addition, Valerie Scott, manager of application management services for HM Health Solutions added that gaining sponsorship required learning the language that appealed to executives.

“”We were going through a transition in our company and we had new executives coming in and the old staff was used to us communicating with them at the technical level. I was getting nowhere and it was very frustrating,” she said. “Then I learned the word ‘investment.’”

Meanwhile, Curtis Yanko, architecture manager for the DevOps Center of Excellence at Cigna explained that sometimes employees can be some of the best champions for DevOps to executives up the food chain.

“We can talk about how much of our portfolio spend is being wasted and how much is being reclaimed into productivity versus rework or idle time—those are great conversations but I made a promise to my senior leadership,” he said. “We do an annual internal survey with how happy you are at your job and I said ‘I am guaranteeing we will watch this going up.’”

As he put it, individual people who once had seemingly unlovable positions—such as managing the ETL—were all of a sudden excited to be working at Cigna.

“People like that go out and champion for me,” he said.

March 5, 2015
Building software quality into your Continuous Delivery pipeline

The modern software development landscape places a premium on speed. It’s good to be first to market. If you’re going to fail then you should fail fast. A rapidly evolving product can be shaped according to customer feedback, and fast turnaround allows you to stamp out defects and hone the software for your audience.

A continuous delivery pipeline enables you to act and deliver more quickly. But you need to build in quality metrics to ensure it retains a laser focus, and doesn’t become an out-of-control fire hose. It may require some front-loaded effort and resources to set an accurate measure, but the resultant software quality turns short term cost into long term savings.

Plan for quality

You can build the expectation of quality into your software development process from the start. Design your tests before a line of code is written. Create a test architecture that can be woven into your CD pipeline. Build a self-adjusting system that applies the right tests at the appropriate time in development, covering unit tests through to performance testing. That way you always have the real-time insight you need into your software quality.

Why wait?

Test-driven development can save time and resources. By waiting for functionality to be developed before writing up your automated functional tests you’re creating more steps to complete than is necessary, and slowing down the process.

You need real examples of each feature and piece of functionality in order to design and implement it anyway, so why not create them from the beginning? They can serve as acceptance tests and keep developers focused on the core requirements for your software.

This can only work if you ensure that you have group buy-in from day one. All stakeholders must be involved in the acceptance test design. If you fail to get the input you need about what constitutes a holistic view of your software quality, then you can expect feedback that raises new concerns down the line. By that time it will be more difficult and expensive to retro-fit new quality metrics.

Aim for automation

The more of your acceptance criteria you can automate, the better. It provides you with a mechanism for taking the temperature of your software whenever you want to, for a clear picture of its health.

Naturally, there will be acceptance tests that you can’t automate. You shouldn’t try to automate everything. There’s no substitute for human insight and it’s notoriously difficult to predict your end user’s first impressions.

Functional tests are ideal candidates for automation, provided your test architecture considers optimization for regression and remains flexible enough to accommodate new examples when required. For the most efficient use of your resources you need to build in an understanding of dependencies. You need control over the decision on what to test and when, based on the quality attributes you’re examining.

Failures will be flagged at the earliest eventuality for all stakeholders, maximizing your opportunity to address them quickly. By building these quality metrics into your CD pipeline you can go live with confidence.

About the Author/Viktor Clerc

Viktor Clerc is product manager for testing applications at XebiaLabs, a provider of automation software for DevOps and Continuous Delivery helping companies to accelerate the delivery of new software. You may reach him at vclerc@xebia.com.

March 4, 2015
DevOps is Agile for the Rest of the Company

IBM held its InterConnect 2015 conference last week in Las Vegas. The event covered on a range of topics including hybrid cloud, mobile apps, and DevOps. One session I attended focused on Agile development and the parallels to DevOps were apparent.

The session I attended at the Mandalay Bay Hotel was titled “How a Major Telco Improved Cycle Time and Business Results by Leveraging Agile”. Elliot Olschwang and Claudio Cozzi—both IBM employees—talked about helping a specific customer reach its goals by introducing Agile development principles. What struck me throughout the presentation were the many ways that the concepts the two talked about implementing at a developer level for this customer are very similar to the concepts companies must embrace on an organizational level to make the transition to DevOps.

Rosalind Radcliffe actually deserves credit for pointing out the similarities to me. When I spoke to her a few weeks ago she told me that DevOps is just the Agile principle taken to the full enterprise. It provides a more comprehensive end-to-end perspective that enables enterprise-scale transformation.

Agile development was an evolution from the Waterfall method. The Waterfall methods treats project analysis, design, coding and testing as separate and distinct phases that have to occur in order. There are many inherent problems with Waterfall, such as the fact that you don’t actually develop code until half way through the project. Testing would often be cut in the interest of time, and even if testing is done it’s actually too late. Any issues identified in testing mean you have to go back at least to the coding phase again—possibly back to design or even analysis.

Agile removed the linear dependencies of Waterfall to enable development to occur more effectively and efficiently. Aside from imposing a more efficient workflow in general, Agile improves quality because testing occurs earlier in the process and more frequently throughout the process. Subsequently the risk to the project is reduced because there is continuous feedback that enables developers to adjust on the fly rather than having to go back to the proverbial drawing board at the end of the project lifecycle.

DevOps is just a natural extension of Agile. It takes the guiding principles and best practices that Agile brought to developers and applies them on a much broader scale across the organization. As I pointed out in a previous article from IBM InterConnect, “Removing some of the friction between different groups and enabling the organization to function more effectively is one of the primary benefits of DevOps.”

Part of DevOps is to streamline by automating those things that can be automated. Allowing tools to perform tasks so that other resources—namely the human beings—can provide greater value by performing less mundane work.

The other part of DevOps, though, is a culture shift. It’s a change in the ruling philosophy of the organization that erases traditional barriers between departments and processes and enables everyone to function more efficiently without unnecessary overhead or “red tape”.

That second part is where the principles of Agile development come into play. Implementing continuous monitoring for continuous feedback that leads to continuous improvement rather than staying mired in a linear flow that creates hurdles and roadblocks and results in resources sitting idle waiting for other teams to complete a task before the next task can begin.

DevOps is hot now, but eventually it will mature and evolve into something else just like Agile has. It will be interesting to see in a few years how organizations that have embraced DevOps begin to take it to a whole new level.

March 4, 2015
Balancing Between Impostor, Blowhard, and Burnout

I’m glad John Willis wrote his post on karojisatsu, or suicide from overwork, and that it has been getting the attention it deserves in the community. If it weren’t for the push behind the article that has kept it surfacing in my feeds the past few days I definitely wouldn’t have realized I needed to stop and write this right now.

It’s great that this issue has been brought to our attention and the community response has been awesome. I’ve seen a lot of people offer their availability for anyone that needs someone to talk with, but there’s definitely some frustration surrounding what can be done to help without offering useless platitudes, as Gene Kim so eloquently put it.

I have noticed a cycle I can get caught up in that can leave me in a bad place, but now that I’m aware of it I’ve been better at avoiding it. I wanted to share my experience so that others dealing with any of these problems have something they can try that may help.

Be Confident, but not a Blowhard
Christina Xu recently posted her thoughts on blowhard and impostor syndrome. I agree we all have areas we’re comfortable in and shouldn’t take on any work we don’t feel comfortable doing. It doesn’t matter if it’s because we don’t feel we’re qualified to do it yet or if we just aren’t interested in a particular space. Anyone that doesn’t immediately jump head first into a technology isn’t suffering from impostor syndrome. Anyone that does or says they can is probably being a blowhard.

I can and do move in and out of a lot of different spaces in tech but I definitely know my limits. I never really learned much C++ and have no business writing it. However, if the need came up and it made dollars and sense I would try to take some time to transition into C++. This would take months, not weeks, and it would be a lot of hard work. I would even argue it would be painful, but I’m not scared of hard work if it’s worth it.

I Allow a Reasonable Amount of Time to Transition
I have no plans to touch any C++ any time soon. I am presently moving back into JavaScript though which shouldn’t be too bad. I’ve been working in JavaScript since I was in middle school but I haven’t done any deep diving in this space in years. I remember wanting to learn backbone and marionette but the need and the time to transition never lined up. I’ve been working with mostly back end code like services, DAO layers, and build tools the past few years. I’ve used JavaScript sparingly for things it does well, but I haven’t had to think very hard about JavaScript in a while. I really need to gather my self confidence as I dive back into JS, and that means I am at risk of being a bit of a blowhard. Allowing time to transition can keep me from getting too far ahead of myself.

Asking someone if they “ know” JS is a very vague question that could lead to a lot of misunderstandings these days. I need things like angular, ember, node, and D3 but they’re all pretty new to me. It’s really awesome that there’s all this new stuff to play with and better tools to use. I feel I can do more with less thanks to the changes that have happened in the JavaScript ecosystem since the last time I was here.

True Impostor Syndrome
I didn’t always have such reasonable expectations about how fresh a particular knowledge base was in my mind. Years ago I would have dove back into JS head first expecting things to be where I left them and then panicked when I realized the landscape had changed and my skills were rusty. If this realization came late at night after a particularly hard day (never mind week or month) it could leave me in a very bad place. In that moment it’s easy to declare yourself a failure of a human being and vow to never code again. This is true impostor syndrome. At it’s worst it can lead to thoughts of ending one’s own life.

I Limit My WIP to Account for Adversity
Impostor syndrome can attack when you’re working in a domain your comfortable in as well. Because there’s so much work available it’s easy to take on too much. When you’re spread too thin, teetering towards burn out and then slip on a banana peel of an issue with no slack in your WIP to accommodate it you can end up in the same pit of despair.

How I Deal with Burnout
I’ve come to the conclusion it’s better to be proactive in avoiding these scenarios than reactive in responding to them but we need to do both. Let’s assume you’re burned out from trying to transition too quickly or taking on more work than you should have. These things often lead into each other I’ve come to find. Here’s what I do when I know I’ve bagged my limit.

Step Away from the Keyboard
Taking time off seems obvious but we can’t always drop everything and take a week or even a whole day off. There are things I do to mandate several daily breaks during different parts of the week. The key here isn’t to decide if I have time to do them, I just drop what I’m doing and go do them on principle. Whatever I’m putting down should be able wait. This is about establishing a lifestyle change more than just taking a break.

Exercise
If you don’t do anything start with daily walks outside or yoga and meditation. I’m better off doing these twice a day for 15 minutes than going to a class once a week. It wasn’t until I started doing these things this regularly that I started to really feel the benefits. Even a stroll around the office building for some fresh air when the heat gets hot can do you a world of good.

Eat Right
GIGO in it’s truest form. Eat healthy stuff and splurge in moderation. My family eats fish and vegetables almost every night because it’s healthy, quick, and relatively inexpensive (if we buy the vacuum sealed frozen fillets). I’ve been working on drinking more water and less coffee. We do have a larger meal filled with red meat and chocolate desserts every couple of weeks at least. I believe any food that’s good for your soul is soul food but it’s usually not very healthy from a nutritional perspective. You should definitely eat it in moderation anyway.

Go to Sleep!
I used to get home and poke at work in front of the TV until late at night with the DVR spewing garbage at me. I got rid of the DVR and only watch shows as it’s convenient. I get to bed by 10pm more frequently and feel like I’m making better use of my evenings. I have trouble sleeping more than 6 hours at a time, but I try to leave room for 8 in case I need it and I definitely take it when I do.

Find a Computer-Free Hobby
I happen to be very passionate about BBQ. I try to smoke some brisket or pork at least once a month. I also play a handful of instruments very casually as I don’t want it to turn into more work. Band practice and playing gigs takes a lot of energy and would not result in the desired effect in my particular scenario.

Volunteer
Nothing will make you feel like you matter in this world than giving back and being of service. I coach my kids’ soccer team but the sincerity of the “thanks Coach!” I get from some of the kids and their parents feels really good. I completely wasn’t expecting it but I now understand why some coaches stay in the program long after their kids have grown up and moved away.

There are a lot of people eager to learn to program that would love your help. Look in your community and see who you can help with HTML and CSS. It doesn’t have to be children either. A lot of adults are look to refresh these skills or are trying to move into technology from another field. They would love a mentor they can learn from and you’ll end up being better at what you’re teaching too.

They also need help at food banks, animal shelters, and many other places. Find something you care about and try to get involved. Trust me, you’ll feel better about life.

No Burn Out BBQ and Free Cookies
I am also here for you if you need me. Specifically, if you’re in the Atlanta area and need someone to talk to hit me up. You could live here or be through on business but I will absolutely break out and come meet you if it’s at all possible. I feel the face to face interaction is needed for me to really connect with someone and give them my undivided attention.

On that note, I’m interested in starting a No Burn Out BBQ to provide an opportunity to do all of these things in Atlanta. Having a BBQ one weekend a month would get us away from the keyboards and outside into the fresh air. It takes a lot of volunteers to get the meal together, and it would provide a safe place to come talk about burn out on top of serving as a monthly reminder to unplug and decompress a bit.

I have a side project with my kids’ that involves sending people cookies and a drawing for filling out a survey. I’ve opened a survey to gauge the level of interest in a No Burn Out BBQ. Feel free to answer this even if you don’t live in Atlanta. If I get a large enough response from a community we’ll figure out how to hold a BBQ there too. There’s also a fair chance that some cookies and a picture in the mail will do you more good than you realize.

Please don’t be shy. You’re in a community that really does care and wants to help. If you prefer something anonymous the Suicide Prevention Life Line is available. Even if you’re not at risk and just want to talk to somebody you can reach out to them 24 hours a day, 7 days a week: 1-800-273-TALK

March 4, 2015
Isn’t Continuous Testing a Longevity Test Anti-pattern?

Accordingly to Jim Coplien: “an anti-pattern is something that looks like a good idea, but which backfires badly when applied. “ In DevOps context, continuous testing is an attractive solution for the narrow context of testing specific software components quickly (in cycles measured in minutes or hours) but may be “bad” if the system that the software is a component of has failure modes (E.g., effects of memory leaks) that will only be observed during long term testing over several days and if the time between releases is greater than that. In other words, the pattern of quick test cycles prescribed by continuous testing appears to be a contrary Anti-pattern to the fundamental concept of long duration “longevity” testing. So what are the well formulated positive patterns that are applicable in its stead?

In my prior blog entitled “From QA to Continuous Testing” I indicated “QA test experts that are most skilled in the best practices of test design are valued because they will produce the best quality products.” QA test experts assert that it is unwise to release a software product that has not been tested for durations comparable to the lifecycle of a release. Only tests that are run for a significantly long time compared to the release cadence are deemed sufficient. Just how long is long enough? This is a subject for further discussion beyond the scope of this short article, but my own observation is that best practices test environments run accelerated level load and impulse load traffic patterns on a build candidate for at least 48 hours or more before a release is determined sufficiently tested for at least limited deployment. Even 48 hours is much longer than typical “continuous testing” cycles. So “what positive continuous longevity testing patterns are applicable in its stead?”

Continuous Longevity Testing Patterns may sound like an oxymoron but in fact longevity testing is compatible within the context of continuous testing principles. Below I describe three Continuous Longevity Testing Patterns that are sometimes used.

Pattern 1: Longevity testing time windows are reserved such as weekends. In this pattern the regular Continuous Integration/Test cycles are interrupted during the weekend and longevity tests are only run on the build that was last created prior to the weekend time window. This has the advantage of not requiring any special test resources other than the resources used for regular CT tests and does not require any sophisticated scheduling to orchestrate the test environment. The disadvantage is that the longevity test results are only available weekly, it is hard to diagnose which build caused a longevity failure and problems with release candidates will delay the release for a week until the next longevity test run validates the fixes.

Pattern 2: Longevity parallel testing is scheduled frequently, perhaps as frequently as every few builds, or perhaps daily. This pattern has the advantages that the regular Continuous Integration/Test cycles are not interrupted any time (they run 24/7), the longevity test results are available for more builds than Pattern 1, which reduces diagnostic complexity and may reduce release delays, but has a disadvantage that separate test resources must be reserved for each longevity test in parallel with the regular CT tests. For complex systems under test with many test targets and lots of builds this can be very expensive so there is a tradeoff between test resource costs, and the cost of delays if not done frequently enough.

Pattern 3: Intelligent longevity testing is scheduled only when certain conditions trigger the longevity tests to be started according to smart rules that correlate software changes to problems with longevity performance. This pattern has the same advantages as Pattern 2 and has the further advantage that the cost of test resources is usually much less than Pattern 2 if the rules for the triggers don’t fire too frequently. The disadvantage is the entire scheme depends on the creation of a rules engine and rules that understand the risk of changes for long duration testing. If the rules are too lenient then this pattern might allow too many longevity failures to escape but if the rules are too tight then this pattern is not any better than pattern 2 and yet more complex to implement.

So which Continuous Longevity Testing Pattern is best? The best answer is the one that best matches organizations’ goals for release cadence, cost of testing, and risk.

So indeed, Continuous Testing is a Longevity Test Anti-pattern but fortunately DevOps infrastructures implemented according to best practices have a variety of complementary Continuous Longevity Testing Patterns to choose from.

At Spirent we think testing has a bright future in DevOps. You can read more about our views at
Spirent.com/solutions/devops

What do you think of these patterns or do you have others that should be mentioned?

March 4, 2015
“I want to do Continuous Deployment”

“We want to do continuous deployment” – i’ve heard this many times, but have yet to find a scenario where it actually fit. Why? Because you are confused. And rightly so. How many two letter acronyms can be thrown at you in one movement. DevOps takes the confusion further and has two of the acronyms identical, but meaning very different things. The CDs, which are not the round shiny discs in this case. This straight forward post hopefully helps clear up some of the confusion.

(more…)

March 3, 2015
Avoiding Common Executive DevOps Mistakes

A seasoned DevOps practitioner with experience leading both HP and Macy’s on continuous delivery transformations, Gary Gruver has learned the ins and outs of implementing Agile and DevOps at enterprise scale through the “school of hard knocks.” Now the president of Practical Large Scale Agile, a consulting business designed to help other enterprises navigate similar DevOps journeys, Gruver stepped in front of the crowd at IBM InterConnect last week to offer up some lessons he learned in those processes. His experience provides some valuable advice for executives hoping to avoid pitfalls while leading the charge for DevOps.

Scrum Doesn’t Equal Agile

The scrum methodology may be useful, but Gruver explained that executives must understand that in the enterprise, Scrum does not equal Agile. The problem with leaning solely on Scrum is that it emphasizes teams locally optimizing their work, while the entire organization is still not fundamentally transforming its ability to release code for the highest priority features on a more frequent basis.

“If you don’t scale over time, you’re going to end up with the classic implementation that you see in most enterprises which is a water-Scrum-fall,” he said. “Your teams are sort of doing stand-ups, they’ve got burn-down charts, they’ve got burn-up charts, they’re collaborating and doing all that work. But they’re still trying to deliver plans that were developed a year or 18 months in the future. And then when you integrate your code it is still taking days, maybe weeks, maybe months or even quarters to release code.”

Too Many Hands On The Steering Wheel

Letting teams loose in the organization lead only by Agile coaches and without some executive and management interface to bring things together results in “too many hands on the steering wheel,” Gruver said, explaining the prevailing attitude.

“You’ve got a layer of management between the CFO and the teams and you know what their role in Agile transformation is? ‘Stay out of the way. Don’t bother us, you don’t know anything about coaching teams and you’re in the way,’” Gruver said. “Most Agile transformations completely disengage the management team. If you get this continuous improvement process going at the executive level and they’re engaged in setting objectives, and they’re spending time in the organization working with teams figuring out what’s working and what’s not, that changes culture.”

Gruver says that executives and the management team should be working as “investigative reporters” looking for impediments to progress and helping to remove them.

Be Realistic About Architecture

It can be discouraging for DevOps champions in the enterprises who take lessons from organizations like Google or Netflix that thrive on the microservices model and then try to apply them within their traditional organization. The problem is that large, legacy organizations often have applications that are “too tightly coupled to the rest of the architecture to be independently developed, deployed and qualified.”

While organizations would do well to decouple these applications to allow for better incremental improvement, that’s not always realistic, Gruver said. This means enterprises will need a lot more coordination effort—led by executive prioritization—to scale DevOps, because “you can’t wait for a complete rearchitecture to fundamentally start improving your business. So you’re going to have to find a pragmatic balance between the two.”

Branches Are Evil

Gruver is a strong proponent for Trunk Based Development and used his executive oversight to set his enterprises on that path to efficiency in spite of strong push-back from the developer ranks.

“Branches are evil, branches are evil, branches are evil. They’re duplication in the process, and especially if you have any manual testing, they’re absolutely going to kill you,” he warned. “Try to avoid them where you can. I get that large, traditional organizations are very dependent on branches and it is going to take you a while to transform that. But every time you see a branch, you should think of it as waste in your system and try to figure out what’s the root cause for having it and how to eliminate that.”

Failing To Set Prioritized Backlog

Gruver advised enterprises that they shouldn’t start their DevOps or Agile transformations without having business managers who manage line of business “What is the business reason for transforming your organization.”

“This is going to be hard. It’s impossible to do everything at once, so you’re going to need to figure out where to start,” he said. “If you don’t have a clear business value for doing this, then it’s going to be hard to get momentum and have the vision for your organization to follow if it doesn’t know what to prioritize first.”

March 3, 2015