devopsy.kinsta.cloud

Tag: agile

Three essential elements to maximize business value with DevOps

DevOps is fast becoming the winning strategy needed to accelerate the continuous delivery of exceptional software without sacrificing quality and cost efficiency. With its focus on better communication, collaboration and feedback within and across development and IT operations, DevOps is gaining many fans – even in the business, where performance is seen as being directly attributable to its adoption.

I’d be the first to agree that buzz plus a liberal dash of hype follows any new tech trend. We’re seeing this big time with DevOps. Already it’s one of the most Googled tech trends.

Many vendors are washing and making ready their ‘DevOps products,’ while the #DevOps noise is shall we say, trending – we just love disruptive technology. If you have read the Puppet Labs 2014 State Of DevOps survey, organizationswith high performing DevOps teams were 2.5x more likely to exceed profitability, market share and productivity goals, and had 50 percent higher market capitalization growth over three years.

Don’t get me wrong I’m a disruptive tech fan too, but there’s a caveat – technology disruption is nothing unless it’s truly transformative– and that’s where I place DevOps. It’s not a technical nicety, but actually a business imperative for companies to thrive in an increasingly software-driven world.

Software applications are rewriting business. They’re no longer in the background ‘counting the beans’ and running the supply chains – they’re front and center – the mobile systems-of-engagement that shape your brand, delight your customers and, here’s really the cool part – disrupt markets.

Sounds great you might say, but there’s a problem– the traditional fixed speed limit on the super highway of business computing.

Enterprises have been stuck in the same gear for years maintaining large, complex/interconnected legacy systems. Mainly transactional and supporting complex business processes, the norm has been to change when only necessary (or forced to change?) – never exceeding the speed limit. Now however, consumerization of IT has dictated rapid delivery of software innovations. This requires a DevOps approach, where teams have a shared responsibility for not just keeping the lights on, but powering business.

Being able to increase pace is important, so too nimbleness. By making IT infrastructure more flexible and the instrument used to digitally engage and interact with customers, cloud drives business agility. But it’ll never realize its full potential if IT continues operating in silos using manual processes. This is why DevOps collaboration, shared processes and tools are so critical for harnessing the cloud – or using and an analogy – if cloud is the supercharged engine of business – DevOps is the fuel that powers it.

So with mobility and cloud computing being essential to a software-driven business, what additives do we need in our ‘DevOps fuel’ to spark agility and generate the greatest returns?

There are three big ones:

Agile Parallel Development

DevOps must reduce all elements of waste. This is a tough nugget to crack because existing IT environments generate tremendous inefficiencies. Often separate dev teams have to contend for physically constrained resources, with quality compromised by poor test data, shortcuts and bottlenecks. Testing especially is seen as IT’s “angry monkey”; something to be loathed, when in fact it should be a unified process that bakes excellence (functional and non-functional) into everything – from mobile apps and API’s to back-end systems – from dev day 1 and persisting right through to production.

With Agile Parallel Development and shifting testing earlier in the lifecycle, teams avoid delays in the development of high-quality code. Toolsets are important here, especially service virtualization, which by removing testing constraints reduces cycle-times plus prevents bugs finding their way into production. And, since business will increasingly engage customers across digitized multi-channels (mobile, API’s), teams will exploit real-time intelligence to test new ideas and experiments – pinpointing how functionality and performance converts to business gain.

Continuous Delivery

Release delays are a classic business fail. Now, success depends on delivering a continuous flow of value from new mobile apps, APIs and software innovations. These types of services will be constantly enriched to capture new markets and customers yes, but they’ll also be operating within shorter windows of business opportunity – faster lead times mean everything.

For IT, this ‘quick or dead’ imperative means teams must challenge a culture based on avoiding change and fear-of failure by using Continuous Delivery processes and tools to accelerate and streamline complex, multi-tier releases; automating the testing, deployment and promotion of changes across all environments. Additionally, these processes should integrate other management requirements – for example, establish service-level monitoring and reporting with the application when released.

Agile Operations

Agility must extend to operations. For too long we’ve been asking questions like – “how long can we go between failures?” or “who’s to blame for this problem service?,” but we should be asking – “If we want to increase customer conversion rates from our mobile app, what levels of performance are needed?” or “how can we detect and remediate performance problems before code reaches production?”

This shift requires Agile Operations – shared processes and tools that optimally manage applications, but also feedback intelligence to the extended team. For example, Application Performance Management should pinpoint operational problems faster, but also ‘gift’ development rich insights they can use to improve code quality. Furthermore, teams will leverage advanced analytics (like how, where and when mobile apps are being used) to drive just-in-time enhancements –that’s a new take on end-to-end– customer ‘engagement-to-experience.’

Business demand for software innovations is insatiable, and for good reason. Now the simplest mobile app experience or secure API can transform your business or screw it up completely.

By establishing a rampant innovation culture with DevOps and the right processes and tools, IT will not just meet expectations – it’ll be the spark for growth and innovation.

About the Author

Aruna Ravichandran is vice president of DevOps Solution Marketing and Management at CA Technologies, responsible for cross-portfolio solution marketing for DevOps across all product lines. Before joining CA Technologies, Aruna held several management positions in product marketing and engineering at Juniper Networks and HP. Aruna earned an MBA and master’s degree in Computer Engineering from Santa Clara University and holds a bachelor of science degree in computer science from Bangalore Institute of Technology. Ravichandran is on twitter @aruna13 and on Linked in

September 10, 2014
The DevOps Litmus Test: Is Your Company’s Culture Ready to Get Agile?

New ideas don’t come to those who are afraid of breaking the rules. Improving something usually means you’ll have to push the status quo aside and try something that hasn’t been done before. What may be a standard operating procedure will have to make way for a brand-new approach with its own set of rules and goals.

This applies directly to DevOps, which requires changes in how departments operate, how employees interact and how problems get solved. DevOps may be an idea your company is considering, but if the culture isn’t compatible with it, DevOps will never become an active part of your day-to-day operations.

If DevOps Is the Engine, Culture Is the Fuel

Let’s say two employees who don’t normally work together wind up taking an unprecedented approach to solving a problem. To their surprise, this new way of working together is incredibly efficient and a big improvement over their usual methods. The two recommend this kind of collaboration to others in their respective departments when it comes to spotting problems, fixing them and preparing for future ones.

That’s the moment when you’ll know how receptive your company is to DevOps. This kind of cultural litmus test determines whether department silos can be broken down and new avenues of teamwork can happen. No matter the company or industry, its culture is the foundation for DevOps adoption. When that culture fits, DevOps is an ideal vehicle for improved productivity, with culture being the fuel moving it forward.

Getting the Right Cultural Fit

In order for DevOps to happen, you need the right conditions, similar to how certain plants only grow in the right ecosystem. Team members must be able to interact in ways they may not have in the past. The right culture is one that allows flexibility, spontaneity and resourcefulness to take place.

On the flip side, DevOps won’t work in a rigid environment where departments are siloed and methods can’t be questioned. DevOps has a slim chance of taking off in a company that wants to hold onto previously established rules about divisions between departments. Anyone who stands up to say, “That’s not how we do things,” will be a hindrance to progress. Greater benefits are in reach, but to make them a reality, a company and its employees have to be willing to make some changes and allow agile collaborations to flourish.

DevOps Culture in the Real World

For those who have already put DevOps in action, culture was, and continues to be, a central issue. Household names like Twitter and Netflix may be known for their DevOps practices, but many lesser-known businesses have found that their cultures identify with DevOps collaborations. The recently published e-book from Rackspace, The DevOps Mindset: Real-World Insights from Tech Leaders features interviews with a variety of DevOps practitioners. The importance of culture is a common theme throughout these discussions.

These firsthand accounts emphasize how encouraging new ways of solving problems depends on the proper culture. As James Kenigsberg of 2U explains, “You don’t have to be in Silicon Valley to embrace the DevOps culture; anyone can do it anywhere with the right people.” Bharat Krish, CIO of HBO Latin America, also emphasizes culture as an essential factor. “When I think of a DevOps culture,” Krish says, “for me it’s building a foundation where development can happen in a more agile fashion.”

Start with the Biggest DevOps Requirement

Knowing that DevOps benefits one company doesn’t necessarily mean it will automatically benefit another. Company processes, structure and communication standards all play a part in determining whether DevOps is right for your business. DevOps isn’t a product that can simply be plugged in – it requires a new way of thinking about agile software deployment.

Culture has to be at the center of a DevOps adoption, and assessing your company’s culture is a crucial first step in determining whether DevOps will be a good fit for your organization. As long as your company is flexible enough to allow some cross-departmental collaboration and silos to be broken, then prepare for DevOps to take you to new levels of productivity. You’ll find that breaking the rules isn’t such a bad thing.

August 27, 2014
Keep Calm and Refactor Operations

Developers have been refactoring forever to gain efficiencies and avoid mistakes. Let’s apply more than just theory to ops to do the same.

There’s about a million articles, blogs, tweets and infographics available today that bespeak of DevOps as this or that or the other thing. But one core truth remains: DevOps in part attempts to apply methodologies like agile to operations in order to gain efficiencies and avoid mistakes. We need the efficiencies because the number of apps that need to be deployed, updated, migrated, and maintained is increasing, not decreasing, and there simply isn’t enough time in the day (or during the monthly change window) to get it all done.

We need to eliminate mistakes because they are, according to data from initialstate, very costly. It takes 30x longer to fix one bug (an error) than it does to write one line of code. Anyone who has ever spent time chasing a wild pointer will immediately say that’s too low an estimate, but anecdotally I’m willing to go with that figure. It is an average, after all. To put that in perspective, the same data indicates that developers must find and fix 100 product bugs per 1000 lines of code. That’s a 10% error rate.

I’ll let you ponder for a moment what that means to DevOps and the desire to treat infrastructure as code.

Exactly. You probably want to avoid that kind of error rate as you go about automating and orchestrating provisioning, configuration and general management of all that infrastructure. Just a few errors can effectively negate time gained by automation. So how do you avoid that?

One of the ways to avoid some of that is refactoring.

Refactoring is most commonly used to extract pieces of code (script) that repeat over and over into composable functions (in a procedural language) or methods (in an object-oriented language) or services (in a micro-service or SOA-based system). It’s a primitive form of abstraction that serves to produce more manageable, extensible code bases.

Basically, you’re taking a set of actions that you repeatedly perform and turning them into a reusable, repeatable “thing”. In the case of operations that “thing” is probably a script.

Say, for example, that every time you deploy an application you’re also deploying a load balancing service because, of course, you’ll want it scalable and/or reliable. You always perform the same set of steps to complete this task with only a few changes – usually IP addresses, host names, or other network and app-specific settings.

But you’re doing it manually, or writing a new script every time. Manually making changes are, in most cases, where errors are introduced. Fat fingers are a common cause, especially when you’re not paying close attention because, well, boring! You’ve done this fifty million times and quite frankly, you’re sick to death of the tediousness of it all.

We’ve all done it, don’t pretend you haven’t both thought it was boring and made a mistake. You’re in good company, trust me.

What developers do when this is the case is refactor the common code and reuse it across the system. Those bits that get changed are parameterized and everything else stays the same. Even DBAs do this, they just call the resulting code a stored procedure.

The result of such efforts are self-contained automated tasks that, when used to orchestrate a process, enable composable operations. You string together a set of these scripts and voila! Provisioning complete, minus the fat fingered errors or missed manual steps.

Refactoring code is much more complex than this, of course, as would also be the case within operations, but the general premise here is that the practical application of DevOps includes the process of refactoring operations. It’s about restructuring it into more composable, logical and discrete tasks that can be automated and orchestrated as a means to achieve more manageable and extensible operations. It’s about identifying those steps in a process that are repeatable and can be refactored into a single, shared executable task.

When starting an initiative that applies DevOps it is likely you’re going to have to prove the time you’re taking is worth the investment. One of the ways to do that is by refactoring those operations that are continually used and reused to enable a more consistent, predictable and repeatable provisioning and deployment process. By choosing those steps that are most often repeated and refactoring them into their own composable scripts, you can gain efficiencies as well as reduce errors (and eliminate the time they require to find and fix).

August 18, 2014
The DevOps Diet (Getting Lean)
IT has gotten fat – fat silos, fat processes, fat procedures. Adopting DevOps is a great opportunity to look at some of our unhealthy habits and identify where we can eliminate unproductive waste from our process diets. DevOps will ultimately teach IT how to exercise more with less effort, trim the fat and get lean.

According to lean principles, the seven main areas of waste are:
- Defects – variations from requirements that result in interruptions and re-work
- Overproduction – delivering something more or before it is required
- Inventory – carrying excess raw materials, work in progress (WIP) or finished goods
- Over-processing – doing more work than is required
- Motion – moving people or equipment more than is required
- Transportation – moving products from one location to another
- Waiting – doing nothing or moving slowly while waiting on a previous step
What are the greatest sources of waste in your IT organization? Are defects being regularly passed downstream? Are ITSM processes overproducing through complexity and bureaucracy? Are Agile teams dealing with excessive work in progress and less finished product? Are unresolved bottlenecks resulting in frequent delays while waiting for something to get done or someone to be available? Could automation reduce “motion sickness” and make transportation nonstop? These questions will hopefully facilitate dialog between Dev and Ops about analyzing and streamlining existing workflows.

While waste is fattening, it is not necessarily ineffective. Projects and other tasks are still getting done and therefore some areas of waste may be difficult to recognize. To become leaner, start simple by identifying and eliminating one unhealthy habit from your IT diet. Engage stakeholders from Dev, Ops, customers and suppliers (if appropriate). Discuss ways to improve the work with healthier habits. Agree on what to do next or what not to do anymore.

Every diet should be supported by recipes for success. The upcoming DevOps Cookbook by Gene Kim, et al. will provide more tangible guidance and recipes for healthy IT habits. More on that in future blogs.

Most of all, identifying and eliminating waste through Lean DevOps is a great opportunity for collaboration. After all, it’s always better to diet with a friend.
August 11, 2014
Microsoft strives to embrace open source culture

In many ways Microsoft is going through the corporate equivalent of a mid-life crisis—it has reached a stage where it longs to drop a few pounds, and turn back the clock a few years. After years of tremendous success, it’s now starting to show its age, and it is struggling to adapt to the new tech landscape and compete with younger, leaner, more innovative competitors.

The shift began under ex-CEO Steve Ballmer. Ballmer’s approach, though, was misguided. Ballmer’s Microsoft suffered from hubris, and ignored the seismic shift in technology until it was too late. When Ballmer finally recognized the existential threat, and tried to turn the ship around, it was done from within the established bloated corporate culture. There was an effort to rearrange the chairs on the deck with a management reorganization, but it simply wasn’t enough.

Then Satya Nadella took over. Nadella has a more aggressive vision. It’s still not an easy job, nor a goal that can be accomplished overnight with a behemoth the size of Microsoft, but Nadella has dramatically changed the culture, and the vision for where Microsoft is heading.

Part of the new and improved Microsoft includes embracing communities that Microsoft has long ignored or alienated, and shifting the way things get done at Microsoft so the company can be more agile. It is evident in how Microsoft partnered with Chef and Puppet Labs to integrate popular DevOps tools into the Azure cloud platform, and it can be seen in how Microsoft is adopting open source.

“Recent articles have shown that Microsoft is “open sourcing” code to their internal teams. The fact that that is advertised as innovation is telling,” declared Michael LaVista, CEO and Founder of Caxy. “That’s what smart organizations have always done. In fact, Microsoft has somehow found itself the underdog in this sense and probably needs to do things like this to attract talent.”

Leveraging open source concepts internally isn’t quite the same thing, though, as actually making Microsoft code open source. Open source projects like Apache, Project Libre, or Docker share source code with the general public, and anyone can review the code, and submit changes to improve or fix the code. Open source communities collaborate to continuously enhance the code. Microsoft appears to be doing something similar, but without exposing its source code to the general public.

LaVista suggests that Microsoft should take that next step, though, and work collaboratively with the open source community outside of Microsoft’s walls. If it does so, Microsoft will some issues to address. Will it accept code fixes from a 14-year old on the other side of the world? Will exposing source code for the Windows operating system, Microsoft Office, or other popular Microsoft applications open up a Pandora’s Box of security concerns?

“They will have to embrace what happens to code when it is open sourced. Once the code is out there, everyone can see the flaws,” explains LaVista. “The benefit of the model is that you have so many more eyes looking at it from more perspectives. If Microsoft is used to running their own software and making their own decisions about what to work on and fix, this could pose problem.”

Perhaps. LaVista makes good points about concerns Microsoft would face, but in and of themselves they’re not a reason not to adapt and evolve. Yes, exposing Microsoft source code to the general public would probably result in a spike in security concerns, but once the dust settles the collaborative approach of an open source community addressing those issues could result in quicker patch releases, and make the software more secure.

LaVista recommends that Microsoft use companies like Red Hat, IBM, and HP as examples of how a larger corporation can be successful in an open source culture. Only time will tell. We’ll have to wait and see to what extent Microsoft makes the shift to open source, and we’ll have to wait and see how Nadella’s Microsoft adapts, and whether Microsoft can successfully reinvent itself to remain competitive in this new tech landscape.

August 6, 2014
Two Ways DevOps Unlocks the True Potential of Agile

In a recent post on InformationWeek, Erik Weber, a professional Scrum trainer and Agile expert, explores four signs that you’re doing Agile development wrong. Upon finishing the article, I was reminded that Agile in theory is often very different from Agile in practice, and when that gap exists within IT, it can have massive consequences on both development operations and the business.

At the risk of stealing some of Mr. Weber’s thunder, I’m going to spoil the first two signs he discusses – namely, “failure to ‘get done’” and “failure to launch” – because I believe they are common challenges for many IT groups. The good news is, organizations can close their gaps and realize the true value of Agile development by incorporating DevOps processes and tools into their systems development life cycle (SDLC).

Getting to “Done” Faster

“If it takes you longer than a few weeks to turn an idea into a production-ready feature, you are not Agile – full stop,” says Weber, and I would agree. The whole point of Agile was to move away from the slow, monolithic development methodologies of the past by breaking the process into smaller chunks and having developers, testers and business development people working concurrently to speed new features to market. Most Agile practitioners would say they’re doing just that, but many are still not “getting to done” as fast as they would like. Why is that?

According to Weber, the biggest problem is that many IT groups leverage Agile techniques for development, but then do all of the testing at the end. Obviously, this creates numerous problems that can delay project completion. As we all know, the later bugs and glitches appear in the SDLC, the longer it takes to fix them. When that happens, costs go up, delivery pushes out and customers are unhappy.

The DevOps principle to employ here is getting the operations team involved in testing earlier in the SDLC – essentially shifting their work left – and the tool that can help is service virtualization. By simulating constrained or unavailable systems across the SDLC, service virtualization enables developers, testers and performance teams to work in parallel for faster delivery and higher application quality and reliability. With DevOps, they can realize truly parallel Agile development without constraints – and ultimately get to “done” faster.

From “Failure to Launch” to Continuous Delivery

“If you haven’t actually released your software to production – and on to customers – for three or more months, you are missing the big picture,” argues Weber. He believes that many product managers and executives hold back official product releases because they “must have all the features,” and when this happens, they forfeit the ability to both get new features into customers’ hands on a regular basis and react more nimbly to their wants and needs.

There are numerous reasons why the “all features” mindset persists, but a good way to prove its obsolescence is by incorporating the DevOps principle of continuous delivery into the development process. Continuous delivery means always being in a ready state for pushing applications and services through the SDLC and into customers’ hands – and release automation is the tool that helps organizations reach that state.

Release automation enables continuous delivery by automating complex, multi-tier release deployments through orchestration and promotion of applications from development through production. As a result, visibility improves across an organization’s deployment chain, giving it the flexibility and agility to continually release new features to market and react quickly to changing market demands. Once an IT organization reaches a state of continuous delivery, it becomes easier for executives and product managers to pull the trigger on a given release – because they know that the next one isn’t far behind.

Agile is as Agile Does

Weber offers four signs of ineffective Agile development in his article, but I would argue that there is a unified idea running through them all: It’s not enough to employ Agile practices; you have to follow through until you meet your Agile goals. Ask yourself, “Are we really delivering releases to market faster, at a higher quality and at lower cost to the business?” If the answer is “no,” the right DevOps methodologies and tools might be just what you need to put these failures behind you and finally realize the true value of Agile development.

June 4, 2014
Your End Users Want Everything: Here’s How to Give it to Them

In today’s application-centric business environment, the end user is king. Whether we’re talking about being first to market with the next big thing or adding new capabilities and features to an existing application, so much of our current development time, resources and energy is spent on trying to meet the needs of end users.

It’s not difficult to see how we got here. The proliferation of social media and mobile and other connected devices has empowered end users to make their voices heard – and whether their opinions are positive or negative, we have no choice but to thoughtfully consider them if we want to produce market-leading applications.

At this point, you might be thinking, “How is this different? End users have always been king.” You’re right, but the difference between then and now is that we used to build roadmaps and select features based on what we thought they wanted. Now, thanks to social forums, user reviews and other avenues for collecting feedback, we know exactly what they want. The trick, of course, is finding ways to deliver on that knowledge quickly and at a high level of quality without breaking the budget.

Taking Agile to the Next Level

Agile was one of the first methodologies to incorporate user feedback directly into development. You know the drill: choose a small set of capabilities to develop, release the build to a select group of users, collect and incorporate feedback into another build and continue iterating until the application is ready for production.

One of the goals of DevOps – which CA defines as best-practice application delivery methods that improve communication, integration and collaboration between development and operations – is to enable Agile-like development practices that incorporate end-user feedback from after an application is on the market.

We talked a bit earlier about collecting such feedback. Whether you read forums, poll customers or even collect usage data directly from the application, you know that getting the feedback isn’t the hard part – it’s incorporating that data into the development lifecycle in a way that meets your speed, quality and cost requirements.

Starting with Pace-Layered Approach

When it comes to being nimble in the software development lifecycle (SDLC) and reacting to the marketplace, one of the challenges many enterprises have is that their IT infrastructure is complex, comprised with legacy and modern technologies that have been assembled over time. According to a major analyst firm, the key to speeding development in such environments is to segment the infrastructure into layers: namely, systems of record, systems of differentiation and systems of innovation.

For systems of record, think foundational technologies like mainframes that have typically lengthy development periods, versus systems of innovation, such as mobile apps or cloud services, that have a lighter footprint and are more flexible. Once you’ve categorized your components into these layers, you’ll be able to react to customer feedback most quickly with your systems of innovation – and there are two types of tools to help you do just that.

Application Delivery Tools That Speed Development and Delivery

With a service virtualization tool, you can simulate constrained or unavailable systems across the SDLC, allowing developers, testers and performance teams to work in parallel for faster delivery and higher application quality and reliability. In other words, service virtualization enables you to decouple your systems of innovation from the slower moving systems of differentiation and record, so you can quickly deliver on customers’ requests, even amidst all the change and complexity in your IT environment.

To further accelerate the SDLC, a release automation tool can automate complex, multi-tier release deployments through orchestration and promotion of applications from development through production. This tool is your assembly line, driving the application from code development to production, so you can get new innovations into your customers’ hands faster than ever before.

Addressing end-user feedback is an ever-evolving, constantly moving target. But if you follow Gartner’s pace-layered approach and plug your systems of innovation into the right application delivery tools, it’s a target you’ll always be able to hit.

May 29, 2014
Deputizing Everyone for Security – Building Agile Assurance

Those of us highly focused on the delivery pipeline of DevOps will wonder why we should include the security guy to the party. After all, aren’t they just going to slow down the process and make it harder to deliver good features to end customers?

My prior post about introducing SecDevOps by example was not about defining a new role, but introducing how security can and should be part of the automation delivery suite. Whether you are talking about traditional infrastructure change process or automated instantiation of new nodes to handle customer demand, the tools and processes of DevOps merged with security requirements are available to provide security accountability and assurance. Those of you coming from the traditional security space, which might fear security automation, should consider just how exciting SecDevOps delivers on agile assurance.

Rich Mogull provided an excellent example and code to show how firewall changes in a real life situation can be fully automated. There is no need to be squirrely about what is emerging. Everyone needs to be involved with development and operations to take more opportunities to minimize risk.

Dwaye Melancon also recently wrote about the issues of security and risk when the baton is handed off from one owner to another. This too is another great example of where process and monitoring automation aids in delivering continuous security assurance.

I believe everyone craves for a future when security events are automated with full transparency from beginning to end. If security had a real closed-loop transparent agile process, then just imagine how easy it could be to deliver on both compliance and real risk reduction. As Richard Stiennon recently stated in a CloudPassage webcast “Compliance standards are woefully behind with the reality of operations”. Haven’t we all had an experience with an auditor and a clipboard asking for specifics that are clearly aligned with another age of computing?

Wouldn’t it be better to give auditors the same tools, to get a picture of accountability and risk decisions made in the DevOps cloud environments? Take for example the recent HeartBleed events that many security teams are still struggling with. As a DevOps professional, I would have rather just showed the security team and the auditors the truth – that none of my team did anything at all to mitigate this risk. Truth be told, our orchestration tools had already installed the new version of OpenSSL and generated audit logs before many people had their first cup of coffee.

Security can be improved with more integrated automation and by deputizing everyone to do more with DevOps tools. A SecDevOps approach will improve the compliance and audit process with tactical security tools for all. Burn the clipboards and provide an enlightened way to prove best security practices from the collective group of Dev and Ops. Maybe give the Security guy a break and invite him to the after audit beer bash too.

May 21, 2014
Waterfall and Agile, the effect of automation and continuous integration
Agile’s A Silly Idea?

I was a young developer working at a large enterprise when I first heard about some of the principles that later became part of the Agile process back in the mid- to late-90s. I was skeptical about what I heard, to say the least. After all, what we today call waterfall had been the norm for a long time; software got built and then made stable in the end. Yes, it required a lot of wailing and gnashing of teeth, but wasn’t that the way it was supposed to be?

The idea of pair programming at first seemed like a joke to me when I first heard it: talk about a waste of time and resources, and the perfect way to allow a mediocre developer to skate by and let someone else do all the hard work. Just the perfect thing for big companies: let the dead weight glom on to the ones who were productive.

Even the names, titles like Scrum and Extreme Programming sounded like a joke. Ooh, you’re an Extreme Programmer… let me kiss your feet! Ah, the good ole’ days as a young developer, with a huge ego, showing everyone else how to do it. But I digress. Anyway, I was of course skeptical, having the first discussions around Agile handed down to me from on high, from some nameless technology group that couldn’t possibly understand our world, or understand that we were building high quality software saving our company millions of dollars a month in automation.

I was never put into the XP training programs that were being instituted: I ended up leaving that company before I was exposed to it. So, I went from that job to a start up product company, StillSecure, where we also fully embraced waterfall, but a less documentation-heavy version of it, and we didn’t have business analysts. And, rather than 18 month schedules, we were working on 3-4 month schedules. It was waterfall-lite, and it did work a lot better (even if to meet our deadlines, we had to cut scope or take a hit on quality). We came a lot closer to our goal on the 3-4 month schedules than we ever did on an 18 month one.

Sometimes Agile Just Doesn’t Work

At one point, though, we did attempt some 12-18 month projects, and failed miserably: the complexity got the best of us, and it was too easy to allow scope creep to kill us. In the end, because of the complexity of test automation on our product, we got to where we had to settle into a 6-7 month release schedule: we had to build enough into a release to make it worthwhile for all the testing that had to go into it. We did automate some testing, but all the really time-consuming work involved network equipment, such as switches, VPN concentrators, and routers in the lab, and testing different scenarios with different network configurations. Something that even now cannot be fully automated, or at least not at a reasonable cost.

Agile without Automation is Decidedly Not Agile

Along the way, we started building a new product, a SaaS-based firewall management system focused on the cloud. Since we were starting with a clean slate and a very small team, we decided to try to be more Agile-like. We moved to a two-week iteration model, but released about every six weeks to our customers. In hindsight, two things prevented us from realizing the best benefits of Agile:
1. We were completely focused on building new features, and invested no time in test automation.
2. We had a really strong QA lead that was very effective, and quick enough to keep the developers on their toes fixing bugs for a week or two following each iteration.
Okay, so #2 was more an excuse, not really something that prevented us. However, when you’ve got someone like that, it makes it difficult to say: we’re going to slow down and build test automation so that we can build features faster. However, that’s a short-sighted view: sure, you can get away with it early on, but complexity builds and builds to the point where a single person can’t keep up with the testing demands, and then you’ve got to make the hard choice: hire another QA person, or go back and automate as much testing as possible. You take the hit one way or another: but building test automation as you go lets you establish a credible cadence and ensures that you can survive things like a code refactor without investing a lot of time to prove out that you didn’t break anything. That means a lot as your code base gets large: it’s the difference between having to build your product on an increasingly shaky foundation, or allowing you to quickly and cheaply improve your foundation as you go.

I mentioned in my last post my frustration with being expected to predict the future months or even years in advance, and being held accountable for my predictions despite any number of factors changing compared to the original plan. Using shorter iterations did indeed allow us to better predict what we would get done in any given time period, but my management still expected that I would meet a long term roadmap, on a set schedule. That part didn’t fit well with Agile, it was still waterfall-thinking with an Agile wrapper. And, we failed miserably when tracking against those longer-term expectations. In the end, that was the primary reason that I could get behind Agile methodologies: the understanding that methodologies which require and expect a software project to unfold perfectly, without changes over the course of months or even years, is a methodology built to fail.

To compensate for this fact, competent waterfall project managers create a strict change management process to track all the reasons why a software project plan doesn’t come to fruition as everyone expected up front. This prevents surprises for management, and makes sure that everyone knows which decisions were made, and why. Without that list of excuses, any long term project would be considered an abject failure: even small projects experience relatively large numbers of change requests during the course of a long development cycle, and more often than not, in my experience, those changes increase the effort required to complete the project, because they often account for complexity that was not known at the time the system was designed.

Agile admits that things change over time, sometimes drastically, and builds that into the process. It’s easier on the team, the managers, and the leadership of the company: no one gets a false date to commit to a customer, and yet the team can continually make measurable, reliable progress toward a long term goal. And, because you change less between each test cycle (with continuous integration, you run tests each night, or prior to each build), you don’t end up with a hot mess of code wherein you have to spend weeks or months just to get to stability before you can fix all the other new bugs you just introduced.

But is Agile actually more productive?

The following chart shows an estimate of the ongoing code stability during a typical 6 month waterfall project with no continuous integration versus an Agile project. The dip from weeks 2-7 are the coding phase of a waterfall project, where the code is “on the operating table” and completely unstable, having no integration testing done on it. On the Agile side, the code may fall somewhat in terms of overall stability, but continuous integration testing helps prevent massive drops in stability that then requires massive effort to recover from.

This graph is entirely anecdotal, based on my experiences with Agile and Waterfall. What I’ve seen is that over the course of a Waterfall project with no continuous integration, the stability of the code essentially hits zero (the system does not work at all, and may not even compile cleanly) very quickly without continuous integration, and only slowly returns to a high level of stability as system integration testing proceeds. In a project that leverages continuous integration, it’s impossible for this deep dip in stability to occur, because a continuous integration environment would break with this kind of instability. However, some dips do occur, as a result of missing test cases that cannot be caught by the continuous integration environment. Even with the best efforts of developers, it’s impossible to catch every potential case, so stability drops slightly from the maximum in this environment, and fluctuates. This graph changes completely for a waterfall project that leverages continuous integration: it ends up looking more like Agile as far as code stability goes. Does that then provide some evidence, at least, that Agile methodologies are valuable only in so far as how much automation of the testing process they do? That leaves us with two controversial open questions:
1. Which methodology ends up as the more productive one, generating more features for customers per unit time?
2. And, which methodology provides better time to market?
Those aren’t the same question: time to market provides you with something to put in front of customers as early as possible. Generating more features in a given time lets you build more functionality in less time overall: helping you keep up with or pass competitors. Some might argue that waterfall is best on #1, and Agile best on #2. I’ve seen some waterfall teams build deep functionality very quickly, at very high quality.

It’s the Automation, Not the Process

Whichever meets your goals better, one thing is certain: automating things that can be automated is always a good idea. In fact, that mantra has built the entire industrial age. In my opinion, waterfall methodologies can take advantage of continuous integration and test driven development, and yet still retain its very thoughtful, well-documented, very architecture-focused view of the world. The transition to DevOps, whether from an Agile or a waterfall methodology, is pretty easy from there: invest in automation up front, and you’ll go faster over the long term. Next time, I’ll talk about my experiences with DevOps at JumpCloud, and what it enables us to do that we couldn’t do otherwise, and the implications that has for reliability and security.
May 14, 2014
A Winning DevOps Approach for Established Enterprises: Pace-layered
Does your bank offer the ability to deposit a check by taking its picture with your smartphone? When that feature debuted a few years back, I remember thinking it was interesting and convenient enough that if my bank didn’t offer it soon, I’d be willing to switch to one that did.

I bring this example up for two reasons: one, it’s a microcosm of the larger race to innovation that is happening in every company and industry as we speak; and two, it involves banks, which are good touch-points for a discussion about the unique challenges established enterprises face as they attempt to implement DevOps practices in their IT organizations.

We know that DevOps means incorporating best-practice application delivery methods that improve communication, integration and collaboration between development and operations, but what is its ultimate goal? Quite simply, getting high-quality services to market faster than competitors – at a cost that aligns with both IT budgets and business goals. And as my check-depositing story illustrates, if you’re not first in the market, you at least need to be nimble enough to catch up, or you risk losing customers.

For long-time enterprises, this can be easier said than done. In my previous blog about DevOps culture, I used Etsy as an example of an organization that has had great success with its DevOps transformation, but the e-commerce company has been around for less than a decade. Its young IT architecture is apples to the oranges of established enterprises, whose complex environments often comprise a mix of legacy and modern technologies that have been cobbled together over time.

As a result, the DevOps journey is typically longer and more difficult for such organizations – but that doesn’t mean they can afford not to pursue it.

Improving Deployment Practices at Every Infrastructure Layer

When you look at the IT infrastructure of an established enterprise, you normally find three layers of technology. Let’s call the first “systems of record.” These are the legacy transactional systems (often mainframes) that have been with a company since the beginning. They do their jobs well, but their size and inflexibility make them resistant to the type of Agile development that DevOps typically embraces.

The next layer is “systems of differentiation,” or the various hardware, software and middleware components an organization accumulates as it grows, merges with/acquires companies and enters new markets. While these systems are often more Agile-ready than the systems of record, their decoupled/distributed nature can complicate and slow down development and deployment processes.

Finally, there is the “systems of innovation” layer, which is made up of mobile and Web applications and leverages the latest technologies. Because of its light footprint and flexibility, this layer is the most adaptable to DevOps practices. In fact, as you move up the layers from systems of record to systems of innovation, the holy grail of high-quality, fast and low-cost deployment becomes more and more attainable.

What does this mean from a macro perspective? Basically that established enterprises need to take a pace-layered approach, where application delivery occurs at different paces within the different areas of their architectures. The point, though, is to improve in each layer.

So if an organization pushes out a mainframe update once per year, perhaps it can move to a bi-annual schedule (with support from the right people, processes and technology, of course). On the other hand, a Web team that publishes updates once or twice a week could leverage DevOps tactics and technologies to adopt a daily update cadence. Bottom line: Different deployment paces are fine across layers, as long as improvements are occurring across the board.

Kick-starting the Journey with Service Virtualization

Now that we know why the DevOps journey can be difficult for established enterprises, let’s look at how one such organization, namely Union Bank, overcame these challenges and achieved success.

According to an article on Bank Systems & Technology, Union Bank was looking for ways to compete with top-tier banks, so it began “acquiring other institutions, updating legacy systems and launching new front-end applications.” In order to support these efforts, the bank implemented a service virtualization solution, which simulates constrained or unavailable systems across the software development lifecycle (SDLC). This allowed its developers, testers and performance teams to work in parallel for faster delivery and higher application quality and reliability.

Here are some of the results Union Bank has achieved since:
- Cut testing environment setup time from 42 days to 3 days
- Slashed three and a half months from mobile and Web application development time
- Reduced the cost of testing cycles by 65 to 85 percent
Obviously, a service virtualization solution is only one piece of the DevOps puzzle, but Union Bank stands as a powerful example of how a long-standing enterprise can leverage such a tool to be more innovative and nimble in the marketplace.
April 9, 2014