The concept of consumerization of IT, Bring Your Own Everything (BYOx) and self-service technologies has amplified the amount of ‘Shadow IT’ in development shops. Developers can procure, setup, and start using new components, and developer tools in hours. This is a trend that lurks in the dark. C-suite executives can either ban the trend altogether to stay on the safe side, or venture into new waters and embrace innovations that open doors to unforeseen opportunities, efficiencies and productivity standards.
Category: Enterprise DevOps
-
Can your DevOps and APM Initiatives Increase Customer Loyalty?
I’ve reached a point where I can rate most services based on the quality of my digital interactions. So much so in fact, that any human interjection into my daily digital consciousness has become, well, almost quirky.
It was with some surprise that after a prolonged period of digital engagement with my bank, I received a call from a customer service representative. After the initial shock, I was brought back to earth by a pretty standard question:
Based on the service you’ve received recently, how likely is it that you would recommend our company to a friend or colleague? Please rate on a scale of 0 to 10.
Now with a background in marketing, I understand what’s in play here. It’s an attempt to build what’s called a Net Promoter Score (NPS); a simple management tool to gauge the loyalty of consumers. If I respond with a 9 or 10 I’m considered enthusiastic – a promoter. On the other hand if I think the service sucks and score 0 to 6, then I’m a detractor. The metric is calculated by subtracting the percentage of detractors from the percentage of promoters. Yep, management theory doesn’t have to be complex.
NPS as an indicator of business performance is important because most leaders understand they’re competing primarily on the basis of the customer experience. And, since this experience is predominantly based on a journey of linked digital engagement, maintaining the quality of these interactions is vital.
So what has all this to do with Application Performance Management (APM) and DevOps? Well, lots – let me explain.
If we think about digital engagement, there’ll be many points across a customer journey (from app on-boarding to service renewal) that’ll help determine the overall experience and whether or not customers become promoters. I call these “magic moments”; points of interaction where businesses have an opportunity to deliver an experience that transcends beyond the purely transactional into a more powerful emotional connection. It’s vital therefore that these moments become the focal points for our APM strategies – only this requires some new thinking. Here’s my take:
Monitor the Magic – IT teams must identify all the key magic moments (especially those that have the highest correlation to customer loyalty) and implement automated performance collection and measurement. Some of these may form part of a linked value chain (e.g. an airline flight experience = seat selection + mobile boarding pass + baggage collection + loyalty rewards), or be services designed to drive a recurring revenue model (e.g. free shipping / discount options), or even be delivered by partners through API’s. Whatever the model, APM must be capable of providing a unified end-user view into every contextual, situational or transactional interaction – mobile app and web-based – because it’s here that loyalty is determined and promoters are formed.
Make Metrics Matter – when a prospect navigates to your application (perhaps via some promotional web-advertising) that’s great, but it’s not yet magic – that is until the visit converts into a sale. It’s at these points that we should be connecting our “listening” strategy (via APM) to the customer and determining effectiveness in a business context. This can be tricky because traditional (but still important) metrics like MTTR are introspective, whereas what’s really required are external customer-centric indicators. To address this, DevOps practitioners should consider applying newer APM analytical methods and instrumentation in order to learn the relationships between service elements (e.g. web site sign-on / authentication time, response times) and customer experience – using the intelligence gained to drive improvements where they’re most effective – yep, you guessed it – at those “magic moments”.
Share the “Profits” – an advanced experience-based APM strategy must also include working with groups who can profit from rich information insights. From a DevOps perspective, this will involve development leveraging the data as the operational feedback needed to steer both functional and non-functional improvements – for example, mobile app usage and behavior analytics are critical early indicators of functional usability. But given the businesses reliance on digital services, many groups beyond IT will also profit. For example, if it’s reliably determined (and presented) that application performance is a key driver of Net Promoter (or any other business metric), which in turn is linked to improved financial outcomes, then this could serve as the basis upon which to prioritize future investment.
Instrument Culture – aligning with metrics like NPS is valuable because it helps shape collaborative, customer-centric behaviors. However, this requires tools that can embed these behaviors into everyday operational practices. Improving customer experience is a shared goal, so APM processes must facilitate the feedback of intelligence between Dev and Ops, while also meeting the needs of every function (Developers, Application admins, Level 1 and 2 support etc.) within each group – especially in context of the role each plays in improving customer experience.
As companies increasingly engage customers via digital channels it’s a given that IT will be measured on how it increases customer loyalty. While business metrics sounding like NPS may at first seem foreign, they present us with a unique opportunity to demonstrate what DevOps and APM is really all about – winning more customers and getting them to stick with you.
-
A Three Step Roadmap for Enterprise DevOps
The extremes of application release velocity are converging in the enterprise. Whereas Fortune 500 enterprises may have historically taken months to years for each release cycle, they are exploring new approaches from new age Internet technology firms that release new versions of an application several times in a day.
According to a recent report on DevOps adoption, 86% of IT and business decision makers have either implemented DevOps or will roll out DevOps initiatives within the next 3 years. There is a strong appetite for enterprise DevOps, but where do you start?
For DevOps projects, Ops will need to shift from a systems management approach to a programmatic approach. Ops should work with Dev to figure out the automation required to provision and deploy the application. Here is a three step roadmap for enterprise DevOps for those organizations looking to roll out successful DevOps initiatives:
Step 1: Select The Right Application Candidates.
Not every enterprise application is a right candidate for DevOps. Traditional applications are not architected for DevOps velocity, whereas enterprise applications designed with agile and iterative techniques are excellent candidates for DevOps. DevOps may be an overkill for applications where the underlying business process rarely changes. Dev and IT Ops should choose one or two applications for DevOps style implementation.
Step 2: Manage Your Application Lifecycle With Automation.
Provision applications as part of the build process itself. During deployment, Ops should be able to execute a piece of code to identify application monitoring needs. Your automation framework should manage application health without manual intervention.
- Can you fix application issues using runbooks?
- Do you have scripts for scaling, maintaining, and load balancing the application?
- How do you plan for new releases?
- How do you upgrade the application?
Treat your infrastructure as code. Application configuration, deployment, and maintenance all need to be part of an automation framework. Ops should build concurrency with the application. You should be able to configure, deploy, and bring the entire application live within no time.
Step 3: Embrace Compliance Driven Operations.
Your IT Ops automation framework should deliver the right visibility to support more frequent releases without complex manual processes.
- Are you able to automate monitoring of the infrastructure for delivering the best end user experience?
- How do you ensure that the right infrastructure resources are available for running the application?
- Can you keep your infrastructure current by maintaining the right patch levels?
Enterprises can modernize application delivery pipelines by selecting the right application candidates, managing application lifecycles with automation, and embracing compliance driven operations. While you might never deploy code 23,000 times a day like Amazon.com, the Enterprise DevOps framework will help you release applications faster, with higher quality and lesser risk of deployment failure.
About the Author / Varma Kunaparaju
Varma Kunaparaju is CTO at Vistara and a seasoned technologist with over 20 years of experience leading engineering teams to deliver enterprise software. Reach him on twitter at: @varma1 -
The right answer was “process”
The question is “why is there no one-size-fits-all automation and orchestration solution”?
Automation, a component critical for continuous deployment particularly as it extends beyond app operations and into the broader environment to encompass network and security operations, too, continues to be a custom build kind of thing. There’s no single automation tool, framework or pattern that can be generally applied to a significant number of environments. Mi casa es no su casa, after all. That’s because environments don’t actually look like power point slides; your environment is not the simplistically diagrammed firewall –> load balancer –> web server typically found in decks extolling the virtue of one solution over another. They are far more complex, comprising not just multiple tiers of interdependent and interconnected apps and services, but multiple chains of infrastructure services spanning the gamut of security, availability, mobility, performance and access control. The specific flow through and between these services is unique. Indeed, the intimate relationship between network latency and app performance is well known, and thus the navigation through the critical yet complex site of services required to deliver an application could easily be considered a competitive advantage if it’s able to keep performance under the 5 second toleration limit on response time apparently now the standard for consumer satisfaction.
So it’s no surprise that automation and orchestration, responsible for automatically setting up the triggers and routes required for application traffic to correctly traverse the right set of services for optimal performance, would not be a simple, turn-key solution. One size does fit all. It must, like its business process focused counterpart – BPM – be customizable and easily changeable.
BPM? Are you seriously?
I’m dead seriously. BPM – business process management – has been around far longer than automation and orchestration of IT systems and its focus is, as its name implies, on process. Automation is included, but it is the orchestration of business processes that often change in the face of variable business conditions that makes it an invaluable tool for optimizing business and a source of competitive advantage. “The ability to change quickly is essential. Businesses change their key processes 4-7 times per year. The driver for change can be internal or external. New opportunities can arise. New partners or customers need you to support a different way of doing business. Federal or international regulations can require you to change your processes.” (Making the Case for BPM, Jim Rudden)
Consider for a moment this data from a 2012 Bearing Point survey on the goals and impact on culture:
Top BPM goals are increased efficiency (93%), increased transparency (90%) and the realization of standardization potentials (85%). A number of BPM goals have an impact on corporate culture, e.g. increased transparency (rank 2), increased customer orientation (rank 5), disclosure of weaknesses/ bottlenecks (rank 6) and the reduction of mistakes/ error rates (rank 9).
– Bearing Point Business Process Management Survey 2012
If many of these goals and impacts sound familiar to you, they should. Because what DevOps ultimately needs to focus on is exactly the same as the focus of BPM – process. In an application world, these processes require applications and services. IT must be able to support these process changes in the application and service infrastructure.
BPM (Business Process Management) has known this for years, and it is the orchestration of the process – the workflow – from task to task, from approver to doer that generates the efficiency gains achieved with that technology.
Indeed, adopters of DevOps who focus on the “big” picture would do well to consider the similarities to BPM which touts as benefits such things as:
- Process quality improvements
- Continuous process improvement
- Reduces costs
- Improves business agility
It would be difficult if presented with that list of benefits with no attribution to determine whether it was referring to DevOps or BPM. That’s because the operationalization of business through process orchestration is not all that different from the operationalization of IT through process orchestration.
And processes are not, nor should they be, turn key and one-size fits all. They should be tailored to address the needs of a specific operational or business goal, i.e. deploy an application or secure an order from a customer, based on the systems in use to achieve those goals. BPM shines where it is most able to address bottlenecks and obstacles caused by handoffs between systems and teams. DevOps will shine where it is most able to address bottlenecks and obstacles caused by handoffs between systems and teams. It isn’t about racing through a given task with automation. Speed isn’t efficiency and, if you do the math, a focus on automation without similar focus on process can lead to even greater inefficiency.And that’s why there is no one size fits all automation and orchestration tooling; because the goal isn’t automating tasks, it’s orchestrating process. Coordinating workflows. Ferreting out inefficiencies.
Orchestration is the glue that binds automated tasks together. For example, recently a missed route update caused some unexpected down time during what was considered a routine* decommissioning of gear. A missed route update. In other words, someone forgot. That task might have been automated, it might have just needed to be kicked off. But it wasn’t.
That’s the kind of thing that would likely not happen if the process was governed by some form of orchestration; if there was a clear list of what needed to be updated or changed based on the removal of that particular component. If orchestration was involved to provide visibility and oversight of the operational processes needed to perform such a routine task. This does not mean the tasks should have been automated; in fact automation is not strictly required to achieve the visibility and oversight necessary because processes are about workflows and hand offs, not specific tasks completed during the steps that comprise a process.
Which leads us right back to Business Process Management. Take a deeper dive into the aforementioned Bearing Point survey and you’ll find more similarities with DevOps:
- Crucial elements for sustainable BPM are clearly defined roles (95%), top management support (92%) and a clear mandate (92%)
- Typical implementation barriers such as the lack of resources (53%), sponsorship (32%) and experience (25%) need to be identified early and addressed in a structured way
- BPM as a continuous (=sustainable) approach is already adopted by 59% of companies, coordinated and implemented either centrally or locally
- Involvement of stakeholders (90%), process oriented thinking (90%), organizational institutionalization (82%) and a well-communicated concept (75%) are elements of communication and change management which must accompany the establishment of sustainable BPM
That’s because both are ultimately about process, with automation being only one of the tools in the tool chest that support implementation of a continuous approach to enabling agility in the process layer. Indeed, about 38% of the companies in the survey use software to support their BPM initiatives while another 37% do not. A spreadsheet with the right Six Sigma calculations can be as powerful as any automation framework, toolset or API in identifying the dead space between tasks that introduce inefficiencies and delay application deployments.
BPM has, like DevOps, brought tangible, measurable results to the table. Bearing Point found that almost three quarters of the surveyed companies (72%) were able to achieve measurable benefits through BPM such as process lead time (72%), process cost (68%) and error rate improvements (67%).
So rather than focusing too much on automation, adopters and proponents of DevOps would do well to up-level the conversation and start looking at process orchestration and improvement as the goal. The successes BPM has had on the business side of the equation tell us that doing so will provide a better path to achieving success with DevOps in the operational side of the equation.
* I’d argue that in today’s environment of hyper-sensitivity to availability that nothing that touches a service or gear in the critical path of an application is “routine” but, hey, it is what it is.
-
The ROI of Enterprise DevOps
Ever since the 1996 film Jerry Maguire, I can never think about any technology development without hearing Cuba Gooding Jr as professional footballer Rod Tidwell screaming out, “Show me the money!” So I was not surprised when the CIO of a large enterprise recently asked me the enterprise exec version of that famous line:
Q. Where is the ROI in Enterprise DevOps?
Fortunately there is plenty of evidence to show enterprise executives exactly where to look when trying to determine the ROI of DevOps.
“Show me the money!”
Here on devopsy.kinsta.cloud, Rajat Bhargava has posted on talking to your CFO about DevOps, outlining four key areas of financial impact: generating cash, reducing expenses, enabling growth, long-term stability. I also had some ideas about financial aspects of getting executive support for DevOps in a great article by Erica Chickowski. And site editor Alan Shimel has posted some excellent insight on vendor-sponsored research from IDC on the cost-benefit of downtime avoidance.
“Show me the money!”
Not enough? Well, we also have several years of research on ROI factors associated with DevOps adoption in both small businesses and large enterprises. The company I work for, CA Technologies, has commissioned independent third party research into enterprise DevOps adoption since 2013; Puppetlabs has run similar research [PDF] with good data on all sized organizations, but especially mid-sized businesses.
You can see in the chart below the areas where benefit accrues through the adoption of DevOps (taken from CA Technologies/Vanson Bourne research released in November 2014):
Still need more? No problem! There is also good data on how much of a return you can expect from DevOps. Again, the following chart is taken from the same CA Technologies/Vanson Bourne research:
Figure 2 Respondents report seeing double-digit improvements in key technology and business areas. Total varies by benefit. Figure 1 & 2 Source: Vanson Bourne/CA Technologies 2014
“Show me the money!”
Okay, I get it – you still want more, and you want a straight answer – a single, simple heuristic to work with. In talking with many enterprise CxOs, faster application delivery is a pressing goal, so just this one factor can act as a key ROI driver. At the Continuous Delivery (CoDe) conference in Copenhagen in October, Jan Bosch (Professor of Software Engineering at Chalmers University of Technology) showed very elegantly how faster delivery acts to reduce (i.e. shorten) the ROI period.
His theory is based on the premise that for many (most? all?) businesses, time to revenue is dictated substantially by product release cycles. Maximum revenue accrues around product release, then tails off before spiking again for the next new product release. For example, a smartphone manufacturer will get revenue in 6-month peak and trough cycle as they release a new device twice a year; a software vendor will get a revenue peak with a new game release, which tails off until new DLC is released or a new game hits the stores.
Because DevOps can accelerate business cycles, Jan theorized, business can bring in revenue faster, which in turn makes a DevOps-based business more profitable over the mid- to long-term. This is especially critical in the modern application economy where almost every business is a software business.
So, for example, if DevOps allows a business to release twice as fast, they can effectively double their annual revenue, as seen below.
Figure 3 Proportional revenue acceleration from doubling product release frequency with DevOps
The impact is exaggerated for this illustration, but based on actual data (see above), a 20% increase in annual revenue would be on par with enterprise averages.
“Show me the … costs!?”
Of course, any discussion of ROI must realistically reflect costs too. Adopting DevOps in a large enterprise is not free – it requires investment in people, process, and technology. On the people side, even just a small cultural change often requires new activities and other behavioral shaping. Process change requires training and often impacts short-term revenues. Technology investment includes procurement of automation and measurement solutions – key components of a CAMS (Culture, Automation, Measurement, and Sharing) approach to DevOps, especially in a larger enterprise.
Many of the investment factors can be seen in the following chart, again using CA Technologies/Vanson Bourne research:
Figure 4 Which of the following is your organization likely to invest in over the next year as part of your implementation of a DevOps methodology? Total = 499 ITDMs who have already or plan to implement DevOps in the next two years. Source: Vanson Bourne/CA Technologies 2014
“Show me … my own ROI!”
These ROI factors are of course a combination of averages, research data points, and anecdotes. As with all generalizations, when it comes to a specific enterprise your mileage may vary. However, this is certainly a good starting point when building a business case to prove the expected ROI for your organization.
You may find other areas for ROI too – upstream in the PMO, DBA, or downstream in NetSec, training, documentation. If you have additional data points on finding the ROI for DevOps, I would love to hear about it. Add your ideas here in the comments, or connect with me and/or devopsy.kinsta.cloud on Twitter – at @AndiMann and @DevOpsDotCom respectively.
-
Digital Transformation with Agile and DevOps – Something “Wicked” This Way Comes
Our world is full of wicked problems, like climate change, urban planning and social injustice. They’re called wicked because they’re hard to recognize, define and solve. Furthermore, complex interdependencies mean solving one aspect will just expose or create other problems.
Today’s digital transformation imperatives are creating wicked problems for IT. So wicked in fact, they make managing technology in the traditional sense look like child’s play.
Suddenly, businesses are faced with complex dilemmas. How can they compete in a world where their customers place less emphasis on physical products and more on the total experience? How can we adjust marketing from driving consumer behavior to responding to it? How can they adapt business models in context of the customers’ world? – a world that’s increasingly dependent and driven by digital technology.
The short answer is of course software, but the problem is our approach to delivering it just isn’t suitable in a wicked business world – why? Well its simple – we keep working on the assumption that we understand the problems we’re trying to solve, when in fact we don’t understand the problems of our customers at all – that is, until we have a solution.
Head scratching stuff huh, but consider this –
Did you know you had an entertainment problem before Apple gave you iTunes? Was booking hotel accommodation such a pain before AirBNB introduced the whacky notion of people renting out their own rooms?
So faced with such wickedness what can we do? Well, we can start by accepting our current IT failings, develop some wickedness of our own, and mitigate against what can derail your efforts.
Small is Beautiful … really it is!
In the past we’ve tried to solve business problems with what I call IT “bigness” – big iron, big software applications, big teams, big projects, and big expectations – oh, and let’s not forget big funding. The results, however, haven’t set the world on fire have they? – Big IT white elephants, big budget blowouts and big executive fallouts (literally and through the revolving door).
But surely we can’t lay the blame squarely on IT. After all, traditional models of software design and project management where we diligently survey the client’s needs are baked in common sense. Add to this the rigor and procedure around sign-offs, budgeting, design, development, testing and deployment – doesn’t this guarantees success?
Of course it doesn’t. Apart from the big fat fails, we also end up with what I call a big “sucking culture”– your software sucks, IT operations sucks, this job sucks, our business sucks – we’ve all been there, haven’t we?
All of this isn’t due to bad people and technology. It’s because we work on the premise that the business understands what it wants – it doesn’t. Much better then to put aside “bigness” and start listening (maybe insisting) that the business first lays all the wicked cards on the table, like – what are our markets and how is this changing? What are our target demographics? What are our barriers to entry?
After all this ‘warts and all’ revelation, it’s time to get busy. Now, with a shared understanding, we can start quickly prototyping elements that perhaps only solve one small problem. Then we continuously develop, test and demo, knowing and accepting other problems will surface that need to be solved. And, as we know from agile development, it’s all a case of ‘rinse-and-repeat’ – hopefully achieving a state of nirvana – giving your customers software answers to problems they don’t know they have. Problems that lead to increased market share, revenue and profitability.
So is a liberal scoop of Agile or its younger DevOps cousin the answer? Not necessarily — that is until some essential toppings and flavors are added to my somewhat vanilla perspective:
- Executive Flag Waving – try selling a methodology or philosophy based on the premise you’re trying to solve problems you don’t yet understand. Good luck with that. If executives simply want faster, better, cheaper, it’ll be much easier (and safer) to hold back grandiose talk about experimentation, “failing fast” and continuous improvement. Understand, however, that without tangible commitment, half-assed agile or DevOps is like a fast melting ice-cream cone – messy and hard to handle.
- Behavioral Hacking – even with executive champions, persistent and rigid organizational behaviors will need to change. Just hiring lots of agile coaches or mandating DevOps won’t work until there’s willingness for people to be engaged around new ideas and fresh thinking. With that in mind, be prepared to do some heavy lifting – socializing and selling new concepts, but also looking for opportunities to show how the smallest of changes (e.g. team refactoring, incentives) lead to improvement.
- Show me the Money! – So how to you fund agile and DevOps? It’s going to be tricky to cost something exactly when you don’t know the exact outcome of a new project, but businesses may initially insist on this to minimize risk. My advice here is to tread carefully. Stakeholders should be more interested in ROI than coming in under budget, so fixed-cost might not be actually what want anyway. Your job therefore is to convince them by continuously demonstrating ROI – not just from a single project, but also by the accrued savings that agile and DevOps deliver the whole organization when they systematically reduce technical debt and waste.
Developing the wicked traits I’ve described doesn’t mean IT has suddenly become an evil entity. Nope, it’s just a case of tapping into fantastic business opportunities in a constantly changing digital world. Agile and DevOps can help you solve and even create some wicked problems for your customers, but that involves confronting and challenging entrenched thinking as it relates to just about everything.
-
Q and A with Carmen DeArdo: DevOps at Nationwide
Just before the IBM Interconnect show in Las Vegas last week, we had a chance to speak with Nationwide’s Carmen DeArdo and talk about some of the challenges he has faced at Nationwide as DeArdo and his team work to bring DevOps and Agile principles to their delivery value stream.
DeArdo is no stranger here at devopsy.kinsta.cloud, and recently shared his column Nationwide banks on DevOps to drive towards Continuous Delivery. At Nationwide, DeArdo is the director of application development tools and technologies. DeArdo is responsible for driving continuous delivery utilizing DevOps, Lean and Agile techniques across mobile, distributed, mainframe and other technologies.
Here, we interview DeArdo on the importance of transparency into DevOps processes and the importance of attaining efficiencies throughout the delivery value stream.
devopsy.kinsta.cloud: Talk to us about how, as you optimized certain aspects of your delivery value stream, you found wait states in other areas of the chain created delivery backlogs?
DeArdo: If you actually look at the entire delivery value stream, we started out optimizing the middle because we started with Agile design, development and acceptance testing. We gained a lot of benefit out of that. If you take a step back and think about your entire delivery stream, and your continuous delivery efforts, what you realize is that on the upfront process you end up with wait states and lack of visibility flow at that end.
What is happing is that as your teams get better, they actually run out of things to do. There’s nothing for them to pull forward anymore.
Then, on the downstream side of delivery, you still have the issues through your path to production – especially if you have managed test environments. You still have manual processes, manual testing steps in deployment. Even if you’ve automated some of them, you still have manual processes. And you end up moving at the speed of the lowest common denominator. What you find is that you’re going to get diminishing returns now unless you start to address those wait states or contention points on either side of design, develop, and test.
These inefficiencies, on the planning side, start to bleed back into the business. Many times your business areas still have a fixed mindset in terms of their budget and yearly cycles. They’re not used to determining how to continuously feed high priority work into the backlog of your Agile teams. And then on the other side of the pipeline, how can you take what’s being produced by these teams and move that more quickly into your path to deployment?
devopsy.kinsta.cloud: That’s interesting. Your challenge is taking your continuous deployment and continuous improvement mindset and then feathering that into the annual and quarterly planning mindset of the business?
DeArdo: On one side, what you’d like to have is more of a continuous planning, continuous flow so you’re not dumping a lot of stuff in a bursty way onto your teams. They’re going from wait state to being overburdened. You don’t have that continuous flow. And then on the downstream side, how do you start to automate some of the things and get rid of some of those manual practices? We’re very good at deploying to our continuous integration and deployment to our dev environments.
But then beyond that, things tend to get bogged down. We sort of go slow at the beginning and slow at the end and go very fast in the middle. Well, if you’re going to improve your overall speed, you need to look at those beginning and end spots and say, “How can I try to address those?”
devopsy.kinsta.cloud: To improve throughput throughout, it sounds like you need more feedback, more planning throughout the year with the business.
DeArdo: There are a couple of things we need to do, and part of it is around driving more towards continuous planning. The other side is about of having an automated deployment capability. Giving the business more capability, it goes faster if it makes sense from a risk and value capability perspective.
We’re actually focusing more on that latter part right now. We are building this automated delivery pipeline, which starts with release planning and then goes all the way through deployment.
devopsy.kinsta.cloud: What are some of the high-level challenges that you’ve had optimizing that last mile phase?
DeArdo: I think one of the biggest challenges, although people like to talk about tools, is is around the variance in your own processes. If you have too much variance in your own process, then you don’t want to instantiate that into a tool because all you’re taking is a bad current state and making it a permanent current state.
So we have meetings with vendors. The last one we had, we’re pretty far along this path, but the last one we had, we probably spent 90% of the time just talking about the variance in our own process and the fact that it’s not one-size-fits-all, but you have to have a small number of patterns that you’re trying to instantiate and implement. You have to be able to tell the difference between business value-added variance and waste variance.
I sometimes quote “The Wizard of Oz.” It’s like are you a good witch or a bad witch? Are you good variance or are you bad variance? Everybody wants to think what they’re doing differently has value, but in reality, most of it is arbitrary. There might have been a good reason for it at one point, but at this point, there is no value add in having that variance.
One of the variances we have is our path to production: what environments, what phases and things like that. One pattern might be an urgent need to get a business capability out very quickly. So I have a certain environment, a certain way to get certification in an expedited way. It’s not about going through an environment. It’s about proving that you have the quality that you need to deploy.
That one pattern might have a lot of automated testings run in one environment that you can get to very quickly to certify the quality and then release it. Another pattern might be around a record keeping system which is managing financials. You just have things that you have to go through. The system will have dependencies. So you’re going to go through a more elongated path to production.
That’s just an example of two patterns you would use based on a technology and a business need, not just because I’m in Organization A versus Organization B or Organization C. I don’t care what organization you are, if it fits this pattern and this technology, if it fits this pattern, use that and then if it fits this other one, use that. Those are good variances based on the fact that there’s a business need. There’s a technology driver that defines how you’re going to deploy.
devopsy.kinsta.cloud: That is an excellent explanation. I think many people get caught up in their “one right way” of doing things and have a tough time looking objectively at alternative ways.
Let’s talk about your “hidden factory” metaphor. Is the “hidden factory” a metaphor for a lack of transparency into different areas of IT?
DeArdo: Right, and the fact that you really don’t have a model. You have views. One of the things I point out is that you have a business development view, you have a project planning view, and you have release metrics, project metrics, and you have a release view. You have a path to production. None of that is visible. Some of it is hard to find. Some of it is visible if you’re in the tool of choice. But there is no visibility and no linkage or integration of that as you move from putting in your business needs at one end and seeing the code moving into production at the other end.
DevOps is about shining the lights on that. My view is you can’t manage what you can’t see. Another example sometimes I use is a basketball coach. Let’s assume you’re a basketball coach but you’re not allowed to watch the games. What you get after every game is a score sheet. It has the score. It has maybe your shooting percentage and turnovers, assists, but that’s it. Now you have to practice for the next game to work on improving something. Well, how do you know what to improve? So let’s say your shooting percentage was low, assuming you have enough metrics to know what low is.
Maybe your shooting percentage is bad. Maybe you think, “I’m just going to practice shooting.” Well, maybe that’s not it. Maybe the wrong people took the shots. Maybe you took contested shots, bad shots. There are a number of reasons you can have a bad shooting percentage, none of which have to do with the fact that you have bad shooters.
But that’s sort of the way we manage IT today. We’re asking executives to manage things they can’t see based on a set of metrics, but they don’t really watch the game.
-
How Deploy Frequency Impacts Infrastructure Stability
I recently took some time out from reading the Internet to read a very specific piece of research I found on the Internet from IDC – its “DevOps and the Cost of Downtime” to be exact. Sponsored by AppDynamics, this fascinating read included a variety of statistics that are helpful in understanding the impact of (or a lack of) DevOps on organizations given certain financial and competitive impacts.
I won’t bore you by repeating what you can so easily read yourself, but I have pulled out a few key stats that are worthy of further investigation – that of the failure of stability on the business and the predicted increase in deployment frequency.
- The average hourly cost of an infrastructure failure is $100,000.
- The average hourly cost of a critical application failure is $500,000 – $1M.
- The average number of deployments per month is expected to double in 2 years.
Interestingly, that prediction on deployment frequency is a good thing and carries with it the possibility of making sure the costs associated with the other two rarely (if ever) happen.
We’ve already gotten to a point where “speed” with respect to DevOps is effectively equivalent to “time to market”. But there’s another measure of “speed” that’s just as critical, particularly when we apply it to maintaining stability in the infrastructure. That measure is MTTR (Mean Time to Resolution). This measure is very important when you consider that 60% of production failures are caused by human error or lack of automation according to research from ZeroTurnaround. A goal of DevOps, then, should be to reduce that number as much as possible and introduce more stability into the environment. Frequency of deployment can actually be of assistance in realizing that goal.
So how does frequency of deployments help do that?
First let us consider the number of possible points of failure that might cause downtime – both in the application and core network infrastructure. Then let us consider that each of those possible points of failure also have multiple potential causes of failure. A single service in the logical critical data path within the core service infrastructure, for example, can easily cause “downtime” that, in the case of IDC, would be characterized as “infrastructure failure” and run a tab of $100K per hour.If that application is a critical business application, the tab runs much higher per hour.
Obviously the speed with which we restore stability (service) in such an event is a factor in the overall impact to the business.
This is where DevOps comes in and where operationalization can play a significant role in mitigating the impact of downtime by aiding in faster time to resolution.
Consider that one of the most difficult tasks in the network is not provisioning or configuration, but troubleshooting. To be fair, troubleshooting in app dev, too, is one of the more time-consuming and frustrating tasks, consuming 75% of a developer’s time (according to research from InitialState). That’s close to the 65% of network engineers who are dedicated to “break and fix” tasks, who spend their time troubleshooting network issues up and down and all along the network stack. Needless to say, if you just deployed ten or fifteen different configuration changes splayed across the core and per-app service infrastructure and suddenly something is “wrong”, it’s going to take some time to figure out exactly which configuration change caused the problem in the first place.
The relationship between number of changes and frequency of deployment is no secret; if you’re deploying changes more frequently then it’s likely there are fewer of them. With an average sprint cycle of 3-4 months, the amount of change is logically not going to exceed that of a cycle that takes 12-18 months. The same holds true for the infrastructure; more frequent deployments should mean fewer changes).
Logic (and math) tells us that the fewer lines of code and configuration that have changed, the less time it should take to identify the culprit should something go wrong, thereby hastening the time to resolution. Assuming that an agile, frequent deploy rate results in fewer changes to the environment it stands to reason that this approach should also reduce both the number of failures and the time to resolve any that do crop up. Frequency is a significant factor in reducing failures by increasing its overall stability.
This is backed up anecdotally by Puppet Labs findings in its “State of DevOps” in which organizations adopting a DevOps approach reported 50% fewer failures and a 12x improvement in time to resolution. That’s stability at work and it’s driven in part by the frequency of deployments and its positive impact on MTTR.
So let’s apply this approach (and logic) to the rest of the services along the critical data path in the data center infrastructure, i.e. apply it to application services like caching, web application security, load balancing, identity management and application access control. Automation to reduce misconfiguration caused by human error and smaller, more frequent changes that enable a faster time to identity the inevitable issues that will be introduced. No methodology or approach is going to completely eliminate errors, after all. There’s a limit to how perfect we can get and it’s pretty much a Zeno’s Paradox kind of problem. While we keep solving current issues, more config and code is being deployed that is statistically likely to contain yet another error, and so and so forth.
But the goal is not necessarily to eliminate all errors (though that’d be nice, it’s simply not practical and it’s the primary reason why our goals for uptime are called “5 9s” and not a “1 with two zeros”) it’s to reduce the number of errors and their impact on downtime. And that is something we can achieve without running afoul of Zeno and his paradox.
Smaller, more frequent deployments can result in a more stable environment in which downtime is reduced to a level consistent with acceptable risk.
DevOps, whether applied directly to infrastructure or not, will play a role in improving its stability which in turn has a direct and very real impact on the business by eliminating (some) downtime thanks to more frequent deployments.
-
Streamlining no matter how many speeds your IT operates at
IBM InterConnect 2015 took over the Mandalay Bay and MGM Grand conference centers this week with an estimated audience of more than 20,000 attendees. The event is filled with DevOps evangelists, hybrid cloud experts, and IBM customers converging to share information in presentations and panel discussions.
On Monday I had an opportunity to attend one such panel discussion titled Mobile to Mainframe Experts: DevOps Best Practices for Systems of Engagement and Systems of Record. Yeah, the title doesn’t exactly “pop” but I didn’t let that dissuade me. The panel included a number of respected names in DevOps such as Carmen DeArdo from Nationwide Insurance, and Rosalind Radcliffe and Sanjeev Sharma from IBM.
In spite of the title of the panel discussion most of the conversation revolved around the concept of two-speed IT. In any organization—no matter how streamlined—there are going to be different processes or teams that function at different speeds. Applying the Pareto Principle, even if the organization figured out a way to operate more efficiently there would still be 20 percent doing 80 percent of the work and 80 percent doing the other 20. It would just be a matter of degree.
The problem is that if the speeds are too different problems arise. First, it’s just a waste of resources to have skilled workers sitting idle waiting for a different group or process to catch up. Second, those workers will often get antsy and go rogue—writing code and introducing features that weren’t part of the initial requirements and aren’t part of the planned testing. Now you’re just adding chaos that disrupts the workflow even further.
According to the esteemed panel the goal is to find ways for those processes and teams to function at their own pace without it impeding the overall efficiency —hence two-speed IT. Rather than trying to force a square peg into a round hole, or getting frustrated that the square peg can’t just be round in the first place, you have to find ways to maximize efficiency with the teams and processes you have.
Carmen DeArdo from Nationwide Insurance applies another term as well: variable-speed DevOps. In essence two-speed IT frames the challenge, and variable-speed DevOps is the set of processes and tools organizations can utilize to address the challenge. Removing some of the friction between different groups and enabling the organization to function more effectively is one of the primary benefits of DevOps.
When I spoke to Rosalind a couple weeks ago she stressed repeatedly that the foundation of effective DevOps is breaking down silos and finding common ground. That doesn’t mean all the teams have to be restructured under one organizational unit, or even that everyone has to adopt the exact same tools and processes. It means figuring out how they’re different and how they’re alike, and determining ways to make the most effective use of the common ground where their similarities overlap. It provides a starting point to get the ball rolling, and then things can mature and evolve from there.
In the session at IBM InterConnect Rosalind agreed that variable-speed DevOps is a great approach, but cautioned attendees regarding how it’s used. She told the audience that variable-speed IT should be driven by business needs, not by tools or processes. In other words, you have to stay focused on the business requirements and the need to achieve business goals and then find ways to do that. You can’t try to shrink or adapt the business requirements and goals to fit the processes and tools you already have.
DevOps tools and processes can—and in most cases should—speed processes and teams up. One of the hallmarks of DevOps is speed. Even as organizations embrace DevOps and introduce more automation, there will still be different teams and processes functioning at different speeds—and a need to implement variable-speed DevOps to enable those different speeds to exist in harmony.
-
The Ultimate Endpoint of Enterprise DevOps
There’s a reason we so often use the word “journey” when we talk about DevOps – it is a critical aspect to a practical approach to DevOps. While we focus a lot on culture in the DevOps community, as I have previously said, culture is an output, not an input; it is the destination, not the journey. By contrast, the set of ‘best-practice’ application delivery methods that improve communication, collaboration, and integration between development and operations are the ongoing journey that drives this transformation for continual growth and improvement, and helps us reach the destination.
Which leads to the question for this next article in the Enterprise DevOps Q&A:
Q. What is the Ultimate Endpoint of Enterprise DevOps?
My initial reaction when asked this was to wonder whether the DevOps journey even has a defined endpoint. While many might say ‘culture change’ is the destination, many others would say there is no ultimate endpoint.
However, for enterprises especially, with their rigorous set of annual, quarterly, and even monthly goals and metrics (and associated performance reviews, bonuses, stack ranking, etc.), it is difficult to absorb such boundless activity. Larger organizations often need, if not one specific endpoint, at least multiple SMART waypoints along the journey.
If we follow through on the journey analogy, using CAMS (Culture, Automation, Measurement, and Sharing) as a rough roadmap, we may start trying to change the culture by removing structural barriers to collaboration, or attempt to shape new sharing behaviors. For large enterprises with goal-oriented metrics, automation is happening further down the road – which makes it a worthwhile next waypoint for organizations that have started their journey but are asking “What’s next?”
Waypoint 1: Release Automation
Release automation is a key technology that integrates dev, test and ops at that precarious gap between them – code release. Release automation solutions work by modeling components (e.g., images, packages, source code, configurations, etc.) of the development, test, pre-prod and production environments for each stage in the application release life cycle, and then fully automating the deployment and promotion of changes across those environments.
This is important in DevOps because in enterprises where release automation does not occur, dev and ops (and/or test, QA, staging, acceptance, etc.) staff have to perform these release steps manually. This not only limits their productivity and pushes out application delivery dates, but is also highly error prone.
Here’s an example from a recent case study by Enterprise Management Associates (disclaimer: I worked for EMA a long time ago and think they are excellent people and very strong analysts). During the Christmas season, a consumer electronics company’s website failed to perform as large numbers of customers logged on to configure and provision their new home entertainment systems. The company deployed a cross-functional “WebOps” group to investigate the problem, which was occurring within their software deployment processes. In short, a lack of standardization made it possible for code, configuration or process errors introduced during deployment to create flawed execution environments.
The company responded by implementing a release automation solution that can execute a given workflow and configurations automatically, so each environment mirrors the last and errors are reduced or eliminated. As a result, the company has moved from a manual deployment method, where IT staff had to execute multiple scripts in sequence by hand, to an automated one in which IT can “deploy with a high state of reliability in 15 minutes with few adverse after-effects.” This allowed the company to standardize its deployment processes, improve the consistency and reliability of its website and ultimately meet the needs of its customers and vendor partners.
Waypoint 2: Continuous Delivery
Automating these handoffs is a great waypoint for enterprises to measure their journey. It is also a great starting point to the concept of continuous delivery – end-to-end application delivery that combines an ecosystem of people, processes and technologies into an automated, iterative, ‘assembly line’. In such a scenario, release operations build on release automation to integrate and orchestrate promotion across environments, enabling predictable, monitored and measurable continuous delivery from development to production.
At a high level, development involves four phases: planning, building, testing and releasing an application. When an organization achieves continuous delivery, every technology and people process that can be automated between these phases is, so for example, development data from a release automation tool could be fed into a portfolio management solution, which would help automate approvals and future planning phases. In addition, if you used shared services in composite architectures, each team didn’t build the same thing over and over. You built it once and all shared it.
Then, as iterations occur and the portfolio management tool amasses more and more data, IT leaders will see first-hand how different methods led to better results (e.g., when this technology was used, the project was delivered X% faster). And of course, this kind of real-time data can be used to make better, more nimble business decisions that enable companies to differentiate and be more reactive in the marketplace. This is often exactly the type of intermediate measurement that larger enterprises need to justify the changes they are making; to justify the journey, with real business metrics.
DevOps: A Continuous Journey of Continuous Improvement
Returning to the question I posed at the beginning of this blog, I can tell you that neither release automation nor continuous delivery are the endpoint of DevOps – but they are certainly significant milestones to shoot for on the journey. So if they’re not the answer, what is?
To be perfectly honest, I’m not sure DevOps has a concrete endpoint. As the cliché goes, it’s more about the journey than the destination. As I have written in previous blogs, DevOps is all about delivering high-quality applications faster and at lower cost. As long as you’re continuously improving in that direction, the endpoint really doesn’t matter.
