devopsy.kinsta.cloud

Tag: metrics

  • DevOps Without Scale, Part 2

    DevOps Without Scale, Part 2

    In Part 1 of this series, I covered why it’s important to be aware of your product’s health and how you can do so. I covered the user experience and technical types of metrics and how to get started with collecting them. In this part of the series I will walk you through how to use all these metrics to maximize the user experience and product uptime.

    Metrics, Metrics Everywhere

    There are no silver bullets here and you have to be aware continuously of your product’s health and how it fares against user expectations. It surely takes time and effort, but it’s an essential part of building a good product. Imagine if chair designers didn’t care for how people use their creations; those chairs would not be comfortable to sit in. Similarly, understanding user needs is a paramount part for any software engineer.

    Once you’ve established the basic understanding of your application patterns, you are not done. Every release is an opportunity to introduce more latency or failures and the baseline is ever-shifting. It is easy to significantly affect the latency and as easy to break an edge case and drive user failures up by a few percent with any release. The only way to ensure high-quality user experience is to be aware of your product’s health and your users’ expectations at all times.

    This is the case where less is more and users prefer fewer features that work and are responsive. Try it out and see what your users say!

    User Experience Monitoring

    The first part of a good user experience is low error rates. For web products, these include all http error responses as well as all application errors returned to the user. For APIs and GUIs, this includes application failures (and http errors for REST APIs). One recurring comment I hear is that users submit invalid input, which invariably causes error responses and so it should not count. It is certainly useful to be able to differentiate hard failures from bad user input, but the line is more blurred than you think. For example, a new release might change input expectations and previously valid requests can start erroring out. Is that bad user input or did you just break your product?

    In either case, if the failure numbers are high, users won’t be happy and you need to deal with this problem. The solution here can be non-technical: Maybe you need better documentation or user training, or maybe the API or UX design is cumbersome and drives misunderstanding and misuse. Whatever the case is, it must be addressed. Users are as likely to walk away from a cumbersome product as from a product that doesn’t work.

    As with everything else, I suggest that you identify a few high-level and most important parts of the product and keep track of those metrics. If you do notice an issue, then you have the more detailed metrics on hand for a deep dive.

    The other part of good user experience is performance, and the approach here is no different. Typically you need to track the latency of the same user actions as you’re tracking for failures. Performance tracking is where you really need to understand what’s acceptable and what’s not for your users and that depends on the domain and your users’ expectations. Two seconds to open a reservation on a public travel site seems like a long time, but two seconds to pull up a loan record for an internal user at a financial company is not bad.

    An important fact to remember is that you will always have some failures in the system and that’s normal. No matter how great your product is, how clear its documentation, and how in tune you are with your users, there always will be some failures caused by bad user interactions. At some point, once you feel good about your product overall, you will have to assume that the given error rate is your baseline. Then you need to pay attention to variation over time and revisit critically every once in a while.

    Predictive Monitoring

    In addition to user experience metrics, there are key technical metrics that are good to watch, as they can point to upcoming problems. Some of the common ones are CPU and memory utilization, thread and database pool sizes, VM heap size and garbage collection times. Just like with user metrics, you need to know the operational signature of your application. But in addition, you need to understand how changes in these metrics impact your users, if at all.

    A growing thread pool might mean that response times are slowing down or it could simply mean that more users are accessing your service. A growing email queue size means that emails aren’t going out as fast, but is it bad? Only knowing your domain and users will give you the answer. If the CPU is hovering at 80 percent everything is fine—for now, but you need to scale up quickly before a tiny addition of traffic kicks your whole app over.

    When tracking hardware metrics everyone pays close attention to overutilization because that’s what causes outages. But what about underutilization? Performance fluctuates up and down and at times you can find yourself underutilizing resources. Paying attention to this is a good way to stay frugal and keep costs in check.

    Alerting

    How do you actually know when something breaks? You can’t stare at the graphs all day long. Once you get a feel for the normal characteristics of your system, you’re ready to create automated monitors and alerts. One of the most important things about automated alerting is that it always must be trustworthy. The moment you and your team start getting false alerts, everyone will learn to disregard some of them and the the value of the system will plummet. Thus I recommend you start with auto-alerting only on several most critical metrics. Run the whole system for a few weeks and slowly add more metrics over time.

    Another aspect to be aware of when setting up automated alerting is metric variations over time, the most common one being day vs. night. Nobody wants to get paged because traffic is too low at 3 a.m. At the same time, if your metrics skyrocket or drop suddenly, it can be a cause for concern. Are you being DDoSed or scraped? Is there a network failure that’s preventing your service from being reached?

    Tooling to the rescue, DataDog has built-in capability to create alarms, called Monitors. They provide a few different ways to ensure no false positives. For example, instead of setting absolute thresholds, you can create a change alert that fires only on deltas. And adding a new monitor only takes a few minutes.

    Finally, someone needs to get paged out when an alert goes off. DataDog can do email notifications but nobody checks that at night and even during core hours it takes time to notice a new email. For critical issues like yours, someone needs to get notified immediately.

    Services like PagerDuty and VictorOps have made it trivial to do this. As expected, DataDog integrates with both of them natively. Both services support on-call rotation scheduling with an escalation policy, calling out to phones, text messaging, emails, and more.

    Production Troubleshooting

    So you’ve been alarmed and need to quickly figure out what’s going on and what to do. Your next stop is typically logs, but looking through log files one server at a time is a sure way to waste quite a bit of time.

    Log management tools and services have matured over time and there are good alternatives to Splunk. Loggly and LogEntries cover more than just the basics and provide indexing and search capabilities, graphing of log occurrence, understanding of JSON, and out-of-the-box integration with just about all logging frameworks out there. At their core they provide a quick way to look through all logs without having to go to individual servers. Moreover, you can monitor various aspects of logging à la DataDog and alarm (via VictorOps or PagerDuty) when metrics are off.

    The above approach already solves most needs, but I’d like to touch on a more advanced problem, especially for a microservice architecture. When multiple services are involved, finding the failure at the entry point is easy but figuring out which back-end service failed in the call chain while processing the user request can be quite difficult.

    An architectural pattern that I call log stitching solves this issue. With log stitching you can pull up all logs across your entire architecture for a given user request. This allows you to reconstruct the complete call chain and quickly find the root cause of failures. Conceptually this technique is straightforward: You generate a unique request ID at the entry point, pass it down the chain for every call, and log it. Now you can pull up all logs with that request ID by searching for it. In practice it can get complicated because you need to: a) ensure consistency across all calls, b) account for all entry points including batch jobs, and c) preserve the request ID internally in multithreaded services.

    Log stitching is not something I recommend from the get-go; however, it’s definitely something to consider as your system matures. If you are going down the microservices path, this really becomes a must since tracing call chains get progressively more difficult as the number of services grows.

    Putting it All Together

    There is not much development work in putting it all together, but nevertheless you will need to put in some time. I’d like to emphasize how important it is to work in iterations. Set up one to three alarms, connect your DataDog to VictorOps or PageDuty, study them for a couple weeks and then move on. Keep in mind that you will not get it right at first; your thresholds will be off and you will misinterpret or misunderstand system behavior.

    Operational pattern analysis is not trivial, but don’t spend too much time analyzing the hell out of everything up front. Configure some alarms, get them out there, learn and adjust as you go. When you find an alarm that went off erroneously, take some time to understand why. Specifically, you need to focus on why you thought it was the correct alarm setup vs. why you now know it is not. You will learn by leaps and bounds if you take this systematic approach.

    As I mentioned above, configuring Loggly or LogEntries is super easy and you should do that right away. Log stitching, however, is an involved endeavor that requires planning. Due to its nature it only becomes useful when adopted by all, or most, of your systems. Thus, you should only start it when you feel the need and when you have the buy-in from the rest of your development organization.

    What’s Next?

    Part 3 will cover the topics of continuous integration and continuous delivery, breaking them down into multiple stages of complexity so you can pick and choose what makes the most sense for you from the cost/benefit point of view.

  • DevOps Without Scale, Part 1

    DevOps Without Scale, Part 1

    When most people think of DevOps, what comes to mind first are continuous integration (CI) and continuous development (CD) pipelines. While they are arguably the most important parts of DevOps, they are not all of it. So, then, what is DevOps? It is a philosophy of how to build and operate software products and systems to provide the best user experience. It’s a way of thinking that blends the previously independent disciplines of development and operations so that software products can both evolve functionally and be stable operationally. In practical terms, it means development teams are involved in operations and, in many instances, fully own the operation of their products.

    The top companies are already there because they otherwise couldn’t deliver their massive and feature-rich products with 99.9 percent uptime. But there are still too many companies and developers who aren’t “doing it.” And it’s typically reflected in product quality.

    It seems DevOps has been stigmatized as difficult, requiring expensive tools and generally classified as needed if you’re big. In this blog I’ll walk you through different aspects of DevOps and share the business value behind each, the common implementation techniques and the free (or inexpensive) tools that will let you jump-start your DevOps in a matter of weeks.

    Product Health: The User Experience

    Our goal as engineers is to build products that will be used by others. Typically we call them users and sometimes we have a strenuous relationship with them. But if users don’t like our products and aren’t using them, we have failed at the very core of our job. So what do users want—besides features, that is?

    Users want products that are reasonably fast and work when they need them. Your site going down is an obvious example of why it’s important to be aware of your product’s health. But there are more nuanced examples. Sometimes a new release breaks an edge case and isn’t caught by regression testing. Sometimes a service becomes slow—it still works and nobody is complaining, but it’s dangerously close to being unusable. You need to be aware of your users’ experience so you can be proactive in fixing problems and delivering quality products.

    Thus, the first set of metrics you need to measure are user experience metrics. These typically include success/failure and latency of all external endpoints: REST, web pages, etc. If the users are external, you want to measure the experience from their point of view, which includes the network latency and HTML/CSS/JS execution time for web pages. If this was a car, these metrics would be your low coolant red light. If the light is on, you must stop as soon as possible and add coolant before your engine block cracks from overheating.

    To measure and monitor customer experience for web pages, Google Analytics is by far the most popular choice. It’s easy to set up and gives you exactly what you’re looking for: error stats per page, page load times, including individual page components and even browser parsing time for HTML. You can slice and dice the data by geographic region, browser type and so on to zero in on improvement or problem areas. Google Analytics is a great tool for tracking other product aspects as well, and you can kill many birds with one stone by adopting it.

    However, Google Analytics will provide data only as long as there are users and your site works. But what if the site breaks, maybe partially or maybe only for a certain geographic region? And what do you do for public APIs that aren’t instrumentable with Google Analytics? You’ll need a synthetic monitoring service. These services emulate clients by calling your APIs or opening your web pages in an automated fashion. They are typically scriptable and run from multiple places around the world. They can tell you when your services are up or down and can measure (emulated) user latencies by geographic region. Pingdom is one of the newer generation of such services that delivers what you need for a modest price. It also integrates with a graphing and alarming tool called Datadog, which I’ll talk about later.

    Product Health: Technical Innards

    Now you have a solid understanding of your user experience and can detect user problems in real time. But how do you find the root causes? And can you do even better by detecting user problems before they happen? What you need to do is go a level deeper and monitor key technical aspects of your apps so you can correlate them with user problems. The technical metrics are important to watch because they can be early signs of trouble, which can be averted if you act quickly.

    This can be a dangerous territory as the first desire is to monitor everything. Please keep your cool here, lest you drown yourself in hundreds of metrics that nobody understands and miss the important ones. In software, failure typically happens at integration points, so that’s what you should focus on: database and cache interactions, calls to other services, disk operations, queue sizes (for pub/sub architectures) and so on. Typically you don’t need to instrument business logic, as it either works or doesn’t and problems are caught in testing. This is where it is easy to create many metrics in a blink of an eye. A notable (and rare) exception is load-intensive parts of code that you want to monitor for performance reasons.

    If you see latency in database calls creeping up, cache hit ration going down or thread pool or queue size growing, you can frequently identify and fix the problem before it affects the users. Going even deeper, you want to keep an eye on the virtual machine (VM) and server performance. If your garbage collection times are increasing, if your thread count grows or if your server is showing signs of strain, it’s time to pay extra attention. If this was a car, these would be your check engine yellow light. If the light is on, you need to check it out soon but can keep on driving for a little bit longer.

    The most popular way to measure application metrics is with code instrumentation. Presently, Coda Hale (a.k.a. Dropwizard) Metrics framework and its ports to most popular languages are the way to go. Given the abundance of documentation and integration with all sorts of systems, this isn’t difficult. There are a plethora of coding patterns or external libraries you can use to avoid writing boilerplate code. Note that you want to include as much as possible in measuring requests (serializing, logging, etc.) so you need to put this logic as high up the filter chain or middleware as you can.

    The last piece of the puzzle is VM and server monitoring. Traditionally, this has been the territory of specialized ops tools, but Datadog, a service that I will talk about in the next section, provides this out of the box. Its agents run on each machine or VM and collect server metrics, JVM/JMX data, IIS info (yep, it supports even Windows) and more.

    Processing and Displaying the Data

    Now your code is instrumented and generates all sorts of interesting data. What do you do with it? And where does the data go in the first place? All this data is useless if it just sits there. You want to be able to look at the trends, correlate them and get a basic set of statistical information such as averages, means, standard deviations and so on. You also want to see pretty graphs (who doesn’t?) and be able to create dashboards. This is the cool part, where all the previous work comes to fruition.

    Datadog is a great way to start. It provides everything you need and more. All you have to do is install its agent on each box and configure the Metrics framework to send data to Datadog. Most Metrics ports already include Datadog adapters so there is no extra coding required.

    Now you have everything and can slice and dice the data whichever way you want. You can export data for further analysis. You can check it periodically for problems. You can even display dashboards on big screens around the office (looks cool, eh?).

    Putting it All Together

    I have seen all of this put together in a couple of weeks. However, if this whole area is new to you it might take longer, as you’ll be learning the concepts along with the tools. Sometimes the deadlines are tight and even a few weeks is a luxury. However, launching a product without any insight into user experience or product health is not a good idea. If I had limited time and had to prioritize, I’d start with instrumenting user experience metrics and hooking it up to Datadog. Even for web pages you can start with only monitoring the REST services that power them, or the controller layer, if you’re on model-view-controller. Soon you will realize how nice it is to have hard metrics and will want to add Google Analytics and deeper code monitoring to your product.

    All of this might seem like a lot, but you’re only dealing with four tools here: Pingdom, Google Analytics, Metrics framework, and Datadog. If your product is an API then you don’t need Google Analytics and if your product is internal, you don’t need Pingdom.

    As I mentioned earlier, Metrics framework is ported to multiple languages already. But even if you have specific requirements that aren’t supported out of the box, it’s easy to extend it. For example, at Guaranteed Rate we wrote and open-sourced the Metrics/DataDog adapter for .NET and a library for code instrumentation and DataDog integration for Clojure.

    .NET is particularly known to be behind on DevOps. Until recently one of the reasons was the lack of tools; however, that’s not the case anymore. All tools mentioned here, as well as Part 2 and 3, work in .NET and have adapters and ports where needed. With .NET entering the new open-source age, there are no more excuses to stay behind.

    Give yourself two weeks, read the docs, go one step at a time, and you’ll be surprised how far along you can get!

    What’s Next?

    Part 2 will cover what you can do to maximize product uptime based on the data you now have. It also will discuss making sense of the data, automated alerting and troubleshooting when the inevitable production problems happen.

    About the Author/Jack Korabelnikov

    JackKorabelnikov_avatar (small)Jack Korabelnikov wrote his first computer program when he was 10. Another 15 years later, after success in the role of senior engineer at Orbitz.com he transitioned into management. He still enjoys the tech side of things and continues to tinker with cool new tech. He is presently the VP of Engineering at Guaranteed Rate, leading the company’s technical evolution and the buildout of next-gen mortgage products.

    Prior to that, he worked at Orbitz Worldwide in multiple business areas such as hotels, private label and B2B web services. He played a key role in rolling out agile and, most recently, was the main driver behind the continuous delivery and DevOps adoption across the organization.

    jack@korabelnikov.com | LinkedIn

  • Culture Change: Metrics, Tools Make a Difference

    Culture Change: Metrics, Tools Make a Difference

    Change is in the wind.

    This time of the year, as winter weather gives way to springtime renewal, this concept isn’t a cliché but rather a concrete, visceral experience. You go outside, smell the earth defrosting and know for certain that change is happening all around you. You can feel it.

    Such is the case on the current DevOps landscape. Seemingly every conversation, with everyone from end users to industry analysts, reaffirms that people are making significant progress toward greater adoption. For an increasing number of organizations, DevOps is maturing from a set of budding goals into a tangible reality.

    At the center of this massive shift within the software factory is the issue of cultural change; as we know in all things DevOps, culture trumps all else. So how are organizations affecting, empowering and measuring all this seismic transition, in particular the human elements?

    During the recent DevOps Virtual Summit, leading practitioners representing industries ranging from banking to travel and technology shared their views on the progress of cultural reinvention.

    From management tips, to tools and even metrics useful for driving transformation, here are some of their observations:

    1. What’s the most critical cultural aspect of planning?
      Kazi Whitfield, director, Quality Engineering, MUFG Union Bank, N.A.: Scalability on this is the key; if you’re doing this at a new organization, you need to understand how to scale it, and starting at some smaller level helps you get there. It also helps each individual and those components that are changing to truly understand what’s changing, and why it’s changing; I think that’s a huge part of the conversation that has to happen.
    1. What’s the biggest obstacle to cultural change?
      Eric Warheit, senior manager, Environment Management, Southwest Airlines: The challenges that we’re working through are organizational in nature. It goes back to the question of localization, and people maintaining a mindset that they’ve previously had, and what we do to create the conversation around what’s possible and the key impacts that we want to have. Ultimately, it’s about being productive with quality, and creating those common values.
    1. How can tools enable cultural change?
      Eugene “Bear” Lehenbauer, group manager, Information Technology, Citrix: If I have to pick one that links them all, a ring to rule them all, that’s when you have an orchestration platform that can take what individual teams are already using and plug right in, or you can take what needs to be automated and automate it; that orchestration harness can now become that common thing that links them.
    1. How can you ensure cross-culture collaboration?
      Warheit: Creating that concrete example, that proof case tying analytics to it, so that we have real measurable ways to make good business decisions, and being able to foster that perspective so that we can look across the organization, as opposed to in a localized way, is really critical to be able to foment that collaboration that’s needed.
    1. How do you measure cultural change?
      Daniele Romano, product owner, Continuous Delivery Pipeline, ING Bank: The most important thing as an organization is to able to apply an analytical approach, meaning that for each team or department you should look at the current situation and have in mind the target situation, and then have different means to guide everyone toward that target.
    1. What’s a key cultural change metric?
      Lehenbauer: We’ve a developed an agility rating scale that allows people to earn scores for something as basic as change management. Based on how those key metrics behave, and there are several of them, but they are fact-based and they’re user impact driven for the most part, they reward the right behaviors. Those types of things can draw out some of DevOps principles results, but also have hard data behind them.
    1. How do you recognize advocates of cultural change?
      Whitfield: Being able to phase reward and recognition into the process is key. Organizations are doing a lot of great things and making millions of dollars, and as the individuals that are coding or testing are driving it, you have to understand how you get to those people’s hearts, to make them passionate about what they’re doing. That’s something that becomes key and it’s something that’s different per organization.
    1. What’s the best way to gauge successful cultural transformation?
      Romano: There is a really only one critical human metric, which is happiness. So, if engineers are happy and they can deliver fast, at a high quality and with intense focus—that’s a very unique but important metric.

    So there you have it, some of today’s most critical elements in driving the cultural transformation necessary to succeed in the DevOps world.

    If you missed the DevOps Virtual Summit you can still view recordings of all the sessions including this customer panel, and stay tuned to this space for more excerpts and highlights.

  • Changing the Culture of Metrics Analysis

    Changing the Culture of Metrics Analysis

    You could say the development and operations teams gave birth to the idea of DevOps, as it is believed the chief benefits are garnered in those two functional disciplines. Because of this, metrics nearly always are conducted from the points of view of cost reduction, speed and efficiency. This makes perfect sense when business treats IT (and IT sees itself) as a service- or cost-based organization. But what if DevOps has a few more characteristics that could actually change that culture? What if, for the first time, DevOps could enable IT to become a real partner with the business organization it serves?

    To make this happen, business and IT must begin to share attainable goals. Imagine the impact of a highly functioning, enterprise-class DevOps service within IT; upon the time to market of products and services the business is looking to launch. A proven, consistent ability of IT to deliver on time or ahead of schedule (i.e., reliability) then begins to change the dynamics of what marketing proposes and delivers. Time then becomes a valid metric to consider when looking at customer adoption rates. DevOps usually does not directly impact initial market demand, but the impact of delivering new services or features months or years ahead of previous projections can dramatically impact time to market. For example, if your business is able to change its billing model or billing methods six months to a year ahead of every one of your competitors, what impact might that have on market adoption? At that point DevOps becomes a key business differentiator, until the rest of the universe has it.

    Consider an additional impact of a highly functioning DevOps service on feature delivery or the feature road maps themselves. The implementation of DevOps inherently delivers smaller units of change on a higher frequency, reducing risk and allowing iteration and alteration of future plans based on the real-time feedback of change success. Marketing focus groups may provide an original road map, but the impact of DevOps can be integrated back into those same focus groups. Instead of imagining what might be possible, DevOps allows delivery of what is possible to see, feel, touch and prompt further imagination. The net impact is an increase to the pace and depth of innovation itself. You know what your customers really want, and you can give it to them quicker. What kind of impact does that have on the reputation of your business if it becomes a market leader because of this? Or what happens when a large organization is then able to deliver with the speed, agility and flexibility of a small firm?

    Cost Reduction vs. Revenue Generation

    Now consider the value of cost-reduction metrics compared with revenue-generation metrics. It is easy enough to determine that the pre-DevOps delivery cycle might have been 12 weeks per feature as an example, or that each build took six hours to complete and now post-DevOps takes only 12 minutes to run. Metrics like these will begin to show an increase in productivity by existing development staff—or, maintaining the same feature delivery rate, a reduction in the number of personnel needed to achieve them. Similarly, it is easy to show a reduced number of trouble tickets post-DevOps, and a concurrent reduction in the average time to resolve each ticket. DevOps then has enabled the CIO to continue to take arbitrary budget cuts demanded by the business each year. This is not partnership—over time this is a recipe for the reduction, pace and quality of innovation (the smartest people leave first during layoffs; they don’t need to stay).

    However, even significant cost-reduction metrics pale in comparison to revenue-generation metrics. As an alternative, imagine what the net growth in customers of a given product might be, over a six-month period of time, because a desired feature set was delivered early? And what if it is two to three years before your competitors can implement the same high-functioning DevOps services you already have in place today? If a business makes $2 per-widget profit and captures 20 percent additional market share over six months, equating to “x” million new customers … well, you begin to get the idea. All of the sudden IT is no longer viewed as a cost center by business; it is viewed as a revenue-enabler. The immediate business response is, “If I allocated more budget into the types of staff facilitating DevOps services (running the gamut from coding to operations to marketing), could I gain even faster delivery rates or deeper feature sets or more market share by pioneering what has never been pioneered?”

    The two main reasons this realization hardly ever occurs in business today are that 1.) IT has failed at delivering on promises so often it has trained the business to believe it “cannot” deliver on time or on budget; and 2.) few businesses conduct accurate time-to-market metrics measuring the delta of what additional “time” does to the equation, largely because they never had to. DevOps provides the catalyst to begin to re-examine both of these phenomenon.

    But it goes deeper than that. Changing the culture of innovation, in which I am eager to test new ideas faster, means I need more ideas all the time. The competitive advantage our nation, our businesses and our careers have always benefited from is our ability to create and innovate. Other nations around the world, or other organizations your company competes with, may be able to do repetitive tasks faster or more efficiently. But rarely do worker-drones create or produce a wealth of ideas. But if we begin to focus on the long-term impact of fostering innovation in our own country, and in our companies, we will remain perpetually out front. The goal of DevOps is not merely efficiency, it is also to spark the imagination. When our focus and our metrics reflect this, we all win.

    If you would like to continue a conversation on how to create and implement a highly functioning enterprise class DevOps service, feel free to contact me.

  • The Big DevOps Question: How Do We Get Started?

    The Big DevOps Question: How Do We Get Started?

    Every conversation I have with clients regarding DevOps eventually reaches a point where the client wants to know my recommendations for getting started with a DevOps program. Some have already embarked on a DevOps program, but want me to critique their approach against what I see others doing. Others simply don’t have a clue how to make the necessary changes in their organization.

    Ultimately, I sense in all these conversations a lack of confidence that their intended outcome is achievable and that they’re on the right path.

    The first questions I ask clients are, “What are you looking to accomplish, and what are your business drivers?” Interestingly, 90 percent of the time clients are unclear why they want to embark on this journey. Sure, DevOps makes sense academically, but if you don’t have a clear understanding of your expectations for improvement most likely you’ll end up with more questions.

    Sometimes it helps if I give prompts:

    • Are you looking to improve quality in your production releases?
    • Do you need to release updates and patches into production more quickly?
    • What metrics are you currently looking at, and what makes you believe these are not optimal for your business?

    This last question is really important. Quite often data represents invalid perspectives. Other industries’ and services’ metric of how often they deploy to production or how long it takes execute a release cycle might not be right for your business or industry. I hope medical device and airplane equipment manufacturers take a lot longer to bring something to production and test it more robustly than Netflix does its website.

    In an earlier post I wrote that the need for DevOps is effectively the admittance of a problem; to wit, the cure is continuous delivery. Hence, I often tell clients that a very realistic place to start is not with the actual work, but with the communications methods and pathways that will affect implementing continuous delivery within the organization.

    So many clients want to start their DevOps journey in application development and talk about what they’ve done with continuous build. That’s fantastic, but it has more to do with Agile software development than it does DevOps. If all that work creates bottlenecks in testing, you’ve accomplished very little. Now you must fix the flow to balance out development and testing. Trying to achieve continuous development through serial analysis and correction will take a long time and show little progress along the way—and often leads to abandonment.

    However, if all the leaders of all the various areas involved in the continuous delivery cycle discuss how their efforts are impacted by what the other leaders’ groups do, everyone can see the bottlenecks and constraints and they can prioritize changes. Management often fosters much of the cultural impairment, as these changes can impact everything from workload to bonuses. Getting everyone involved in working together to eliminate the bottlenecks often buys credibility for the program and minimizes some of the individual adoption issues.

    Don’t jump into the deep end of the DevOps pool just because everyone else is. Before embarking on your DevOps journey, do a deep analysis of your organization’s goals and baseline your current levels of performance. Discuss what needs to change based on real business goals and drivers (e.g., it takes us too long to issue regulatory changes to the system). Identify bottlenecks and constraints that exist in the current environment today and put together a plan to eliminate as many as possible.

  • End-User Analytics: A Reality Check to SLAs

    End-User Analytics: A Reality Check to SLAs

    Everyone understands that a great user experience is vital for customers, but few organizations give the same level of consideration to their own employees. This lack of insight into the internal end-user experience leads to frustration, inefficiency and waste. Detailed analysis can lift the curtain, exposing the gap between your service level agreements (SLAs) and the reality of your current systems and software.

    Some of your employees may have more applications than they really need, while others may lack the tools to do their job effectively. It’s also likely that some of your employees are making do with outdated software that’s no longer fit for purpose. Before you can set about changing that, you must gain an understanding of the overall picture.

    What does your user experience look like?
    Many organizations will have trouble answering this question. They may not know how many applications they have, which ones are being used, and how well they’re performing.

    The rise of shadow IT—and the fact that 40 percent of spending no longer controlled by the IT department, according to Gartner’s “How to Manage the IT Budget Wisely Through Cost and Value Optimization,”—are further muddying the waters.

    To determine your company’s user experience, ask yourself the following questions:

    • How many users do you have, and how are their roles defined?
    • What devices are accessing your systems on your daily basis?
    • Where are all the access points located?
    • What content is being accessed?
    • What applications are being used?

    By trimming the unused apps and consolidating common apps through a virtualized XenApp or similar environment, you can redirect resources to focus where they can deliver the greatest benefit.

    SLAs are failing
    It may be tempting to take the SLA model and apply it internally, but SLAs are too simplistic and one-dimensional to deliver the insight that’s required here. Cloud service providers might focus on availability and uptime, but what do those metrics tell you about performance? Issues such as slow-loading pages usually aren’t included in the metrics, but they really should be if you want to assess the user experience.

    There’s also a trust issue here. Can you validate that your service provider is meeting expectations? What about security? Relying on provider self-reporting and a set of metrics that don’t reflect accurately the performance or the user satisfaction with the platform or application are inadequate if you really want to understand what’s going on and act to improve things.

    End-user analytics
    You need insight to real-time and historical data on all your hardware, users and applications across multiple environments inlcuding traditional, mobile and virtual. Good end-user analytics will flag security risks and suspicious behavior, record when an application hangs or crashes, and identify whether specific issues are widespread or localized. Records should be comprehensive enough that you can audit areas of concern and run root cause analysis to find ways to improve. Turn your end users and their devices into roaming sensors, which paint a factual view on usability and performance.

    Finding out how your employees really work is invaluable insight that can streamline many emotional conversations. Finding out what applications and devices they’re using—and how they’re using them—can unlock many potential benefits for different areas of your business. You can ensure costs are kept down and you maintain regulatory compliance while maximizing agility. A fresh insight into risk and productivity will help you to make informed business decisions.

    The end user knows best
    The only real measure of software quality that’s worth its salt is end-user satisfaction. Forget about what applications have been designed to do; forget about developer and service provider promises that don’t include an end-user analytical strategy. Cut to the chase and look at what’s actually happening; analyze every user session and employ business intelligence to extract actionable insights that will have a tangible impact.

    Give users the flexibility they want and the tools they need to do the best job possible. Happy and productive staff are the foundation of a good business. Seek out service providers that provide transparent end-user analytics; this will develop a foundation of trust and collectively you can provide guidance based on factual performance. Hone in on boot times, application performance or network speed: You dictate the metrics that are important to your business and metrics that make a difference to your customers.

    By acting without analyzing and putting your faith in SLAs, you’re basing the company’s future and its future path on pure guesswork. With end-user analytics, you can find out exactly what’s actually needed and ensure that you’re on the right track.

    About the Author
    crop_Nicholas Lee of FujitsuNicholas Lee is head of Global Digital Programs for Fujitsu, the Japanese information and communication technology company.

  • 7 things 2016 has in store for the future of DevOps and IT

    7 things 2016 has in store for the future of DevOps and IT

    With only a couple weeks left in 2015 it’s time to start looking ahead to what next year has in store. Let’s be honest—most of us have already mentally checked out for the holidays anyway, so it’s already 2016 for all intents and purposes. I spoke with a handful of industry experts to get some thoughts on what we have to look forward to in 2016.

    Let’s just jump right in:

    1. Log Management will play a larger role

    According to Christian Beedgen, CTO of Sumo Logic, log management will emerge as the most underrated opportunity for IT. “Thinking about log data may seem too far in the weeds for most tech industry professionals. However, using analytics to monitor, manage and gather insights from logs will be the only near-perfect way to make sense of the increasingly complex and cloud-based architectures.” Log management isn’t exactly new or cutting edge, but Beedgen believes that 2016 will see more vendors trying to move into the log management space, and more organizations trying to use log management to make sense of their big data mess.

    1. Business leaders will demand better insight

    On a similar note, Rick Fitz, senior VP or IT Markets for Splunk, believes that analyzing data and metrics will also play a role from a business strategy perspective. “In 2016 we will see data science demystified, moving from Ph.D. to MBA, with easier ways emerging to consume, analyze and correlate machine data. As business leaders increasingly go digital, this will result in greater “digital insight” into engagement, satisfaction, competition, effectiveness, response, revenue and other business goals.”

    1. Sharing economy will transform IT

    Companies like Uber and AirBnB have transformed entire market segments. The sharing economy has changed the way businesses engage customers and deliver services, and that includes IT through aspects such as cloud services, microservices, APIs and more. Fitz predicts, “Next year this ‘Uberfication of IT’ will turn into a ‘Balkanization of IT,’ driving stakeholders to demand better insight, governance and control of federated technologies to integrate adoption, usage, monitoring, security, cost control and more across shared services.”

    1. Enterprise DevOps will grow rapidly

    The DevOps revolution has taken IT and app development by storm in recent years, but most of the traction has been in startups or smaller businesses that have more freedom to shift gears and embrace new concepts. Enterprise adoption has been growing, but at a much slower rate. Andi Mann, chief technology advocate for Splunk, said, “In 2016 we will see rapid acceleration of DevOps with 50 percent or more of large businesses finally adopting ‘Enterprise DevOps’ — a pragmatic, scalable approach to adopting the culture, process and technology changes at the heart of DevOps.”

    1. The singularity approaches

    OK. That may be an ominous over-statement. In a nutshell, Sumo Logic’s Beedgen claims that the line between machines and humans will continue to fade. The foundation of this is automation—leveraging technology to perform mundane or routine tasks and sift through mountains of data exponentially faster than a human can. As artificial intelligence evolves, though, the scope of what can be automated expands as well. “In automating processes, a human can today wield the power of a legion of operators with a single hand,” stated Beedgen. “Likewise, machine learning will increasingly be adopted for its usefulness in augmenting human understanding of complex interaction and large data sets by uncovering the unknown unknowns.”

    1. Automation and analytics will improve IT service health

    DevOps and microservices have transformed many areas of development and IT infrastructure, but many organizations still rely on decades-old monitoring tools, arcane metrics, and specialized admins dedicated to managing the cumbersome process. Splunk’s Fitz thinks that is getting ready to change. “In 2016, IT analytics will be liberated as easy-to-use dashboards will correlate all sources of service health—legacy tools, APIs, mobile apps, Web logs, wire data, containers and more—while predictive analytics will automatically respond to prevent quality issues.”

    1. DevOps is here to stay

    From the point of view of DevOps practitioners or companies that have adopted DevOps tools and practices, this may seem both painfully obvious, and yet still be good news as well. Christian Beedgen predicts not just that DevOps is here to stay, but that the importance of DevOps and the sphere of influence affected by DevOps will expand. “Increasingly, DevOps will permeate all manner of businesses that rely on information technology to compete. Bimodal IT (the practice of managing two separate, coherent modes of IT delivery) will become the equivalent to “private cloud”—the last forced resistance of the old guard. Both will be forgotten in a short number of years, as public cloud and lean and agile processes dominate the businesses that still compete.”

    There you have it. Now you can enjoy your holiday break in peace and come back in 2016 with an idea of what you have to look forward to in the year ahead.

  • Four Metrics for Driving Cultural Change in DevOps Teams

    Four Metrics for Driving Cultural Change in DevOps Teams

    Everyone wants to optimize a team’s performance, but it’s not enough to just talk about it. To truly impact your team’s performance, you need to take a page from operationally mature DevOps teams and use metrics to gain valuable insight into your team’s work, enhance their capacity and drive cultural change.

    The right data allows managers to make quick decisions with confidence and minimal risk. It also lets managers see the actual outcome of those decisions, in turn allowing them to shape their team’s direction going forward and to create a happier, healthier and more productive team dynamic. However, making the shift to data-driven decision making requires more than just increased monitoring. It requires a cultural change.

    Below are four key metrics DevOps teams should be monitoring, as well as the impact each metric can have on a team’s culture and performance.

    1. Use Time to Response to Establish a Culture of High Achievement
      Time to response (also called time to acknowledgement) depends on individual team members. Incident responders might not always have control over the root cause of a particular incident, but they are always in control of how quickly they acknowledge and respond. Hold team members accountable for their response time by setting internal targets and having high expectations.Most operationally mature companies use IT operations management tools to enforce a response time target. These tools allow managers to set an incident response time window, making sure the next person in line is alerted if an incident isn’t responded to. Tracking escalations also gives managers valuable data about their team.
    2. Manage Expectations With Escalations
      For an organization using an IT operations management tool, an escalation is often an exception – it’s a sign that a responder either wasn’t able to acknowledge an incident in time or that the responder didn’t have the tools or skills to address it. While escalation policies are a necessary and valuable component of incident management, teams should ideally focus on driving the number of escalations down. Managers can gauge their team’s performance by tracking the number of incidents escalated over time, and using that data to determine whether their targets need to be adjusted.That said, there are some situations where an escalation is part of standard operating practice. For example, you might have a NOC, first-tier support team or even auto-remediation tool that triages or escalates incoming incidents based on their content. For these situations, it’s important for managers to track what types of alerts should be escalated and what normal numbers should look like for those alerts.
    3. Combat Alert Fatigue With Raw Incident Count
      As an organization grows, incident counts often grow, as well. However, as a team becomes more efficient and mature, the incidents per responder should lower or, at the very least, stay constant.Implementing IT operations management tools can help teams lower their raw incident counts by helping to weed out low-quality alerts, automate common fixes and build runbooks. They can also help break down incident count by team or service, putting incidents in context with the rest of the organization. This ensures that team members are maximizing their time on attacking alerts that matter and building new features, instead of getting mired in alert fatigue.
    4. Gauge Operational Readiness With Mean Time to Resolution
      Time to resolution, or how long it takes a team to resolve an incident, is the highest standard in operational metrics. The baseline for time to resolution can vary from organization to organization, depending on complexity environment, organization of responsibility, even the industry in which a company operates.Tracking time to response allows managers to determine their norm mean time to resolution and ensure their teams are able to work through the challenges of a major incident.

    Use These Metrics to Foster Intelligent Change

    It’s no secret that downtime is expensive, both in loss of revenue and customer trust. Metrics like the ones discussed are put in place to combat downtime and increase reliability, yet it can be easy for teams to become too focused on numbers and past performance.

    It’s important for team members to not lose sight of the business goals and why they’re tracking those metrics in the first place. It’s also important for managers to avoid over-analyzing the past. Metrics measure what has already happened, and, while they can tell a lot about a team’s previous performance, they should be used as a tool to a better future and not a means of assigning blame.

    Metrics are a means to an end, and having more information than you need won’t help you improve your team and refine your business. Keeping the emphasis on subsequent action is the key to using metrics to drive cultural changes.

  • Lean for DevOps: Measuring the meaningful

    Lean for DevOps: Measuring the meaningful

    At a recent Lean project report out to stakeholders, a piece of analysis was presented which showed the quantity of metrics being captured, in various areas, at different organizational levels. IBM’s Watson had been used to generate the chart, and it told an interesting story. But that story needed some translating.

    metrics eg
    ‘Look at all those metrics under ‘Managers”, said one Director, with enthusiasm. ‘Seems like they’re looking at a lot of different performance measures for their people’.
    ‘Yes,’ said the Lean Coach, ‘but is that a good thing in itself?’
    Another Director chimed in, ‘Well, when I look at how few metrics are being captured on timeliness I wonder how anyone knows how they’re doing.’

    As an IBMer, and Six Sigma Blackbelt, I’m pre-disposed towards data. Information can be power. But with too much being reported, it can be hard to see the wood from the trees. Performance metrics are intended to allow teams and leaders to improve their performance. For many of those people, reading charts and interpreting data may not be a core skill and it certainly isn’t how the organization wants them to spend their time. So those teams generating masses of performance metrics may not be doing themselves, or their leadership, a service.

    The key is to identify the critical few metrics. Ideally leading indicators of performance issues, rather than lagging ones. Leaving aside areas like SLA adherence and budget management to those responsible for governing them, these are the handful of metrics I would suggest a DevOps team needs to watch on a daily basis:

    • Number of bug fixes for top 20 systems
    • Number of critical issues, average time to close and longest time to close – looking just at the average may mask longer running issues
    • WIP: number of projects in each phase of the SDLC
    • Customer feedback for top 5 systems – preferably actual user feedback; failing that whatever you can infer from app usage. Top 5 systems may change over time as you launch new functionality or as usage changes according to the business calendar.
    • Percentage of projects who deliver 100% of their agreed scope
    • Core application availability – actual availability to users, not SLA adherence

    What do you think? What do you measure?

  • 7 Sets Of DevOps Success Metrics From DOES ‘15

    7 Sets Of DevOps Success Metrics From DOES ‘15

    Beyond the moral support, the executive management coaching and the tips on how to build DevOps momentum in the enterprise environment, last week’s DevOps Enterprise Summit 2015 offered a boatload of excellent fodder for those seeking to prove the value of DevOps patterns once they got back home.

    DOES wasn’t just about cheerleading—alongside lessons on the soft, squishy elements of transforming IT, nearly every speaker offered up stone cold numbers to back up why those transformations make business sense. The performance metrics shared by speakers were impressive and could go a long way toward swaying skeptical stakeholders. Here are some of the highlights.

    Ticketmaster

    Ticketmaster CTO Jody Mulkey offered a ton of excellent examples of how an organization-wide commitment to DevOps has fueled innovation and stability at his company. For example, he described how what Ticketmaster calls the ‘Boom Team’ has used a completely integrated team of devs, operations, QA and other staff of about 20 to shift its Ticketmaster.com property—which accounts for 40 percent of the company’s revenue—from a duct-tape and bubblegum affair with 2000 ModPerl at its base into a modernized system continuously delivered by that team. Similarly, its technical operations team has embraced the DevOps ethos in order to deliver what it calls ‘support at the edge’ for WebOps incidents. The success of this approach has resulted in:

    • 90 percent reduction in mean time to repair (MTTR)
    • MTTR from 47 minutes down to 3.8 minutes

    “This was a phenomenal change,” Mulkey says. “The business went crazy over this. These are measurable outcomes that have really helped demonstrate that we are making progress.”

    Target

    Target brought forward speakers to discuss the retailer’s innovation in a number of key areas. Heather Mickman, director at Target, talked about how her API team worked hard to not only easily expose product and store information via its APIs, but to also challenge itself in the way it was delivering its work.

    “We wanted to deliver new capabilities in days and not months,” she explains, saying the team needed to be sure that it could scale its APIs because it new that they would be highly reused—both internally and by partners. Target has:

    • more than 90 APIs
    • monthly volume of calls to these APIs have grown in one year from 1.5 billion to 17 billion
    • deploy 80x per week across stage and production enviornments
    • all that, with fewer than 10 incidents per month
    • this volume helped support in 2014 a 42 percent increase in guest traffic and 280,000 orders fulfilled on Black Friday alone

    HP

    HP CIO Ralph Loura says HP didn’t go for the big bang approach, but through careful pilots and the use of communication afforded by ChatOps infrastructure, his team has been able to drag HP IT slowly into the DevOps fray. That’s a good thing, considering the massive scale of the organization:

    • 7,000 developer and operations staff
    • handling over 20,000 change requests per year
    • supporting 1,400 apps
    • and 1000s of builds per day

    CapitalOne

    After four years of hard work to move toward DevOps and continuous delivery, CapitalOne Product Manager Tapabrata Pal had a number of impressive statistics to share on its its progress:

    • Code commits have moved from random occurrence to hundreds per day for any given product team
    • Integration has moved from 1 per day to every 15 minutes
    • QA performance has shifted from monthly to four times per day
    • Production releases have moved from quarterly to one per sprint
    • Defects have been reduced by approximately 60 percent
    • Feature delivery rate has increased by approximately 50 percent

    CSC

    Paula Thrasher, application delivery lead with CSC, stepped up in front of the DOES audience to show them how it is possible to battle bureaucracy at any organization—coming from the government sector, she has first hand knowledge of what it takes. In one example she shared, her organization helped a client move its cyclew time from automating deployments to staging:

    • from 12 hours down to just 12 minutes and 30 seconds.

    IBM

    IBM’s Angel Diaz, VP of cloud architecture and technology, talked about how enterprises can still innovate at the pace of startups while delivering at scale. The thing with IBM is that it puts its money where its mouth is when it comes to DevOps—the company has doubled down to not only offer a number of products viable for the continuous delivery pipeline but to also completely transform its own business through DevOps practices. Some of the results it has seen by shifting its 2,000-plus developers and 100-plus squads in the Cloud Services division to DevOps:

    • 300 percent increase in “pure coding time” for its Cloud Services developers
    • reduction of deployment time from 32 hours down at first to 6 hours and now to minutes
    • an increase of release frequency from months to every three weeks.

    Disney

    Disney’s Jason Cox, director of systems engineering, got the crowd pumped up with a slick mash-up video of Star devWars and DevOps themes (not to mention a private showing of the most recent Star Wars trailer at the end of his talk!). But he didn’t favor style over substance. In his talk he described how much success Disney has grabbed up through DevOps, including:

    • 75 percent reduction manual steps
    • reduction in deployment time for large online/digital experience systems from 3 hours to 20 minutes
    • ability to provision 300-700 srvers within 15 to 30 minutes