devopsy.kinsta.cloud

Tag: metrics

  • Optimize app performance with the help of real-time monitoring and analytics

    Optimize app performance with the help of real-time monitoring and analytics

    Back in 1980 there was a movie called Nine to Five—referring to the commonly held perception of a standard work day. Of course, that work day evolved to be more like eight to six (or seven), and now it seems like the concept of a work day—or work hours at all—is a distant memory. In order to keep up with the instantaneous, always-on, connected 24/7 pace of today, companies need to be able to understand how their apps are used, and proactively address issues in real-time.

    That’s where companies like MixPanel come in. MixPanel offers an advanced analytics platform for Web and mobile apps so that developers can keep tabs on when, where, and how apps are used, and any problems that might arise.

    I spoke with Tim Trefren, co-founder of MixPanel. He proclaimed, “It’s easy to build a product. What’s hard is building a good product.” I assume that wisdom applies both to MixPanel itself, as well as to the customers that employ MixPanel to monitor and improve their own products.

    Trefren told me that MixPanel provides a very flexible system that enables customers to track and monitor the metrics that matter most to them. It allows for custom metrics, so companies can choose to monitor elements of apps and app functionality that are completely unique. Customers can also implement A/B alternatives and compare metrics between the two to determine which is most effective.

    Monitoring performance metrics, and analyzing that data to fix or improve apps fits into the broader concept of agile development, and rapid iteration that define DevOps. Trefren explained that some customers also employ MixPanel in an innovative way that also applies to DevOps—to actually monitor operational data.

    Trefren says MixPanel may not be the ideal tool for monitoring operations, but it does enable admins to monitor everything from a single display, and it can be used to alert IT personnel when issues arise. For example, if MixPanel starts logging metrics that show an app is not being used at all, that could indicate that a server is offline.

    MixPanel also recognized that it was in a position to do a little more. It offers a service that analyzes captured metrics data, and enables MixPanel to help companies understand who their customers are. Understanding the audience, and the actions they take (or don’t take) with an app provides valuable demographic data that companies can use to target marketing efforts with greater precision.

    Trefren also told me that he is exceptionally proud of the engineering team at MixPanel. While the volume of data being analyzed has tripled in each of the last two years, the engineering team has only doubled over those same periods. That means that the MixPanel engineering team is working more efficiently, and getting exponentially more done in order to keep up with demand, but that’s another story for another day.

    “The combination of power, flexibility, and performance set MixPanel apart from competitors. We’re able to do more, and do it more quickly,” said Trefren.

    That may be true, but MixPanel isn’t alone. I wrote a while back about similar services from AppDynamics, including its DevOps War Room feature. Clearly, the rapid pace of development, and the sheer volume of apps have created a market for being able to monitor metrics, and measure the performance of those apps in real-time.

    MixPanel offers its analytics as a freemium service. Paying customers are billed for metrics per data point, so the overall cost ends up being a reflection of the volume of data being analyzed.

  • The Top Metrics You Need to Track to Improve Operational Performance

    The Top Metrics You Need to Track to Improve Operational Performance

    Living in a data-rich world is a blessing and a curse. Flexible monitoring systems, open APIs, and easy data visualization resources make it simple to graph anything you want, but too much data quickly becomes noisy and un-actionable. At PagerDuty we’ve thought hard about what you should monitor and why from a systems perspective, but what about monitoring data on your operations performance? We’d like to share some specific metrics and guidelines that help teams measure and improve their operational performance.

     

    1. Raw Incident Count

    A spike or continuous upward trend in the number of incidents a team receives tells you two things: either that team’s infrastructure has a serious problem, or their monitoring tools are misconfigured and need adjustment.

    Incident counts may rise as an organization grows, but real incidents per responder should stay constant or move downward as the organization identifies and fixes low-quality alerts, builds runbooks, automates common fixes, and becomes more operationally mature.

    When looking at incidents, it’s important to break them down by team or service, and then drill into the underlying incidents to understand what is causing problems. Was that spike on Wednesday due to a failed deploy that caused issues across multiple teams, or just a flapping monitoring system on a low-severity service? Comparing incident counts across services and teams also helps to put your numbers in context, so you understand whether a particular incident load is better or worse than the organization average.

     

    1. Mean Time to Resolution (MTTR)

    Time to resolution is the gold standard for operational readiness. When an incident occurs, how long does it take your team to fix it? Downtime not only hurts your revenue but also customer loyalty, so it’s critical to make sure your team can react quickly to all incidents.

    While resolution time is important to track, it’s often hard to norm, and companies will see variances in TTR based on the complexity of their environment, the way teams and infrastructure responsibility are organized, industry, and other factors. However, standardized runbooks, infrastructure automation, reliable alerting and escalation policies will all help drive this number down.

     

    1. Time to Acknowledgement / Time to Response

    This is the metric most teams forget about– the time it takes a team to acknowledge and start work on an incident.

    While an incident responder may not always have control over the root cause of a particular incident, one factor they are 100% responsible for is their time to acknowledgement and response. Operationally mature teams have high expectations for their team members’ time to respond, and hold themselves accountable with internal targets on response time.

    If you’re using an incident management system like PagerDuty, an escalation timeout is a great way of enforcing a response time target. For example, if you decide that all incidents should be responded to within 5 minutes, then set your timeout to 5 minutes to make sure the next person in line is alerted. To gauge the team’s performance, and determine whether your target needs to be adjusted, you can track the number of incidents that are escalated.

     

    1. Escalations

    For most organizations using an incident management tool, an escalation is an exception – a sign that either a responder wasn’t able to get to an incident in time, or that he or she didn’t have the tools or skills to work on it. While escalation policies are a necessary and valuable part of incident management, teams should generally be trying to drive the number of escalations down over time.

    There are some situations in which an escalation will be part of standard operating practice. For example, you might have a NOC, first-tier support team or even auto-remediation tool that triages or escalates incoming incidents based on their content. In this case, you’ll want to track what types of alerts should be escalated, and what normal numbers should look like for those alerts.

    About the Author

    David Shackelford HeadshotDavid Shackelford is a product manager at PagerDuty, the leader in operations performance management. David works with teams across the company to plan, build, and ship features that improve operation teams’ quality of life, decrease time to incident resolution, and ultimately improve uptime for PagerDuty customers. Prior to PagerDuty, David worked in education technology, creating integrations between school information systems and digital content, and as a Teach for America corps member, teaching Mathematics in San Francisco public schools. You can follow David on LinkedIn: https://www.linkedin.com/in/dshackelford and Twitter: https://twitter.com/dshack

     

     

     

  • DevOps Sometimes Means Kill It Before Fixing It

    DevOps Sometimes Means Kill It Before Fixing It

    The public attention for DevOps is having a very positive effect on many areas of IT with regard to improving their ability to deliver for the business. Yet, I see little attention being paid to an integral part of this desire to change, which is to assess and baseline the current approach and then strategically select those areas where attention needs to be paid. Alternatively stated, many DevOps efforts are focused on improving current approaches without even determining if the current approach is appropriate for the business.

    In their book, “The Lean Mindset: Ask the Right Questions,” the authors question the focus on productivity as an aspect of implementing lean or agile methods. This comes up quite often in discussion with customers regarding DevOps adoption. There’s a misconception that becoming more productive is the way to fulfill demand from the business. But productivity is not the right question or answer. Regardless how you approach the problem domain lack of infinite resources—money, time and people—the demand will typically exceed the ability to fulfill.

    The way for IT to help the business achieve agility is to deliver the right set of features at the right time, which means the first step has to be looking at what you’re doing today and being willing to stop spending time and effort where there is minimal value or gain. The authors of “The Lean Mindset” have some interesting anecdotes around the realities surrounding this in enterprise settings.

    • Politics – one company decided to financially incent development management to deliver agreed upon features on a behind schedule project even though it meant that testing would suffer and the final product would be unreliable. When later asked if they could have reduced the feature set and delivered fewer high-quality features, the team stated that it could have been done as less than 50% of the features were being used in the final deliverable anyway. The politics of delivering what was agreed upon was more important than delivering fewer high-quality features.
    • Can’t Say No! – many IT executives find it difficult to push back on the business and, hence, end up attempting to deliver all that is already on their plate plus new requests that arise during the year. In many cases, much of what is already on their plate is no longer needed or provides minimal value to the business. Without a system of measurement around cost to deliver versus value delivered, IT cannot justify to the business the need to shift resources elsewhere to ensure optimal value.

    For these reasons, among others, any attempt at DevOps needs to start with an understanding for where resources are being applied today and where they should be applied in the future. Doing useless stuff faster doesn’t help IT and it doesn’t help the business. It’s also why IT/Business alignment is a critical part of any DevOps adoption strategy. IT needs to work with the business in an open manner to select the most important features, capabilities, systems, etc. that will deliver value for the business. This also means IT is responsible for instituting the necessary systems to measure resource consumption and provide the business with the necessary data for them to make intelligent choices about where they want to spend their IT capital.

    There’s a great Peter Drucker quote in the book, “There is nothing quite so useless as doing with great efficiency something that should not be done at all.” So, if your IT Operations team spends six weeks automating deployment for a system that really should have been sunset, that’s six weeks lost that could have been better spent on a more business critical function. Of note, this is not the IT Operations’ fault. If that system is not going to be sunset and has been the bane of their existence consuming time due to outages, then they have driven improvement. The failure is on IT management and the business having not taken the necessary steps to allow these resources to be focused on more important initiatives once again illustrating the importance of executive sponsorship for DevOps initiatives prior to tooling, automation, process improvement or organizational shifts.