devopsy.kinsta.cloud

Author: Ciaran Byrne

  • Measuring the Business Benefits of AIOps

    Measuring the Business Benefits of AIOps

    Staffing levels within IT operations (ITOps) departments are flat or declining, enterprise IT environments get more complex by the day and the transition to the cloud is accelerating. Meanwhile, the volume of data generated by monitoring and alerting systems is skyrocketing, and Ops teams are under pressure to respond to incidents more quickly.

    Faced with these challenges, companies are increasingly turning to AIOps – the use of machine learning and artificial intelligence to analyze large volumes of IT operations data – to help automate and optimize IT operations. But before investing in a new technology, leaders want assurances that it will bring value to end users, customers and the business at large.

    Leaders looking to measure the benefits of AIOps and build KPIs for both IT and business audiences should focus on key factors such as uptime, incident response and remediation time and predictive maintenance to prevent outages that could affect employees and customers.

    AIOps KPIs include employee productivity, customer satisfaction and web site metrics such as conversion rate or lead generation. AIOps can help companies cut IT operations costs through automation and rapid analysis; it can support revenue growth by enabling business processes to run smoothly and deliver excellent user experiences.

    Specific IT and Business Benefits

    AIOps can digest operational data and spit out actionable recommendations for keeping important systems running at peak efficiency. These are the most-cited benefits of the technology:

    • Alert management: In most cases, the first problem AIOps systems will address is reducing the volume of noise: the torrent of alerts that inundate IT operations. AIOps uses clustering and pattern matching algorithms to eliminate as much as 90% of false alarms and other redundant or irrelevant alerts, making it far easier for staffers to focus on what really matters.
    • Incident prioritization and routing: AIOps systems can learn, over time, which types of alerts should be sent to which teams, reducing redundancy and confusion when, say, networking and database teams both get the same alert related to an incident.
    • Event correlation: AIOps can correlate alerts and event data to identify the root cause of an outage or application slowdown so that IT teams can respond faster.
    • Advanced anomaly detection: AIOps systems can identify anomalies to proactively detect abnormal conditions and relate them to business impact. For example, AIOps can predict if a system will run out of disk space based on projected growth or seasonal patterns, even if the growth is non-linear. Or, if there’s a sudden increase in the number of failed server requests, the technology can determine whether the server in question is handling a mission critical task or simply performing routine backups.
    • Automation: AIOps can be used to handle routine tasks such as backups, server restarts and other low-risk maintenance activities which otherwise involve heavy manual effort.
    • Predictive analytics: A more advanced use case could involve predicting events before they happen – such as detecting when network bandwidth is reaching its limit or storage capacity is nearing threshold.

    Seven KPIs for AIOps

    These common KPIs can measure the impact of AIOps on business processes:

    1. Mean time to detect (MTTD): This KPI refers to how quickly it takes for an issue to be identified. AIOps can help companies drive down MTTD through the use of machine learning to detect patterns, block out noise and identify outages. Amid an avalanche of alerts, ITOps can better understand the importance and scope of an issue, which leads to faster identification of an incident, reduced downtime and better performance of business processes.
    2. Mean time to acknowledge (MTTA): Once an issue has been detected, IT teams need to acknowledge the issue and determine who will address it. AIOps can use machine learning to automate that decision-making process and quickly make sure that the right teams are working on the problem.
    3. Mean time to restore/resolve (MTTR): When a key business process or application goes down, speedy restoration of service is key. ITOps plays an important role in using machine learning to understand if the issue has been seen previously and, based on past experiences, to recommend the most effective way to get the service back up and running.
    4. Service Availability: Often expressed in terms of percentage of uptime over a period of time or outage minutes per period of time, AIOps can help boost service availability through the application of predictive maintenance.
    5. Percentage of automated versus manual resolution: Increasingly, organizations are leveraging intelligent automation to resolve issues without manual intervention. Machine learning systems can be trained to identify patterns, apply solutions, such as previous scripts that were executed to remedy a problem, and take the place of a human operator.
    6. User Reported versus Monitoring Detected: IT operations should be able to detect and remediate a problem before the end user is even aware of it. For example, if application performance or web site performance is slowing down, ITOps should get an alert when the issue can be measured in milliseconds; they can then fix the issue before the slowdown worsens and affects users. AIOps enables the use of dynamic thresholds to ensure that alerts are generated automatically and routed to the correct team for investigation, or automatically remediated when policies dictate.
    7. Time savings and associated cost savings: The use of AIOps, whether it’s to perform automation or more quickly identify and resolve issues, will result in both operator time savings and business time-to-value improvements. These have a direct impact on the bottom line.

    These seven AIOps KPIs can be correlated to business KPIs around user experience, application performance, customer satisfaction, improved e-commerce sales, employee productivity and increased revenue. ITOps teams need the ability to quickly connect the dots between infrastructure and business metrics so that IT is prioritizing spend and effort on real business needs. In the future, as machine learning matures, AIOps tools may even recommend ways to improve business outcomes or provide insights as to why digital programs did or did not succeed.

  • Building Your Own AIOps Platform Is a Bad Idea

    Building Your Own AIOps Platform Is a Bad Idea

    It takes considerable expertise to not only build, integrate and maintain an AIOps platform

    In the wake of the economic downturn brought on by the COVID-19 pandemic, investments in digital business transformation have accelerated. The applications that drive those processes are not only highly distributed, but they also operate at a level of scale that no IT team can manage using legacy approaches to managing IT. It can take weeks to discover the root cause of any issue.

    Enter AIOps. Machine learning algorithms make it possible to reduce the time it takes to resolve any issue and enable IT teams to continuously optimize IT environments at any scale. Many aspects of AIOps, however, are still largely unexplored. Rather than opting for proven platforms, some IT teams are building custom solutions in-house.

    I have experienced firsthand several large enterprises that have embarked on this risky journey, including one Fortune 500 company that asked a partner to help develop a solution and ultimately deployed a commercial product. Enterprise IT teams go down this road to solve a specific tactical problem, such as alert noise reduction. In many cases, all the time and effort put into a do-it-yourself (DIY) project simply winds up being wasted.

    Why DIY Platforms Usually Fail

    Given the prevalence of open source AI tools and frameworks such as TensorFlow, Theano or the Microsoft Cognitive Toolkit (CNTK), it can be tempting to build your own custom AIOps platform. It takes considerable expertise, however, to not only build an AIOps platform but also integrate it into an enterprise and maintain it.

    Here are the leading reasons why in-house developed AI projects are risky:

    • You’ll need a properly constructed data lake: AIOps platforms require access to data residing in multiple technology silos in real-time. IT teams that build their own AIOps platforms need to make sure they are gathering all the right log data, metrics and traces alongside data collected from IT service and incident management platforms. These comprehensive data sets are needed to train whatever machine learning framework is in place, which is often selected at random. Invariably, that means building or buying a costly big data platform to create a data lake to store all that data. A poorly constructed AIOps platform will be worse than the proverbial disease it’s meant to cure because the insights don’t accurately reflect what’s actually occurring in the IT environment. Do you have the funds for this and experienced data science experts onboard to get this right?
    • Designing AI-enhanced workflows is unlike other workflows: Getting the data is just the beginning. Determining how the system behaves and affects existing workflows is the next step. IT teams must decide to what degree they merely want the AIOps platform to passively surface recommendations based on what’s observed versus automatically resolving issues based on defined parameters.
    • Deployment is complex: After developing a few AIOps algorithms to produce meaningful results, the next step is to determine how to deploy it in a resilient and performant architecture. What other systems does it need to integrate with and how will results be monitored and viewed?
    • Monitoring user impact is critical: How will end users interact with the algorithm and what is the ideal UI/UX and workflow? How will feedback be provided by end users for improvement and adoption success?
    • AIOps support and maintenance is not a project, but a team. Ultimately, an internal IT team would need to build the equivalent of a product which needs ongoing maintenance and support. The total cost of the custom platform starts to rise as the bulk of the IT team could wind up spending most of their time managing the AIOps platform instead of making continual improvements. Even if the IT team has the expertise required to build an AIOps platform, there’s no guarantee those individuals will always be available to maintain and update it. Very few IT professionals spend their entire career at one organization.
    • Keeping pace with marketplace innovation: Finally, AIOps as a field is still relatively nascent and the startup community has hundreds of millions of dollars in VC backing to support R&D. Advances are being made at a rate most internal IT teams can’t keep up with, let alone evaluate and vet on their own.

    How to Safely Encourage AIOps Exploration

    There’s no substitute for knowing where an organization needs to go and how to get there. A commercial AIOps platform incorporates all the best practices that have been defined by legions of IT experts, along with these benefits:

    • Faster time to value: You can embark on the AIOps journey much sooner. A commercial AIOps platform will begin surfacing insights in a matter of weeks. It will take an internal IT team months to build an equivalent platform with no guarantee of success. Time is better spent on user adoption and adding and refining use cases for business benefit.
    • Seasoned experts: A commercial platform provides immediate access to not only a proven framework but also, AIOps experts who can troubleshoot and optimize issues quickly. There’s almost no AIOps challenge they haven’t seen before.

    How to Measure ROI From AIOps

    Savvy organizations that invest in AIOps are primarily betting on a better way to manage IT that will enable them to accomplish more as a business. The real value proposition of any AIOps platform is that it enables an existing IT team to do more not by just eliminating rote tasks but also making it possible to deploy more applications reliably without adding IT staff. It’s worth remembering that the cost of labor continues to be the single biggest IT expense.

    The return on investment from an AIOps platform can be easily calculated by measuring:

    • The number of incidents resolved in a given period of time.
    • The size of the IT operations/incident management staff before and after an AIOps platform is deployed.

    Arguably, however, those savings pale in comparison to the opportunity costs of using advanced technologies which can increase operational intelligence and deliver new business value in the form of excellent user experiences and high-performing digital services. IT organizations that don’t embrace AIOps will be unable to compete with faster, more nimble rivals that have modernized their IT processes. The easier, more affordable and least risky way to go about this is by deploying a solution already proven and in market.

  • It’s Time to Take a Customer-First Approach to AIOps

    It’s Time to Take a Customer-First Approach to AIOps

    Artificial intelligence for IT operations (AIOps) has several use cases that IT operations managers can’t deny: reduce alert noise with statistically significant outcomes (up to 80%), correlate alerts and events to uncover the critical business issues immediately, analyze data across environments to find root causes and resolve routine issues (like patching) automatically. Gartner predicts that large enterprise use of AIOps tools to monitor applications and infrastructure will rise from 5% in 2018 to 30% in 2023.

    I’ve been working in the AIOps field for a few years now, and I’ve learned that many organizations, even with relatively modern IT operations management (ITOM) software, are behind the curve. Vendors keep releasing new algorithms, yet the original ones are barely being used or worse, still sitting on the shelf in the customer’s environment. This is because it can be difficult to understand when and how to use artificial intelligence (AI) tools in the daily workflow, and because IT professionals may not inherently trust (or even want) machine-driven recommendations and actions. 

    AI and machine learning technologies are upending traditional assumptions and conventions in many industries – and IT is no exception. In the land grab to bring to market the latest sensational AIOps capabilities, vendors have unwittingly deprioritized customer adoption. This has resulted in a chasm: The technology is getting ahead of the customer’s willingness and ability to implement these solutions in their production environment. 

    The feature-first perspective in the software industry is nothing new, yet for AIOps to reach its full potential, it’s time to take a customer-first perspective.

    Let’s walk through a few ideas to minimize barriers to adoption and speed up time to value for AIOps:

    Start on Adoption During Build

    Machine learning developers and product managers need to understand potential customer barriers during the design and test phase and put themselves in the user’s shoes. How will an individual interact with the system and what level of transparency is appropriate to ensure a frictionless experience? This is a best practice for any software development team, but given the mystique of AI software, it’s even more important. Document expectations for the user experience by giving users early access to the software, such as a beta program, where they can freely deliver feedback on UI, workflows and outcomes of AIOps features.

    Connect the Dots with Visualizations

    A significant barrier for any machine learning application is that users may not fully understand the use cases nor trust what the algorithms are doing. Ensure that software-driven actions and recommendations are transparent at the moment and location they occur in the application. It should be easy for users to see the cause and effect of machine-driven actions as well as how the AI reached a conclusion.

    The concept of “explainable AI” has become popular in the last year and vendors can help by introducing visualizations into their tools.  For instance, you could have graphs showing the progression of a model being trained or the learned sequences in alert correlations. In-app simulation tools can help a customer see the impact of an AI-delivered recommendation without actually making the change.

    Amp Up Training

    Create internal-facing white papers or videos to demonstrate the workings behind algorithms to your technical staff and tools especially designed for users. The best way to learn is by doing, so designing visual cues and tutorials within the application will be most effective. Ensure that you have dedicated AIOps subject matter experts who can guide customers through implementations and be on call for advice as needed. Common questions include, when should the AI be allowed to automate an action or decision and how can you override an automated decision? It’s not a bad idea to involve machine learning developers in creating training sessions and guides. Finally, the age-old concept of nurturing power users within the customer organization can go a long way.

    Diffuse Concerns Over Jobs

    Nobody wants to use a new technology which might eventually replace them. For the most part, AIOps is not going to jettison IT operations staff. Instead, AIOps will empower operations and support teams to handle increasing alert volume and complexity with existing staff, while ensuring that expensive engineers such as SREs can focus on higher-level data performance analysis and optimization initiatives.

    Design Algorithms for Flexibility

    Machine learning algorithms break if they cannot respond to the dynamic nature of modern workloads. For instance, if you design the algorithm based on rules dictating actions between specific applications and infrastructure components, what happens when those technologies disappear or are replaced, as they often do in the cloud? Now you’ve got gaps that could result in a catastrophic system failure and an AIOps tool that requires heavy maintenance. Algorithms should be responsive to environmental changes – such as incorporating neural net technology that can adapt to the shifting sands of the IT landscape.

    A survey conducted in April by OpsRamp found that nearly 70% of IT operations leaders plan to invest in AIOps to improve incident diagnosis, troubleshooting and resolution. By all accounts, the year 2020 has raised the bar for IT expectations around resiliency and user experience. AIOps promises to be a pivotal strategy for IT operations success, but it all depends upon user satisfaction and adoption of the technology. Vendors that incorporate customer adoption and value tactics into their development processes will be instrumental in pushing this market forward in a positive way.