devopsy.kinsta.cloud

Tag: continuous operations

  • AIOps: Enabling the Customer Experience Evolution

    AIOps: Enabling the Customer Experience Evolution

    How can IT organizations make sure they’re equipped to resolve operational issues quickly and drive better business outcomes? That’s where artificial intelligence for IT operations, or AIOps, comes in. With the increasingly vast amounts of data generated by infrastructure and business applications, and teams often working in disconnected silos, managing and improving operations through automation — including monitoring and service desk processes — is essential.

    AIOps, a term coined by Gartner Research in 2016, refers to the use of analytics and machine learning to aggregate and leverage historical data from a variety of IT operations tools. AIOps platforms can react to issues in real-time, providing intelligent insights that help teams continuously improve core IT functions and prevent future errors.

    Most IT operations, particularly since the coronavirus pandemic, have migrated to online processes (for example, performance monitoring) that gather ever-increasing amounts of data. At the same time, organizations are under mounting pressure to immediately address any infrastructure problems that arise. In a nutshell, the demands and requirements of today’s IT environments far exceed humans’ abilities to identify, act on information and react to issues at scale and speed. To ensure success, these processes require automation.

    AIOps platforms offer IT organizations this essential operational agility. By moving data out of silos, increasing scalability and speed, IT operations can become more agile – this is particularly beneficial to complex global service and logistics operations with massive data sets.

    In addition, AIOps platforms lower costs by reducing dependence on multiple on-premise solutions, as well as eliminating outsourcing costs. The technology allows organizations to scale infrastructure seamlessly, helping the entire service delivery ecosystem run more effectively and efficiently, and thus improving the customer experience. After all, disruptions in manufacturing production or distribution centers could be devastating, both for customer service and the bottom line. By intelligently automating operations, organizations can boost accuracy, predictability and ultimately, customer retention.

    How to Take Advantage of AIOps

    At the heart of an AIOps platform is big data. That means there is a significant amount of preparation necessary to put the pieces of the AIOps puzzle together so your organization can best take advantage of the platform. Here are the most important steps:

    Collect extensive and diverse data. Data serves as the foundation of implementing a successful AIOps effort, so it is essential to understand how data can be brought together and used effectively. IT organizations should collect data from various sources, including on-premise systems, cloud platforms and applications. Ultimately, the data should be stored in a centralized data lake. An AIOps platform does just that, enabling better decision-making and more meaningful analysis that is quick and thorough, thanks to the use of AI.

    Segregate data into meaningful categories. As data is ingested, it needs to be restructured based on the organization’s operational needs. This is important both for historical data as well as real-time ingested data. Depending on the AIOps use case, you should define categories that align with business rules. For example, for a pharmaceutical device manufacturing company, meaningful categories could include equipment health data, device efficiency data and environmental factors.

    Apply AIOps machine learning to initial big data test cases. Any transformation initiative benefits from starting small. The same is true for AIOps efforts: begin by capturing knowledge, applying machine learning capabilities to limited test cases and iterating at greater scale from there.

    Improve prediction accuracy with measurement and feedback. Once the AIOps platform knows the data pattern(s), it can intelligently predict what comes next, including extrapolation from real-time data. The organization can test and measure; supplying feedback to the model to improve predictions. The AIOps platform can then apply logic to the segregated data and design to define the next, best action. AIOps platforms use historical data, as well as learning from new data, to continuously improve and achieve more accurate decision-making. Ultimately, AIOps is about building a continuous feedback and improvement cycle.

    The Journey to AIOps platforms

    Gartner predicts that large enterprises’ use of AIOps, and other digital experience monitoring tools, to monitor applications and infrastructure will rise from 5% in 2018 to 30% in 2023. That trend is only escalating post-pandemic, as the IT landscape migrates even more to online services and operations. Analyzing data, and intelligently automating operations based on that data, will help organizations achieve greater success.

    Finally, it’s important to understand that it doesn’t take a lot of work to successfully enable AIOps. IT organizations already have the data and the means to extract it. A successful AIOps initiative simply requires the right AIOps platform, an experienced development partner and a defined use case based on a desired business benefit.

  • An Embarrassment of Riches

    An Embarrassment of Riches

    History says people rarely appreciate what they have in the heat of the moment. Our highly connected and high-speed world is no exception to this rule. The things that frustrate us are in our faces, while the things that make nearly everything better/safer/faster are taken for granted.

    That is every bit as true in IT as it is in life. We expect our deploys and releases to go easily, because life is far easier in that regard than it was even 10 years ago. We get frustrated when things don’t work out, because they so often do.

    The one area Agile has addressed is that most shops don’t beat the person responsible for an issue until they leave. That’s not to say no one is judgmental, it just isn’t official policy to brow-beat even stupid mistakes. The goal is to get better, not to bash the person responsible for the mistake. Most of the time this is a downright positive change. Particularly since, in complex systems, it is often not one individual’s error, but many.

    Before Agile, I rushed out a fix to a piece of software used by several large banks. They needed the fix, and I was most qualified to do it. I used the file handle for the printer to open a printer, and wrote to it. All went well in test and we shipped it out. The results were catastrophic corruption of one set of databases for several large banks.

    Unknown to me, someone had re-used the printer file handle if certain conditions were met, and opened it on the database. It being reserved for printers, that never occurred to me, and I simple reopened it and used it. The strange conditions that pointed it at the database were never encountered in test, and I was responsible for some serious damage to our reputation and the banks in question’s confidence in our software.

    Thankfully, our development team was already on “do it fast, don’t point fingers,” so it was mentioned that better testing could have been done, and we implemented plans to do just that, using scrubbed customer data. And the root cause–using a reserved file handle for access to something else–was fixed. My own angst as having created the mess made me sit with operations while they rebuilt the destroyed databases one at a time, using some astounding Unix command line wizardry. In the end, the software and the team were better off for the experience. And I got an early lesson in not trusting that code would behave as-expected.

    The point being, bashing me for a mistake that put us all on high-stress because our biggest customers were rightfully angry would not have solved the problem. Letting me work through, identify root causes with the test team and get things straight worked better.

    We’re facing a similar hill in DevOps today. Do you test in production? How much do you test in production? What if you have similar totally unforeseen issues crop up?

    The answer is you must test in production. Inevitably, there will be errors in production, it is far better to find them early and fix them than to test in production because the app is live and there were things you couldn’t test outside of production. Either way, you’re testing in production, the question is how much control do you have.

    This is one of the strong suits of DevOps. Using things such as feature flags, monitoring and AIOps, you can spin up a small percentage of production servers with changes. Or deploy parallel servers in production for low-server-count apps, then just stream a few customers to them to test out new changes. Yes, those customers might have a terrible experience if there is some unforeseen issue, but better a few customers than all of your customers if you rolled out without testing. For some industries, building a list of people willing to be used to test new features like this can be a marketing ploy.

    Our frustration at things going wrong will still be there, but the joy of catching those problems before major catastrophe hits our production servers should be there, too. Not long ago, you rolled it out to everyone and dealt with the consequences. Crashes were big and even more frustrating. Better to have a few customers impacted and the ability to fix issues before it’s an emergency.

    I’ve said it before, we work in astounding times with giant leaps forward in responsiveness. Take advantage of them. And take time away from making and running great software to appreciate them, if only for a moment.

    Then add more features and fix more bugs. We are in the heat of the moment after all.

    — Don Macvittie

  • Continuous Optimization: An Idea Whose Time Has Come?

    Continuous Optimization: An Idea Whose Time Has Come?

    Agile makes development deliver code faster. CI makes testing move faster and CD makes delivery faster, but what keeps track of the volume of change all of this introduces and delivers optimizations faster?

    That is the question Opsani wants you to be asking. And it’s a valid question. Optimization in a complex and constantly changing software world becomes more important–if it can keep up. Opsani thinks it can, with a little AIOps to help anyway.

    The premise is simple. There are ways to adjust the environment that might increase performance and/or reduce costs. Those ways are individually relatively simple (such as JVM settings or cloud instance sizing), but combined they make a complex web that impacts performance as a whole, with individual bits sometimes helping and sometimes hurting.

    Up JVM memory on a large instance and it might help. Switch to a smaller instance later, and that JVM change may cause problems on the smaller footprint, for example. So something more dynamic is needed. As code changes, best optimization options might change also. As the other parts of the environment change, that magnifies the concern that applications are not running at their best and might be costing too much money.

    Opsani’s answer to this problem is continuous optimization. Using AIOps to check different combinations of settings at a rate that operations staff couldn’t manage even if they did have the time. And they generally don’t.

    From my (not their) perspective, there are two pieces to this puzzle. First is gathering inputs and determining performance, second is tweaking settings. Then repeat. Seems simple but the environment continuous optimization would work in is increasingly complex. Any look at how many places optimization can occur in a given environment–from VM/Instance to code to network to management platform to storage/database choices–there is a lot going on, and all of these settings can impact each other.

    Hence the idea of continuous optimization appeals to me. They’ve got the interfaces that one would expect to gather inputs, names such as Jenkins, DataDog and New Relic. They’ve got the interfaces to enact and test out changes too, with interfaces to products such as Kubernetes, Terraform, Rancher and Wavefront. So it certainly appears they’ve got the basics covered. Of course there are vendor interfaces I’d like to see that aren’t in the list of integrations, and no doubt the same is true for you. But the core is here, getting to the monitoring and management tools most companies use regularly.

    Opsani seems to be aware of the different comfort levels of various organizations, also. Able to run in dev or test as well as production. Since every organization has a different taste for putting new ideas into production, this is a good idea, though personally I would aim for eventual production rollout, just because that is where the rubber meets the road. Having it running as part of normal operations means even that small change that reduces responsiveness of your application by 25% (we’ve all seen it) will be detected, and Opsani can try changing some of the environment to compensate.

    When I spoke with them, we did not discuss cost, but we did discuss savings. For budget-sensitive environments, it is worth looking into the cost in your organization versus the potential savings in actual monthly dollars. Here is a screenshot showing the cost measure combined with the performance measure. Reducing monthly overhead to pay for a tool that also improves performance might just be a no-brainer–if it suits your environment.

    I can’t say if it suits your environment, nor can any other analyst/pundit. That’s for your organization to decide. But the idea of continuous optimization is a good one. Opsani is not alone in working on this problem, so even if this particular tool doesn’t suit your needs, you can look for others that do. I recommend getting the demo and seeing what they’ve got, and how it might help in your environment.

    — Don Macvittie

  • Continuous Operations: Implications on IT automation

    Continuous Operations: Implications on IT automation

    With the increased adoption of cloud-based services, users are now accessing their applications through multiple channels and device ‘endpoints’. User experience expectations, which are constantly rising, now have to be met across multiple channels to avoid customers dropping of your website or app and never returning again. Minutes of service downtime or response slowdown may result in thousands of prospects and millions of revenue lost. This imposes enormous challenges on the operations of the applications and services, which have to ensure the permanent availability of applications and services.

    A continuous truth

    This means that operations of applications and data services need to be continuous without interruption. Availability and Performance cannot suffer from ongoing releases, updates and patches – downtime is a luxury of past times. Continuous monitoring of infrastructure, applications and services has to be tied to automated remediation. Events raised from monitoring trigger automated processes to add server nodes, storage or network capacity in a “hot” mode without service interruption or to re-distribute load to different locations and data centers to fix performance issues.

    All of this is impossible without a solid automation architecture. Orchestration of automation is key.

    Orchestration of automation has to ensure that different tools and systems in the operations management chain are properly connected and hand-off points are automated to achieve end-to-end automation across silos.

    The operations management tool chain includes monitoring tools for the different layers as well as network and storage management, server virtualization, database management, application server management and IT service management.

    With the right automation platform in place you will be able to connect these tools into fully automated end-to-end processes and at the same time ensure that these processes can be monitored and controlled in a reliable manner.

    In addition, pay specific attention to automation of release and deployment.

    It is hard to escape the increasing demand for faster releases to constantly introduce new features, improve usability, fix defects and security loopholes. At the same time, service interruption is not an option.

    Only with the right deployment architecture and tools in place you will able to handle this challenge.

    Continuous operations means users never even know upgrades have happened — until that is they start to enjoy the additional functionality experience that comes from the upgrade itself.

    The end-to-end challenge

    The challenge today is an end-to-end one. As Gartner research director Robert Naegle has said, “Gartner research shows that it is still more common to see automation technologies popping up in functional islands, with I&O management later attempting to aggregate and orchestrate disparate technologies, scripts and workflows.”[1]

    The imperative now is one that centers on bringing continuous automation technologies to bear an end-to-end basis. In this way they can facilitate the seamless integration from the business processes through the IT services supporting those processes and the delivery of applications to differentiate the business.

    [1] Gartner, ‘Hype Cycle for I&O Automation, 2015’, Robert Naegle, Gartner, 7th July 2015