devopsy.kinsta.cloud

Tag: AIOps

Dell Updates AIOps Platform to Better Automate IT Ops

Dell Technologies this week updated its IT operations platform infused with artificial intelligence, dubbed CloudIQ, to add support for its PowerEdge servers. In addition, Dell pledged to add support for its PowerSwitch networking hardware next month as part of an ongoing effort to extend the reach of its AIOps platform across the entire Dell EMC portfolio.

At the same time, Dell next month also plans to employ webhooks and a REST application programming interface (API) to both integrate the CloudIQ cloud service with third-party platforms and share notifications and recommended actions via tools and platforms such as ServiceNow, Slack, Microsoft Teams, Ansible and vRealize.

Dell has also defined an autonomous operations framework that defines the various stages of AIOps capabilities that organizations should be able to take advantage of as CloudIQ continues to evolve.

Drew Schulke, vice president of product management for Dell Technologies, said CloudIQ currently enables IT organizations to achieve level three of that framework. As Dell continues to update CloudIQ alongside delivering ongoing infrastructure upgrades, additional capabilities will be steadily added, noted Schulke.

Dell is making a case for standardizing on both its infrastructure and the IT operations platform it provides. Whether acquired and managed by an internal IT team or consumed as-a-service provided by Dell, the goal is to convince IT teams to reduce their reliance on rival infrastructure platforms that are not as tightly integrated with CloudIQ. In effect, Dell is delivering continuous intelligence as an AIOps service, said Schulke.

Of course, IT operations teams could opt to standardize on an AIOps platform that is integrated with infrastructure provided by multiple vendors. However, Dell is betting that an AIOps framework optimized for its platforms will reduce the total cost of IT in a way that IT organizations will not be able to ignore.

Each IT organization will need to decide for themselves what path forward makes the most sense for them. However, as more AI capabilities are infused into IT operations platforms, it’s becoming clear that many of the tasks routinely performed by IT teams are becoming increasingly automated. Most organizations will need to make a decision in the months ahead about the degree to which they are comfortable waiting for their existing IT operations platform to be enhanced with AI capabilities versus acquiring a new one that might be more advanced.

Regardless of the path forward, the level of cultural disruption AIOps will bring to IT teams will be high. DevOps teams, naturally, have always been at the forefront of IT automation. However, most of the recent advances in IT operations have generally been applied in a piecemeal fashion. Platforms such as CloudIQ are designed to be implemented on a more end-to-end basis.

Thate level of automation will have a profound impact on everything from the number of applications an IT team can support to the roles of the individuals that make up that team. It’s not likely that AIOps will replace the need for those IT teams as much as augment them and redefines the job they are being asked to perform.

September 23, 2021
3 Pillars of Autonomous IT Ops

Imagine a future where human IT operators and intelligent systems work together in a virtual operations room; the systems do the heavy lifting while the humans supervise. This may seem like science fiction, but the truth is we are already on the way to that future with AIOps. While actual AI-augmented teams are still quite a way off, AI/ML tools are slowly but surely becoming more ubiquitous within IT Ops, NOC, DevOps and SRE teams and leading us into this not-too-distant future.

In the last five years, the number of internet users grew by 50%. In the last two years alone, more data was generated than in the entirety of human history up to that point. Today, 5 quintillion (that’s 18 zeros) bytes of data are produced by smart devices daily. These massive volumes of data are having a huge impact on digital operations.

Looking at the enterprise, we see similar trends: More tools generating more data and with it, greater complexity. This is due to many factors, including the transition to the cloud, M&A activity that introduces new tools and data into an existing organization and the adoption of new technologies and methodologies such as CI/CD, microservices and more. At some point, there is so much data and everything becomes so complex that IT Ops teams reach a breaking point where they can no longer manually manage the data being generated.

And when that point is reached, outages and service disruption become an everyday reality, as IT teams are pulled away from their strategic, business-critical projects into what is essentially firefighting.

Implementing more observability and monitoring tools simply exacerbates the problem, and scaling teams isn’t cost-effective in the long term. The more forward-thinking solution is automation, and even more so, autonomy.

The Road to IT Ops Autonomy

Autonomy is the ability of IT Ops to operate with minimal to no human intervention. It relies on the automation of the incident management life cycle so that humans are needed only for supervision, in the same way that a single operator can supervise a modern and highly complex robotic car assembly line.

How can we hope to reach IT Ops autonomy? The following three proposed pillars can act as a guide:

Pillar One: Automate

Pillar Two: Predict and prevent

Pillar Three: Democratize

Pillar One: Automate

We must automate as much as possible across every step in every aspect of incident management. It won’t happen overnight, but with time and with the help of AI/ML it is doable.

Consider the autonomous driving analogy.

Just a few years ago, cars were manual, with little to no automation. Then we moved through partial autonomy to where we are today–conditional autonomy–where the car can drive itself in ideal conditions. This is a level of automation that we are beginning to accept; some of us cautiously, others with more enthusiasm! Meanwhile, progress continues toward higher levels of autonomy, in which vehicles will be able to drive themselves in most conditions. One day, maybe, we’ll get to the holy grail of full autonomy, where the car will drive on its own–perhaps we won’t even need a steering wheel!

Similarly, in IT Ops, automation is a process and it will take time to build not only the capabilities required but also the necessary trust in AI/ML technology. Such a journey typically begins with human-driven automation, as “if-then-else” types of automation are added manually. For example, if a certain alert appears, a ticket will automatically be assigned to a specific person. As our confidence grows, the next stage could be AI-driven suggestions, whereby AI applies what it has learned from previous human actions to suggest what to automate, but the human expert decides whether to accept the suggestion or not.

At some point, we will progress to allowing AI to partially automate some tasks—perhaps those that are less risky, such as collecting diagnostic information and adding it to the alert, so that the incident responder automatically receives both together. We may even automate some parts of our system; perhaps within a staging environment that isn’t production-critical and, as such, is a good place to validate that the automation is working as expected. Over time, as our AI and ML capabilities develop and our trust in them grows, we may eventually reach full AI-driven automation, with no human involvement at all—the AI will simply detect the fault and resolve it.

Pillar Two: Predict and Prevent

Typically, when assigned an incident, a responder will take note of the incident’s priority. But they will also rely on their common sense and past experience in deciding how to respond. So, for example, if a P3 incident had recently escalated to a P1 in 30 minutes and a similar P3 incident now appears again, our responder will probably prioritize it, even over other incidents, to avoid a similar escalation.

While this is one of the advantages of human logic, the flip side is that the IT environments in which we work are vast and complex—no human can possibly remember every potentially relevant detail. What’s more, if a different responder dealt with the earlier incident in our example, the current responder would not have the same insight. What is needed is an element of enrichment that can input leading indicators into the platform for the next operator to see and act upon to prevent an incident.

This is where AI can help.

By identifying patterns in the data haystack—linking different incidents to the same root cause, for example—AI not only generates insights that help resolve these incidents faster but can also predict a certain outcome. This is done by anticipating a future chain of events so that appropriate preventative action can be taken before the event occurs. For now, this preventative action may be manually taken by the operator, but at some point in the future automation may enable the problem to be identified and fixed by the AI with no human intervention before it results in an outage.

Pillar Three: Democratize

The last pillar that will help us achieve full autonomy in IT Ops is democratization of AI. AIOps should not just be accessible to advanced, mature organizations with an army of data scientists working behind the scenes. Just as anyone can accomplish almost anything online these days with just a few clicks, any organization should be able to adopt AIOps and enjoy its benefits!

Democratization is about making relevant information accessible to anyone who needs it, when they need it, and in a manner that is easily applicable and actionable. AI should be accessible through simple user interfaces, with no need for coding; the benefits of machine learning should be available to people that don’t (need to) understand machine learning models.

This democratization can be achieved by making AIOps platforms simple and accessible to all—from the admin to the user. For example, an AIOps platform might automatically suggest how admins should prioritize incidents for operators, to make triage easier and provide instructions that even a non-expert can follow. The more such features are built into AIOps platforms, the more democratized AIOps will become.

Data volumes will continue to grow exponentially in the coming years, and the only way to eventually conquer the tyranny of overwhelming data will be to reach autonomy in IT Ops.

If you’re an AIOps vendor, keep the three pillars in mind so your customers can continue to progress towards autonomy. If you’re an IT Ops leader or tooling architect, and IT Ops autonomy is part of your vision, make sure that your AIOps vendors’ platforms and tools have been architected with these pillars and their principles in mind.

How far are we from achieving autonomy? Only time will tell. But one thing is for sure—we will get there, and if the COVID-19 pandemic has taught us anything it’s that we may be seeing machine-augmented teams sooner rather than later.

September 10, 2021
Why You Should Use GitOps to Experiment With AI

As the pace at which artificial intelligence (AI) models are being constructed and inevitably updated starts to increase, it’s becoming more apparent that the pace at which data science teams are currently experimenting also needs to dramatically quicken.

Many data science teams today are fortunate if they manage to successfully deploy two AI models in a production environment over the course of 12 months. It’s already apparent, however, thanks in part to the rise of digital business transformation, that there are potentially hundreds of use cases for AI models within each enterprise IT environment. Truth be told, it’s hard to envision an application being developed today that would not incorporate one or more AI models. The challenge is, the demand for AI capabilities is now far outpacing the ability of data science teams to deliver.

The best way to solve this issue is to recognize that AI models and the components employed to construct them are, in reality, just another kind of software artifact. In fact, the need to accelerate the development of applications using artifacts is largely a problem already solved. Rather than reinventing the wheel, the time has come to apply what we already know to the development of AI models.

The Rise of GitOps-Based Processes

Modern software development teams today routinely make use of Git-based repositories to share code and collaborate. These repositories are jam-packed with vetted software artifacts that development teams continuously reuse via an opinionated instance of a DevOps workflow based on a set of GitOps best practices.

As a subset of DevOps, a GitOps workflow enables data science teams to employ a Git repository that already exists within their software development environment to manage artifacts. Git enables versions of files to be created using what is known as a Git branch. They also employ concepts known as tags (Git Tag) and remotes (where the server is located). Those three different concepts, or references, exist under the hood in Git. In fact, the entire flow is based on those concepts. When a new feature is implemented, a new branch is created and then merged into the production branch with a tag that says something like “This is a new version that needs to be deployed.”

An AI model, like any other software artifact, needs to be updated. As new data sources become available, an AI model is likely to drift beyond the parameters originally intended. A rapid change in business conditions may also require changes to an AI model. The same processes that DevOps teams apply to update any software artifact can also be applied to rapidly update an AI model. Employing a Git repository to also manage AI artifacts extends the benefits that software development teams enjoy to data science teams.

Often, organizations will have an MLOps solution to streamline AI model development that is completely separate from their DevOps tools. These solutions will have dedicated features like version tracking for data and experiments specifically for AI models. But many MLOps solutions exist in a closed ecosystem, meaning they don’t work off of existing tools like GitHub used on the DevOps side.

Also, data science teams have to separately buy, deploy, and manage these closed MLOps solutions. Going with a GitOps-oriented solution streamlines management of added MLOps tools and makes it easier for both application developers and data science teams to gain visibility into each other’s respective efforts. The end result is not just faster AI experimentation at lower cost; the likelihood that more AI models will successfully be deployed in a production environment increases dramatically.

Software and AI Teams Working Together

The convergence of DevOps and MLOps is already inevitable. The longer organizations continue to employ isolated MLOps platforms to build and deploy AI models, the more expensive it will become to meld MLOps and DevOps processes.

Data science teams are conservative by nature. The difference between success and failure often comes down to how thoroughly an AI model is validated. However, time to market doesn’t have to be sacrificed to ensure an AI model works as promised. There is no such thing as a great AI model that arrives late when rivals are racing to develop similar AI capabilities—there are only missed opportunities. Speed enabled by the reusability of artifacts that drives faster experimentation using GitOps workflows is of the essence now more than ever.

August 23, 2021
SREs Say AIOps Doesn’t Live Up to the Hype

What can you expect when investing in artificial intelligence for IT operations (AIOps)? Real-time visibility across huge volumes of information. Lightning-fast event correlation and anomaly detection. Automated remediation and self-healing, without Ops personnel having to lift a finger.

It all sounds amazing—at least, if you believe the marketing hype. But do those kinds of AIOps benefits actually translate to the real world? According to a recent survey of nearly 300 site reliability engineers (SREs), not so much.

Among SREs surveyed—including engineers for some of the world’s largest, most innovative digital companies—just 12% use AIOps as a day-to-day part of their monitoring toolkit. Nearly 40% never use it at all.

Why aren’t current AIOps solutions living up to the hype? And what can this gap tell us about the state of IT monitoring in general? Let’s take a closer look.

Sprawling Complexity

In theory, AIOps should be a perfect fit for SREs. These are, after all, the people with one foot in the world of operations, one in development, whose job is to make distributed software systems ultra-scalable and reliable. A solution that can turn mountains of data about those systems into actionable insights should be exactly what SREs need. And, in fact, 53% of SREs cite lack of unified visibility across the stack as their biggest cloud application monitoring challenge.

And yet, if we gauge by utilization, today’s AIOps tools don’t seem to be delivering on that promise. Asked about the value of AIOps, just 7.5% of SREs, on average, view AIOps as “high-value”—even less than those SREs (9%) who perceive its value as low.

Why is AIOps so slow to catch on? Ultimately, the barriers facing these tools are the same as those facing human engineers: massive and growing complexity in IT environments. As digital products become more dependent on third-party cloud services, as the number of things businesses want to track grows (from infrastructure to application to experience), the sheer volume, velocity and variety of monitoring data has exploded.

Diverse Data Sources

Compounding the problem, enterprises increasingly rely on multiple “same-service” providers for IT services. That is, they use multiple cloud providers, multiple DNS providers, multiple API providers, etc. There are sound business reasons for doing so, such as adding resiliency and drawing on different vendors’ strengths in different areas. But even when two providers are doing basically the same thing, they use different interfaces and instrumentation, and their data sources often employ different metrics, data structures, and taxonomies.

Whether you’re asking a human being or an AI-driven tool to solve this problem, this heterogeneity makes it extremely difficult to visualize the complete picture across the infrastructure. It also creates gray areas around how best to take advantage of each vendor’s different rules and toolsets. All of which makes automating anything beyond the most superficial operations tasks hugely challenging, if it’s possible at all.

Optimizing AIOps

AIOps really is the solution to this problem—or at least, an important part of the solution. Machine learning is, after all, tailor-made to analyze large volumes of observational and engagement data and react in real-time, as well as to continually identify opportunities for improvement. But before we can get there, we need to change the way we think about AIOps.

Don’t try to do everything at once.

It would be great if we could deploy AIOps, let it learn about our environment and watch it magically start remediating issues on its own. But we’re not there yet. (And if a vendor tries to convince you we are, you should definitely be skeptical.) If we break down our expectations into smaller, more manageable chunks, however, AIOps can accomplish quite a lot.

Event correlation, topology analytics and smart alerting, for example, are all areas where AIOps can provide significant value right now. For these more limited objectives, AIOps can absolutely reduce tedious manual effort for human operators, reduce mean time to repair, improve visibility and more, and digital businesses should be leaning on AIOps to do that.

Invest in AI and ML training.

Part of the reason adoption of AIOps tools remains sluggish stems from the fact that many teams still just don’t fully understand how they can use them. The more SREs are exposed to AI/ML training, the more opportunities they will identify for concrete, meaningful applications of AIOps. For example, only 27% of SREs said, “Experimenting or receiving training to expand knowledge or skills” was a major activity as part of their job.

Embrace “Platform Ops.”

The biggest step we can take is to rethink how we use AIOps within an organization. Today, AIOps tools—like SRE resources themselves—are often dedicated to a specific digital product or service. As a result, enterprises end up trying to apply their AIOps tools, and the SREs using them, to support everything associated with that product (DNS, API, CDN, cloud monitoring, etc). As IT environments continue to get even more complex, though, businesses will likely find that shifting to shared platform tools and teams yields better results.

Instead of trying to apply AIOps across every part of a digital service, imagine applying it to one part of that service (for instance, focusing on just DNS, or just APIs). In this way, the AIOps tools, as well as the teams using them, gain in-depth expertise across all the different interfaces and instruments used for that function and get continually better at normalizing multi-provider data sources across them. At that point, this platform-specific AIOps capability can then be scaled and reused across the company—providing more advanced DNS AIOps (or API, or CDN, etc.) for multiple digital products and teams.

Now, you’ve got an advanced, highly specialized toolset for optimizing an important platform function, regardless of which digital product happens to be using it. You’ve narrowed the scope of how you’re using a given AIOps implementation while going much deeper into that function than a more generic, broad-based implementation could achieve. All of a sudden, even more advanced automation and self-healing becomes possible. And the loftiest marketing claims of AIOps start to look less like hype and more like viable real-world outcomes for your business.

June 24, 2021
Why AIOps Is Critical for Pandemic Business Recovery

As businesses worked to stay afloat over the last year, innovation in many areas fell behind. Business leaders struggled to understand the short-term and long-term impact the COVID-19 pandemic would have on their business, while employees worried about losing their jobs and customers scrambled to rearrange budgets. As we near the end and can finally see light at the end of the tunnel, business leaders are looking for ways to get back on track.

Artificial intelligence for IT operations (AIOps) is the solution many businesses need to free up resources and return their focus to innovation. The data speaks for itself—the AIOps market was valued at $13.51 billion in 2020 and is expected to grow by more than 200% to $40.51 billion over the next five years. AIOps presents several benefits to enterprises, like automating mundane tasks, providing actionable reporting and improving risk management.

But before IT leaders jump headfirst into adopting AIOps, there’s still one common misconception they must address: AIOps isn’t a solution, it’s a long-term strategy. AIOps simply cannot be viewed as a band-aid on the road to recovery. To truly maximize the benefits and bring innovation back to the center of business focus post-pandemic, teams must view AIOps as a long-term strategic investment.

For IT teams looking to adopt an AIOps strategy, consider these three takeaways to jumpstart the journey and maintain it in the long term.

Find a Partner to Offer Market Insight

On the road to recovery, IT leaders may not have the time or bandwidth to truly invest what it takes to implement a robust AIOps investment strategy—let alone where to start. But working with a trusted, experienced partner who does have the time and resources to map out a plan and who keeps a close eye on market trends can help ease any struggles that IT leaders may face.

A strategic partner can also help teams identify what tools IT teams already have in place that will support an AIOps strategy and which will stand in the way of it. Through a comprehensive tools rationalization process, IT teams have a better grasp of what tools they have in place, which are performing overlapping capabilities (in other words, increasing unnecessary costs) and what tools they still need to support their AIOps journey.

The right partner can plot a path forward for IT leaders and give them peace of mind knowing they have an expert in their back pocket at all times.

Identify Use Cases

Use cases for AIOps may have changed over the course of the last year depending on a company’s place in the market, product focus and future growth plans. But as companies bring innovation back to the forefront, they must reevaluate their use cases for AIOps. If you’re an IT leader looking to invest in AIOps, ask yourself ‘Why?’ and determine precisely how AIOps can support that end goal. Your trusted partner can help by giving a different perspective that IT leaders may not have, simply because they are in the weeds day in and day out.

Build a Plan and Execute

With the right tools in place and a partner by your side, it’s time to lay out your complete AIOps investment plan with actionable steps in place. Consider the use cases you determined and your short- and long-term goals for AIOps as a starting point. Remember—this isn’t a plan with a definitive endpoint, but rather a strategy with definitive milestones that can help keep the business up and running and customers happy while maximizing internal processes.

As IT leaders look ahead, they can regain confidence and feel empowered to bring innovation back to the business. No longer should they feel like they’re drowning in incidents and constantly putting out fires. With a long-term AIOps plan in place, IT teams are equipped to thrive post-pandemic and come out on the other side stronger than ever.

June 4, 2021
Dear Vendor: Letter to Technology Vendors

While I was writing my “Dear Management …” post, something was going on behind the scenes that inspired me to write another. We all have things we wish to say to vendors; this open letter picks some of the big points that they seem to invariably struggle with.

Dear Vendor,

We want to use your products, otherwise we wouldn’t even be writing to you. But depending upon who you are, and what year it is, you are struggling to meet our needs.

We understand that your insistence that your new product or acquisition will solve all of the problems we don’t have is probably executive-driven, but give us the benefit of the doubt. We know our environment, and just because we use you for one thing does not automatically imply we want you to constantly push this other thing at us. By all means, tell us about your entire product line, but then back off. We will let you know if anything appears to fulfill a need.

While we’re chatting, can we discuss pricing? Not just cost, but the byzantine structures you create, time after time? If you are going to use 15 different charges and fees to jack up the price there, we’re paying the bill – we’ll see it. You’re far better served by being forthright. Do you have any idea how many stories like, “I am not funding your Ferrari with your commission,” stories (this was actually said; it was hilarious – we went with a competitor) are out there? And they keep occurring. Just this week, the phrase “Your pricing is bonkers!” was submitted to a company. We want a good product at a fair price, not a cheap price up front that will drain all of our budget over time.

Speaking of good products, let’s talk about security. Security issues with your product are our issues more urgently than they are yours. Don’t mess around; when notified, quantify the threat and notify us immediately, with suggested remediations until you can get a patch ready. Holding off until you can patch puts us in the unenviable position of being vulnerable and, worse, not even knowing we’re vulnerable. Use good coding practices so we’re not all caught off guard, and waive those secure coding requirements for no one. Interns and C-level execs are not professional coders (interns aren’t yet, and C-levels no longer are), and any code by these two groups needs to be reviewed by someone that knows your systems before they are sent to us for deployment. Don’t force bad updates on us. Do your best not to give us bad updates at all.

Finally, on the topic of forcing things on us – guess what? We have a need for a specific tool in a specific environment. Telling us that starting next month your services are all “moving to the cloud” does not suit all of our needs. We get it, you want that data. It’s all the rage right now. You should understand that, if you stop serving our needs in your quest for our data, we’ll go elsewhere, eventually, no matter how reluctantly. Cloud is a great platform, but most IT tools need the versatility to run where we need them, not just on one platform out of many.

And, as a side note, your AIOps generally isn’t. We know it’s the buzzword du jour, but honestly, if it’s basically the same product with some logic or a dashboard strapped on, that’s not AIOps, it’s normal product improvement. When you have deep data collection and analysis that uses trained algorithms, then you can join the crowd screaming AIOps. Until then, hush. You sound silly, and it undermines the rest of your messaging.

Just to reiterate: We have a need, show us how you fill it, and let us decide if that suits our requirements. Then tell us about your other offerings and back off. We’re smart enough to smell vaporware, stubborn enough to resist your desire to collect all of our data and wise enough to judge whether or not you and the product you’re pushing this week will be around in a year or two. You serve a purpose, and we’re grateful you are doing this stuff so we can focus on our business. But please, make life easier on us, not harder. And if one more of you sells a product we don’t need to the C-suite, we may explode. That Sword of Damocles that you raised over us will inevitably fall, and your company’s reputation will suffer for it. But we’ll pay the price for trying to implement something we didn’t need in the first place.

May 13, 2021
Why Real-Time Asset Intelligence is a Must

Complexity breaks correlation. Intelligence brings cohesion. This simple principle is what makes real-time asset intelligence a must-have for AIOps that is meant to diffuse complexity. To further create a context for the user, it is critical to understand service dependencies and correlate alerts across the stack to resolve incidents.

Why not just CMDB?

CMDB systems have been useful to break down configuration items into logical layers. But that is not enough, because they can become outdated very soon.

That is when connectivity among assets and their dependencies matter to remain relevant in real-time. This opens new possibilities for an enterprise to tap into. IT monitoring, change management, asset management and dependency mapping for troubleshooting are some areas that have a direct positive impact when enterprises have access to real-time asset intelligence linked to CMDB.

If your CMDB is not accurate with data management systems, it is time you start exploring asset intelligence and asset discovery tools.

Why is Asset Intelligence Critical to AIOps?

Asset intelligence is a critical part of AIOps. Unless you have a log and know about the assets that you have and how they are connected to each other, you cannot correlate to each other very easily. And that’s why real-time asset intelligence is required.

Third-party asset and dependency intelligence is an obvious next step, but those come with more challenges and complexities. Depending on a third party for asset and dependency intelligence would require a continuous feed of data on a regular basis. This process gets expensive and obsolete quickly.

Real-Time Asset Intelligence opens other Possibilities

With automation and discovery tools, CMDB can be kept up to date, in real-time. Real-time asset intelligence with automation enabled by AIOps opens many possibilities, and it contributes many benefits to an enterprise. These include:

IT Monitoring

For alerts and notifications, just a monitoring tool is not enough. The information sent by such a monitoring tool is not up to date. People rely on manual feeds, which are expensive and are not synchronized. Asset intelligence brings forth deep information that builds context which would otherwise be missing. Asset intelligence can point to issues corresponding to servers, topologies, networks, storage, apps or services.

Risk & Compliance Management

An organization must audit IT assets periodically to eliminate risks from the end of life/end of support cycle, vulnerabilities and ensure compliance with regulations and local governing laws. IT asset information is also maintained in many different places and in various forms, in spreadsheets, databases, ERP or CRM, or other systems, which makes it difficult to gather this information. All these challenges can easily increase compliance and vulnerability risks for any organization. AIOps tools automate the process of data collection, analysis and reporting, significantly reducing the operations costs while improving the accuracy and timeliness of the insights.

Change Management

No one has a clue if anything changed in the IT infrastructure. And, even if something does change, no one will know unless updates are alerted and stakeholders notified. With asset intelligence, you will not only know about the change, but before making the change, the solution will also tell you if a change is – or needs to be – made to a business service, as well as that change’s impact on other dependent business services. This helps you evaluate the impact and take measures to contain the blast radius before making any changes.

Asset Management

Asset intelligence provides a 360-degree view of asset inventory, dependencies, life cycle, performance, utilization and compliance. It provides thorough insight into, ‘What do we have versus what is out of date?’ You get to compare those states and inform compliance.

Dependency Mapping

AIOps-enabled asset intelligence also allows for dependency mapping, which is especially useful for troubleshooting work. With machine learning and artificial intelligence, complex environments can be quickly and more easily deciphered, and troubleshooters would know where issues lie. Asset intelligence tells the troubleshooter the most complex part of the problem and what should be addressed or what needs more attention. It maps dependencies that are affected by alert correlation, triaging, planning, and ticket resolution.

May 3, 2021
Achieving Reliable Observability Part 1 – Making Cloud-Native Observability More Robust
I was having a conversation with a CxO level customer as part of an AIOps/Observability workshop, and from what I could tell, most are confused about how to properly operationalize cloud-native production environments – especially the monitoring/observability portion. Here is how the conversation went.

“Andy, we are thinking about getting [vendor] to use for our observability solution based on your recent research. What do you think?”

“Well, I don’t want to endorse any specific vendor, as they are all good at what they do. But let’s talk about what you want to do, and what they can do for you, so you can figure out whether or not they are the right fit for you.” The conversation continued for a while, but the last piece is worthy of being called out specifically.

“So, we will be running our production microservices in AWS in the ____ region. And we are planning to use this particular observability provider to monitor our Kubernetes clusters.”

“Couple of items to discuss. First, you realize that this particular provider you are speaking of also runs in the same region of the same cloud provider as yours, right?”

“We didn’t know that. Is that going to be a problem?”

“Not particularly. However, you may get into a ‘circular dependency’ situation.”

“What is that?”

“Well, as an enterprise architect, I always call for separation of duties as a best practice. For example, having your developer testing the code is a bad idea, having your developer figuring out how to deploy is a bad idea. In much the same way as when your production services run in the same region as your monitoring software – how would you know about a production outage if the cloud region takes a hit, and your observability solution goes down at the same time your production services do?”

“Should we dump them and go get this other solution instead?”

“No, I am not saying that. Figure out what you are trying to achieve and have a plan for it. Selection of an observability tool should fit your overall strategy.”

For those who don’t understand the above conversation, here is the reason why this scenario could be a problem.

Coming from an enterprise architecture background, we were taught, as a best practice, to operationalize production systems to avoid circular dependencies. This includes not having two services depend on each other, or not to colocate monitoring, governance and compliance systems as part of the production systems themselves. If you were to monitor your production system, you would do it from a separate and isolated sub-system (server, data center rack, sub-net, etc.) to make sure that if your production system goes down, the monitoring system doesn’t go down, too. The same goes for public cloud regions – although it’s unlikely, individual regions and services do experience outages. If your production infrastructure is running on the same services in the same region as your SaaS monitoring provider, not only won’t you be aware that your production systems are down, but you also won’t have the data to analyze what went wrong. The whole idea behind having a good observability system is to quickly know when things went bad, what went wrong, where the problem is and why it happened so you can quickly fix it. You can check out this blog where I explain this in detail.

The best practice would be to either:
1. Consider an observability solution that runs in a different region than your production workloads. Better yet, consider something that runs on a different cloud provider altogether. Although it is exceptionally rare, there have been instances of cloud service outages that cross regions. The chances of both cloud providers going down at the same time would be slim.For example, if the cloud region goes down (a region-wide outage in the cloud is quite possible, and seems to be more frequent of late), then your observability systems will also be down. You wouldn’t even know your production servers are down to switch to your backup systems unless you have a “hot” production backup. Not only will your customers find out about your outage before you do, but you won’t even be able to initiate your playbook, as you are not even aware that your production servers are down.
2. Consider having your observability solution in a different location/region, yet still close enough to your production services so latency is very low. Most cloud providers operate in close proximity, so it is easy to find one.
3. Another option is to get a solution that gives you deployment flexibility. For example, there are a couple of observability solutions that allow you to deploy it in any cloud and observe your production systems from anywhere – both in the cloud and on-premises.
4. You can also consider sending the monitoring data from your instrumentation to two observability solution locations, but that will cost you slightly more. Or, ask what the vendor’s business continuity/disaster recovery plans are. While some think the costs might be much higher, I disagree, for a couple of reasons. First, because monitoring is mainly time-series metric data, so the volume and the cost to transport is not as high as logs or traces. Second, unless your observability provider is VPC peered, the chances are your data will be routed through the internet even though they are hosted in the same cloud provider. Hence, there will not be much more additional cost. Having observability data, all the time, about your production system is very critical during outages.
5. A very commonly overlooked consideration is monitoring your full-stack observability system. While it is preferable to have the monitoring instance in every region where your production services run, it may not be feasible either because of cost or manageability. On such occasions, monitor the monitoring system. You could do synthetic monitoring by checking the monitoring API endpoints (or do random data inputs and check to ensure it worked) to make sure that your monitoring system is properly watching your production system. Better yet, find a monitoring vendor that will do this on your behalf.
When/if you get the dreaded 2 a.m. call, what is your plan of action? Just think it through thoroughly before it happens and have a playbook ready, so you won’t have to panic in a crisis.
March 30, 2021
Cisco Leans on HashiCorp to Manage Infrastructure as Code

HashiCorp and Cisco today announced a Cisco Intersight Service for Terraform offering based on HashiCorp Terraform Cloud Business software.

Cisco Intersight is an IT operations management platform that enables IT teams to manage IT infrastructure across a hybrid cloud computing environment. Open source Terraform tools have emerged as the dominant tool for provisioning infrastructure as code in the cloud and, increasingly, in on-premises IT environments.

HashiCorp Terraform Cloud Business is an instance of Terraform that has been extended to include some additional security and governance capabilities enterprise IT organizations typically require.

In recent years, Cisco has significantly expanded its presence in on-premises IT environments as many IT teams opted to deploy its servers alongside its networking equipment. Like every other provider of on-premises IT environment, Cisco has been moving to extend the reach of its management frameworks to public clouds to help foster the rise of hybrid cloud computing.

Prior to the pandemic, most public clouds and on-premises IT environments were managed in isolation from one another. However, in the wake of the economic downturn brought on by the pandemic, more IT teams are trying to centralize the management of IT platforms as part of an ongoing effort to reduce the total cost of IT. The multiyear alliance with HashiCorp fills a major hole in the Cisco management portfolio, as Terraform increasingly becomes a de facto standard for provisioning IT infrastructure-as-code.

Vijay Venugopal, head of product management for Cisco Cloud and Compute Software, said the alliance with HashiCorp will make it possible to provide Terraform support across a management portfolio that, over time, continues to be enhanced with machine learning algorithms that add artificial intelligence (AI) capabilities, also known as AIOps. That platform is already being employed by 6,000 customers to manage more than 600,000 servers, running primarily in on-premises IT environments, added Venugopal.

It’s unclear how the battle for control over hybrid cloud computing will ultimately play out. While providers of on-premises platforms such as Cisco are making a case for using existing management frameworks to also manage workloads deployed on public clouds, the providers of those platforms are now extending their reach into on-premises IT environments. Cisco is, of course, betting that most organizations would prefer to extend an existing framework to achieve that goal, versus having to rip and replace existing IT infrastructure to employ an alternative framework from a cloud service provider.

Regardless of how organizations ultimately decide to manage IT in the age of the hybrid cloud, it’s clear developers, thanks to tools such as Terraform, are exercising more direct control over IT infrastructure. Incorporating Terraform within a larger IT management framework presents an opportunity to bridge the divide between developers and IT operations teams that have historically relied on visual tools to provision infrastructure. The challenge now is finding a way for developers and IT administrators – with varying levels of programming expertise – to work more closely as management of IT environments become increasingly more complex.

March 25, 2021
5 Best Machine Learning Use Cases

Many industries are struggling with the ongoing COVID-19 pandemic, and the IT industry itself and the broader trend of transition to remote work over the last year revealed many areas where traditional approaches to managing businesses created unnecessary waste. Data science and its counterpart, machine learning, revealed that expansion in the ways technology can facilitate new ways of working is nearly limitless.

Machine learning uses powerful algorithms to discover insights based on real-world data. These insights can then be used to make predictions about future outcomes. As new data becomes available, machine learning-enabled programs can automatically adapt and produce updated predictions. As with any tool, machine learning is not a silver bullet. However, there are many situations in which the technology can outperform linear and statistical algorithms.

Here are five of the most common use cases where machine learning can make a big difference:

When engineers can’t code rules for certain problems. Many human-oriented tasks (such as recognizing whether an email is spam) aren’t solvable using simple (deterministic), rule-based solutions. Because so many factors may influence an answer, engineers would have to write and frequently update billions of lines of code. In addition, when rules depend multiple factors and when those rules overlap or need fine-tuning, it becomes difficult for humans to code precise rules. Fortunately, machine learning programs don’t require users to encode actual patterns. These programs only need proper algorithms to extract patterns automatically.

When you need to scale a solution to millions of cases. In finance, for example, you might be able to manually categorize a few hundred payments as either fraudulent or not. However, this becomes tedious or impossible when dealing with millions of transactions. As user bases grow, it’s no longer feasible for organizations to process payments by hand – end-users today want answers about their money in milliseconds, not minutes or hours. Machine learning solutions are effective at handling these types of large-scale problems with little to no human intervention.

When you can do it manually, but it’s not cost-efficient. There are situations in which in-house experts could process many requests quickly and accurately, but at a high cost. For instance, imagine you assess DMV forms for in-state and out-of-state car purchases to determine their validity before passing them on. In this situation, the business processes are well-defined, optimized and serialized. It may take only a few minutes to check each form thoroughly. But allocating so much manual labor to this work is likely not the best use of your budget. Machine learning, on the other hand, offers predictable, pay-as-you-go pricing for fully scaled operations.

When you have a massive dataset without obvious patterns. Consider this – you’ve successfully prepared a well-curated dataset and know the underlying problem you’re trying to solve. However, you don’t see any explicit patterns in the data, preventing you from encoding those validations. Plus, there are many typos, missing fields, and other human-caused errors with no validation in place. You may even know the data is poor quality and can manually determine every affected row. But you can’t see any actual connections between valid and invalid records. Machine learning algorithms can solve this problem. They can find hidden connections between data points that aren’t clear to humans. Tools like interpreting tracers can even describe how machine learning models arrived at their conclusion(s).

When you live in an ever-changing universe (adaptive). The world, and its problems, are always changing. A problem you solved yesterday can, today, easily mutate into something else entirely, rendering your previous solution inefficient or even useless. For example, if your organization processed medical appointment records to extract diagnoses, procedure information and billing codes, your rules might have to evolve constantly. However, you can’t make updates in real-time, 24/7. Meanwhile, incorrectly labeled items could lead to insurance rejections, huge fines and legal penalties. One major advantage of machine learning methods is that they can learn from data across the entire life cycle of your application – from the first line of code written to the moment when the model is finally shut down. Moreover, it’s important for production-grade systems to have feedback loops so that you can catch the moment when your model no longer solves problems correctly.

It’s important to remember that machine learning is a tool – it’s not magic. Machine learning models are, essentially, advanced math-based algorithms, which identify patterns in data and learn from them. However, when properly applied to the right use cases, machine learning can reduce the amount of time spent on error-prone manual IT operations, adding significant business value and greatly reducing IT costs.

March 15, 2021