devopsy.kinsta.cloud

Tag: AIOps

  • NewOps? AIOps? NoOps? Just Don’t Call Me Late for Dinner

    NewOps? AIOps? NoOps? Just Don’t Call Me Late for Dinner

    Once again, we are seeing an uptick in the use of the term NoOps. I always felt that NoOps was a misleading and, in fact, empty phrase. I don’t care what magic you think you have; you are never going to eliminate the need to operate your software, applications and infrastructure.

    I first heard the term NoOps early in the maturation curve of DevOps. Back then, the idea seemed to be that Dev was going to absorb Ops. Between that and the cloud, there really wasn’t a role for traditional Ops to deploy. Managing apps would be easy with the cloud and automation. At least, that was the pie-in-the-sky thinking—but I never bought that.  

    There were other terms that popped up, like NewOps and AIOps, though that became something else entirely. But really, I think the rise of the systems reliability engineer (SRE) role put the NoOps movement in its grave (where it probably belonged). SRE became the shiny new identity of the Ops crowd and SRE continues to be one of the most popular topics here on devopsy.kinsta.cloud.

    This latest incarnation of NoOps seems to be a little different. From what I have seen, it is not really NoOps; instead, it is more of a “not your Ops to worry about” movement. It is shifting who is responsible for Ops from the company developing and using the apps to a service provider (or SaaS provider) who is responsible for Ops. It is more a “MyOps” not “YourOps” movement, but there is still most certainly an Ops there.

    In my neighborhood growing up, we used to have a saying, “I don’t care what you call me, just don’t call me late for dinner.”  Originally, to me, NoOps was very much like that. We changed the name of something, but it didn’t go away or fundamentally change the thing itself.

    Now, we still have Ops, it is just a question of who is doing it. That is not NoOps to me. It is “somebody is doing MyOps for me.” 

    We saw a similar pattern in security. The idea of outsourcing security was never called “NoSec,” because the idea of no security is, frankly, downright scary. Instead, it just meant relying on a cloud vendor or an MSSP to handle your security for you; it was outsourcing your security. An important lesson learned from that was that you can outsource your security ops, but you can’t escape your security responsibility. That is to say, when your customers’ information is breached, saying “We had someone doing this for us!” did not absolve you of responsibility.

    The same thing holds true for this new “OutsourceOps.” The outsourcer may do a better job than you at Ops. They may give you SLAs and SLOs that give you reasonable expectations of performance and remedies if those SLAs and SLOs are not met. But it is still your company’s name and reputation on the line.

    So is what we seeing as NoOps really NoOps? No, it is Outsourced Ops. But as we said in the old neighborhood, I don’t care what you call it, just don’t call it late for dinner.

  • Positioning ML Devs and Teams for Success

    Positioning ML Devs and Teams for Success

    Intelligent applications are (by their very nature) complex. While conventional software basically consists of one thing (code), intelligent software involves code, models and data. As previously discussed, three distinct fields—DevOps, MLOps and DataOps—have evolved to govern each of these interconnected disciplines. 

    Moving through the ML life cycle quickly and efficiently requires collaboration between teams in each of the above disciplines. Organizations should strive to make this collaboration as frictionless as possible to position their ML efforts for success. What follows are four places where we believe organizations should focus: 

    • Infrastructure
    • Team structure
    • Tools 
    • Project ownership and KPIs

    Build Dedicated Infrastructure for Each Pillar

    Infrastructure software is not one-size-fits-all across the three pillars of code, model and data. For example, DevOps teams have historically played the infrastructure role in software development, supporting application or product services. Think of regular software engineering teams who are supported by DevOps, producing different kinds of application services. App infrastructure is so baked-in to software development that most of us take it for granted.

    Just as regular software services require DevOps, data and model services require corresponding infrastructure. Data infrastructure is typically concerned with managing tools (such as Airflow installations, Spark clusters and feature stores), building data pipelines and ensuring that data services remain smooth and operational for data consumers. 

    On the other hand, model services are chiefly concerned with creating, running and organizing models in such a way that others can use them efficiently. Since the AI/ML life cycle is incredibly complex, a host of factors need to be considered (e.g., hardware, parallelization libraries, model packaging formats, etc.) to help data scientists and engineers do their jobs efficiently. Robust ML infrastructure along with a dedicated ML infrastructure team goes a long way in helping organizations scale their AI.

    Optimize Team Structure

    Best practices for ML and AI are continuously evolving, and organizations have tried a host of team structures over the years. We have noticed two popular structures for intelligent application development teams emerged in recent years:

    • Pods

    Pod structures involve smaller teams—usually a data scientist, a data engineer and an application software engineer—working together closely to deliver an intelligent feature or product. Due to their tight-knit nature, these teams are positioned to iterate extremely quickly, which often leads to better products and features. However, the closely-knit nature of pods also comes with a downside, as knowledge sharing between pods with different objectives can be difficult.

     

    • Center of Excellence

    In a center of excellence (or COE) structure, all data scientists or data engineers are “pooled” and then assigned to different products or features depending on project requirements and circumstances. In contrast to pods, COE structures tend to share knowledge very easily. However, this arrangement can lead to slower iteration cycles because team members are not dedicated to particular products or features. 

    In our opinion, both pod and COE arrangements can be effective depending on the realities of an organization. Choosing an optimal team structure depends on how the rest of the organization is structured and how reporting relationships are set up. 

    Select Appropriate Tools

    To select the best possible tools for your organization, choose those that foster collaboration and uphold safety across code, model and data teams. 

    The right tools help teams work quickly and efficiently. Teams should be able to discover pertinent information for themselves without having to pull others off of whatever they are working on. For example, a data science team should be able to discover what data access or learn what features exist. Similarly, application teams should be able to discover what models exist, what versions are the latest or how and when to use models. The more that you can make your code, model and data self-serve, the faster teams will be able to build products that are ready to be put into production. 

    Furthermore, proper tools make debugging faster and easier. If you’ve been around ML-enabled features or products for any length of time, you’ve probably been bitten by undeclared dependencies and apps breaking due to upstream changes. In an intelligent app, finding the root of a problem can be extremely difficult because the root cause can exist in any of the three pillars or any combination of the three. Robust tools provide logs and information to help discover what went wrong more easily.

    Finally, governance should play a key role in tool selection, as safety should be a chief concern for ML teams. For example, consider a model that predicts the demographics of a user based on their app usage. While these predictions might be considered acceptable in certain applications, if a model is being used to make financial recommendations or approve credit applications, use of certain demographics may violate various consumer protection laws. Since a software engineer using this model might not be aware of the nuances related to its use, baking in safety checks on model and data use becomes vitally important. 

    Establish Ownership and Determine Appropriate KPIs

    Whether we are working at a company or in an academic lab, our goal always remains the same: build the best product. 

    But “best product” can mean a lot of different things depending on the use case. For example, sometimes teams may be willing to trade a bit of accuracy if a model fails in predictable ways (e.g., in health care applications.) By contrast, an e-commerce recommendation system can optimize for overall model accuracy even if a model isn’t quite as stable. As a result, choosing the appropriate KPIs for model optimization requires careful attention.  

    Similarly, choosing appropriate metrics that make sense for the product and business usually requires a high-level view of a number of factors — user expectations, revenues, user complaints, etc. In our experience, product owners are usually in the best position to make these decisions. KPIs for each of the pillars supporting the intelligent app should be based on key metrics determined by product owners. While these can range, we’ve seen the following KPIs work well:

    • Code — Number of releases, number of bug fixes, number of features that have shipped, mean time to resolution
    • Model — number of model iterations shipped, model quality in production, model performance SLAs 
    • Data — frequency of data refreshes (e.g., every day or every month?), number of data quality issues, speed of access to data

    The key point here is to drive all of these based on the product KPIs and to focus on the agile principle, which is how quickly are you moving and how many deploys are happening. 

    Know what metrics your team is looking to optimize so that you can invest your resources in the right spot. 

  • What Does AIOps Mean for SREs?

    What Does AIOps Mean for SREs?

    It seems SREs are of two minds when it comes to AIOps. On one hand, AIOps’ potential is pretty exciting. By automating complex workflows and troubleshooting processes, AIOps could make your life as an SRE much easier.

    But on the other hand, some SREs may choose to view AIOps with disdain and distrust. They might think of AIOps as just another trendy buzzword that doesn’t live up to the hype and that can become a distraction from the SRE tools that really matter.

    Which perspective is right? Should SREs embrace AIOps with open arms, or should they resist marketers’ efforts to position AIOps as the latest, greatest tooling innovation in the IT industry?

    Those are subjective questions that we can’t answer definitively, but let’s at least gain some perspective by examining what AIOps means for SREs.

    What is AIOps?

    If you keep up with IT buzzwords, you’ve probably at least heard the term AIOps—which is short for artificial intelligence for IT operations—and that it refers to the use of AI and machine learning (ML) to help automate IT operations workflows.

    The big idea behind AIOps is that, by using AI and ML to perform advanced analysis of large volumes of data from IT systems, IT and SRE teams can solve complex problems more efficiently than they could using a traditional, manual approach.

    AIOps can, for example, help to surface the root cause of a performance issue in a complex, multilayered environment like Kubernetes. Or it could make recommendations about how best to resolve an incident.

    AIOps entered the IT lexicon in 2016, when Gartner coined the term. Now, six years later, it’s a relatively well-established tool domain.

    How SREs View AIOps

    Despite the fact that AIOps has been around for some time, it doesn’t yet appear that many SREs have bought into the AIOps revolution. Catchpoint found in a 2021 survey that just 7.5% of SREs reported that AIOps tools delivered high value to their organizations.

    It’s unclear exactly why there are such low rates of excitement about AIOps, but we’d speculate that there are a few key factors at play:

    • AIOps is a new term for an old idea: Many monitoring and observability tools have included at least basic AI and ML analytics features for a long time, even before tool vendors slapped the AIOps label on their products. SREs probably realize this and, to some extent, view AIOps as an effort by marketers to rebrand functionality that is not actually new.
    • AIOps is hard to implement: Setting up an AIOps tool requires integrating it with diverse data sources and customizing it to fit your workflows and environment. It’s possible some SREs view this setup work as requiring more effort than it’s worth.
    • AIOps can’t replace human insight: While AIOps tools may be useful to a point, it would be unwise to place blind trust in AIOps-based analyses or recommendations. For this reason, some SREs may believe that AIOps encourages organizations to rely too heavily on automated tools at the cost of the expert analysis and perspective that only SREs can provide. (This is similar to the fact that it sometimes makes sense to prioritize human intuition and expertise over playbooks.)

    From an SRE’s perspective, then, AIOps may appear over-hyped, overly complicated and underperforming compared to traditional approaches to SRE.

    What SREs can Gain From AIOps

    SREs’ wariness toward AIOps is valid—but only to a point. It’s important not to let suspicions about the limitations of AIOps turn into excuses not to use AIOps at all. AIOps has some value to offer to SREs, even if it’s not perfect.

    For example, AIOps can play a role in reducing toil. To the extent that AIOps tools can recognize complex patterns or discover interrelated data sets more quickly than human engineers, AIOps reduces the time SREs have to spend manually troubleshooting problems or poring over complicated information.

    AIOps also helps to enable a more proactive approach to monitoring and incident management. If AIOps tools can alert SREs to emerging issues before they would otherwise recognize them, AIOps can help the SREs to get in front of the problems before they turn into true incidents. That’s better for SREs and end users alike.

    There is also an argument to be made that AIOps can help SREs do more with fewer engineering resources. If you can use AI to automate some aspects of monitoring and incident response, you can maintain the same levels of availability and performance with fewer human engineers on hand.

    Conclusion: AI Won’t Replace SREs—But it can Help

    None of the above is to say that AIOps can replace SREs, or that it magically solves every problem SREs face. Anyone who believes AIOps is a silver bullet has bought into the marketing hype to an unhealthy degree.

    Nonetheless, AIOps tools do offer value to SREs by making their jobs easier in some respects and improving reliability outcomes.

    So, while it’s wise to maintain a healthy perspective about the limitations of AIOps, SREs shouldn’t rule out AIOps tools as one way to improve reliability engineering.

  • Techstrong Research DevOps Radar Report 2022

    Techstrong Research DevOps Radar Report 2022

    You can’t control the pace of change; instead, consider that we are in a process of reinvention—both with regard to the way organizations use technology and how vendors position their offerings. At the same time, it’s never been more important for IT organizations to have complete visibility into the way technology is used from both a security and operations perspective. 

    This need to embrace reinvention while continuing to have broad and deep insight across all technology platforms led me to create a 2022 DevOps Radar Report. In this article, I highlight three near-term priorities that must be on every technology team’s radar. In addition, I’ll point out some long-range radar trends that should be on your agenda.

    Collision Warning! Your Current Radar View

    API Security

    APIs are the lifeblood of the cloud-native and app-based economies. Increasingly, poor API security across a variety of applications and systems puts your organization at risk. A compromised API or the use of unsupported or outdated APIs can prove to be an easy attack vector for cybercriminals. In addition, API vulnerabilities are calling into question how organizations approach application security and the usefulness of web application firewalls (WAF) and encryption protocols. Therefore, API security is rapidly becoming a top security priority for enterprise security teams and software professionals. 

    Failing to adequately manage the security of APIs creates a host of new challenges and questions that you must consider. For example, who is using which APIs? Are your APIs updated? Did your DevOps team evaluate the security of your APIs during their selection process? Have you formalized the way your organization is evaluating API security?

    It is increasingly clear that an API breach can lead to the downfall of your entire digital software strategy—but it doesn’t have to be this way.

    At Techstrong Research, we segment the market into “pure-play” API security vendors and larger security vendors with specific API security offerings. Our team will be publishing a Market Vista report on the API security landscape soon, so if you want to share your opinion or thoughts, please reach out.

    Below is a brief list of some of the vendors we are tracking in the API security market. Of course, expect consolidation—some of these pure-play vendors will likely be acquired in 2022.

    Pure-play API security vendors we’re tracking include:

    General security vendors with API security offerings that we’re tracking:

    Observability is Evolving to AIOps

    Modern applications are composed of containers and APIs that your organization can’t control, and into which visibility is difficult. Monitoring and application performance management (APM) solutions aren’t new; for many years log vendors built their businesses on collecting log files and sending alerts … so what’s changed?

    First, logs without context are meaningless. You need to understand upstream and downstream dependencies. Additionally, speaking for myself, I’m a mere mortal—if your teams are also made up of mere mortals, there is no way they can ingest, correlate and understand the tsunami of incoming DevOps log data quickly enough to take proactive measures.

    Besides the need to understand context, we are dealing with multiple cloud and on-premises platforms that each have their own logging protocols. Ultimately, the goal of observability and AIOps is to understand what’s happening across and among all these environments and technologies so you can detect and resolve issues as quickly as possible to keep your systems efficient and reliable and your customers happy. 

    Humans are the Biggest Security Vulnerability

    As an industry analyst, I see our clients pouring resources into cybersecurity efforts to prevent zero-day attacks and sophisticated advanced persistent threats (APTs). These threats, while real enough, are mostly external and may not pose as great a threat as many organizations believe. In fact, the vast majority of security vulnerabilities are caused by:

    • Well-meaning employees making innocent mistakes, including:
      • Falling victim to phishing attacks
      • Poor corporate password policies
      • Cloud misconfigurations and open ports
    • Technical debt—relying on older software and hardware that was never designed for a 100% mobile workforce, for example

    The first step in combating this problem is improving security awareness and building a cybersecurity culture. That starts with everyone in your organization understanding that security is a team sport and that it is a 24/7/365 responsibility. Even if you work from home; even when you’re cooking dinner while checking email on your phone, you need to have your “corporate security/Is it safe?” hat on.

    At the same time, machine learning, AI and automation have a role to play. There are emerging offerings that help enforce DevSecOps best practices and help security teams reduce the constant noise so they can focus on the most vital threats.

    Longview Radar: Plan Your Strategy Now

    To support emerging business models, what do DevOps leadership teams need to be thinking about from a long-term perspective?

    • Web3/Web 3.0 and the potential for change to the internet business model. But there is still a ton of opportunity with Web 2.0.
    • How will you support low-code/no-code? As a business model, vendors are selling directly to business units—it’s likely that your IT team has an existing solution to solve the business pain point(s).
    • Packaged software offerings that include blockchain, NFTs and other digital ledger technologies.
    • How to upskill teams, both from a technical perspective and a business perspective.
    • Providing developer teams with the tools they need while establishing guardrails to enforce governance and compliance requirements.
    • How to manage cloud costs in a predictable way no matter how many cloud vendors your organization uses.

    Conclusion: You Can’t Put the Brakes on Change

    Attempting to control change and the speed of reinvention is futile. You need to get comfortable riding the waves of industry change, or you’ll be swept away! Every IT leader needs to learn how to roll with and manage the ebb and flow of business, IT and personnel change.

    The best place to start is by building a successful culture within your organization. Although we often talk about technology-focused solutions to business problems, the first step on this journey doesn’t involve tech at all; it involves helping IT and business teams understand each other’s pain points and learn how they can each support a continual reinvention process.

    I will be tracking these incoming radar targets and other associated trends throughout the year and I look forward to your input. Do you think I missed something? Have a question or want to share what your organization is doing to support the need for continual reinvention? 

    Feel free to engage with me – @DDKirsch, LinkedIn or, of course, email. 

  • AI: It’s What Staff Crave!

    AI: It’s What Staff Crave!

    If you’ve seen Idiocracy (Side note: I am not recommending the movie; I’m using a phrase from it we’ve all heard. The movie itself was a comedy not at all meant to be family-friendly when it was made, and changes in social mores since its 2006 release make it far less acceptable today), you are familiar with the title phrase I’ve used here. That’s what is repeated almost religiously to justify why they buy/use/drink Brawndo, an energy drink.

    One of the scenes involves people telling the star that plants need Brawndo—“It’s got electrolytes … That’s what plants crave!” This is repeated over and over until he asks, “What are electrolytes?” After some stuttering and failures to answer, someone says, “It’s what they make Brawndo out of.” Someone else inevitably tacks on, “It’s what plants crave.”

    Now, substitute “AI” for “Electrolytes”. That’s where we seem to be. Ask a vendor what algorithms they’re using, where their training data comes from, the difference in results between the test reservations and the real world. Most will end up saying a lot, not answering your questions and ending with, “It’s what staff crave!” or some such.

    A few companies will lay it all out there, assuming you can parse through the information and are familiar with the AI training algorithms they are using or the algorithms they’re using to classify inputs against the dataset. And these companies are grand because they (a) know what they’re actually doing beyond “It’s got AI/ML, it’s what staff crave,” and (b) Expect that you are knowledgeable about the data you’ve asked for or are capable of looking it up in this age of wonder that has boundless information at your fingertips.

    Don’t let your vendors get away with just repeating “It’s got electrolytes” over and over. AI and ML are a massive help in some areas, but they’re not yet a panacea and success rates vary wildly whether we’re talking problem domains, datasets, algorithms, training style, etc. Make them tell you the details. Where the data comes from and how big that dataset is. How much was reserved for testing and whether it was reserved randomly. Ask them what they hope AI will achieve, then ask them how that correlates to human interaction. Most AI today is still evaluating or replicating human interactions with something—most AI in IT particularly so. What algorithms were used to validate? And finally, make sure to ask for real-world results. I know people in data science and one has confidently said to me, “My models are 95% accurate.” When I finally dug into that claim (because my experience is that real-world use of AI/ML is almost never that accurate) it turned out that claim was true … for the reserved test data. So, the same dataset used to train got the same results on the reserved subset. That’s a good thing, but it is no indication of how said algorithm will do in the wild with unfiltered data. The big problem really is the switch from test data to real-world data.

    In fact, in IT, we are a little better off than most spaces wrestling with AI/ML, because very often there is no difference between test and training data in IT. A network security vendor will test against actual traffic—because they have access to massive volumes of it and it shortens the iterations to use what’s at hand and what the system is being trained for.

    And keep rocking it. The current iteration of IT is the opposite of the dark comedy future depicted in Idiocracy. You all are putting up systems and keeping them running at increasingly fast cadences; sometimes remotely because of COVID-19 or other work-related issues. Don’t stop turning out success for your coworkers and customers and take a moment to force vendors to explain what they’re doing with AI. If you’re a generalist like me, ask over email, though, so you can go look up the answers and make sure you understand the details.

  • Extending GitOps to the Enterprise

    Extending GitOps to the Enterprise

    GitOps is a relatively new term, but it continues to gain momentum as more organizations embrace the paradigm. In DevOps, GitOps lands on the software engineering side of the development and operations continuum. Even though GitOps is more likely to be adopted by smaller organizations, larger enterprises can successfully implement GitOps with just a few tweaks to the overarching framework.

    Let’s start at the beginning. Wouldn’t it be great if we could combine DevOps and GitOps approaches and use the Git distributed version control system as the ultimate source of truth? Then, when there’s a dispute over the correct state, people know where to go for the correct version.


     

    In GitOps, the system’s desired configuration is stored in a revision control system such as Git. Any difference between the desired state stored in Git and the system’s actual state indicates to DevOps teams that not all changes have been deployed. These changes can be reviewed and approved through standard revision control processes such as pull requests (PRs). When a PR is approved and merged to the main branch, an operator software process is responsible for changing the system’s current state to the desired state based on the configuration in Git.

    GitOps doesn’t require a particular set of tools, but the tools must:

    1. Operate according to the desired system state stored in Git.
    2. Detect differences between the desired and actual states.
    3. Perform required operations on the infrastructure to synchronize the actual and desired states. 

    In an ideal implementation of GitOps, manual changes to the system are not permitted and all changes to configuration must be made to files stored in Git. The infrastructure and operations engineers’ roles in a GitOps model shift from performing infrastructure changes and application deployments to developing and maintaining GitOps automation and helping teams review and approve changes through Git. 

    GitOps is great for non-critical environments, but enterprises are less likely to adopt GitOps due to a number of challenges facing GitOps users.

    Challenges with GitOps

    GitOps is applicable only to a subset of the software development life cycle (SDLC). This is important because GitOps tools are sometimes marketed as a one-size-fits-all solution that will solve all release problems—this is simply not true. First, GitOps requires that your deployment artifacts are already there. This means that tasks such as

    • Compiling code
    • Running unit/integration tests
    • Security scanning
    • Static analysis

    are not a concern of GitOps tools and are assumed to already be in place. GitOps also doesn’t address promotion of releases between environments.

    Auditability

    A default GitOps process is not fully auditable because of force push. Force push essentially allows users to remove any unwanted blocks of commits from Git history on the central repository.

    Business Approvals

    PRs are a typical approval model used in GitOps. Developers make changes, create PRs and then an approver may accept that PR. This would be recorded and a change would be deployed. A PR-based approval model assumes peers are already using Git and are familiar with PRs. 

    Generally this works great until we need to include business approvals. Business personnel may not be familiar with Git or PRs. If a team needs business approvals, they have to build a process on top of GitOps. This drastically increases lead time. 

    Governance 

    GitOps is dependent on a pure PR-approval based model, one in which the PR approver is responsible for reviews. After a PR is approved, there is no way to enforce enterprise policy rules. Enterprises need to control what end users can do on specific clusters to ensure those clusters are in compliance with policies to meet governance, legal requirements or enforce best practices.

    An Enterprise Approach to GitOps

    However, there are ways to solve this problem at the enterprise level.

    Alerting on Drift Detection 

    Enterprise customers get alerted whenever there is a drift in their repository and, based on that, customers can choose to approve or reject the drift. If rejected, the merged PR should be reverted to the previous state to ensure that Git is in sync with the desired state of the cluster.

    Use a Pipeline for Promotion of Releases Between Environments

    To promote an artifact between environments, users would have the ability to use a pipeline with deployment stages, performance monitoring and log analysis/approvals. 


     

    Let’s say the developer commits the source code and the GitOps deployment happens in the QA environment. The PR can be automatically approved upon meeting the policy standards set by the company. Then it can be pushed to a production environment.

    Governance 

    There are many reasons why developers and platform engineers integrate policy-as-code into their current GitOps processes. For one, this strategy helps to accelerate application development and deployment because it helps solve many of the change management hurdles that slow development pipelines.

    On the development side, policy-as-code helps app developers understand and abide by the company’s configuration, security and compliance policies. For example, a developer may not remember—or may have no reason to know—when deploying a load balancer onto Kubernetes in AWS is or is not sanctioned. Policy-as-code solves this problem automatically.

    The ability to run policies during GitOps deployments and adhere to those policies as a part of the deployment and upon any violations—alert, audit, notify and automatically roll back the changes from Git to the previous successful compliant state so that Git remains in sync with the cluster.

    GitOps + AIOps

    There is a notion that once GitOps is in place, everything will work seamlessly with no problems. Not so much; it is important to continually monitor deployment state to ensure everything stays as it should. That is where AIOps comes into picture.

    GitOps enables DevOps and development teams to see any manifest change, infrastructure change or artifact change. Post-deployment, the GitOps operator can instantly understand the changes in the service health, see what caused those changes and automate the process of investigating them using AIOps. If any severe anomalies are found, the operator can determine the current state of the system and, should anything differ, revert to the previous compliant Git commit. 

  • Ensuring Network Connectivity With SD-WAN and AIOps

    Ensuring Network Connectivity With SD-WAN and AIOps

    Recently, Facebook experienced an outage. If companies relied on that app to do business, their IT teams were likely swamped with end-user complaint calls. 

    How would teams traditionally respond? They’d dive deep into the rabbit hole, analyzing issues on a case-by-case basis, asking countless questions, including: 

    • Could there be a problem with the enterprise network?
    • Is the service provider experiencing issues?
    • Are just a few users affected or is the issue more systemic?

    Traditionally, this manual process requires significant knowledge sharing and time to complete as teams break the problem into smaller pieces to triangulate an issue’s root cause.

    Oftentimes, it’s an enterprise network that experiences connectivity issues—not apps like Facebook. 

    Some companies pay an arm and a leg to leverage highly reliable WAN links, such as MPLS, for connecting to the internet and connecting branches to their data centers. Conversely, they may save a buck or two by using commodity links, like cable or DSL, which deliver less reliability. Both choices introduce tough tradeoffs—could there be a better way to ensure enterprise-wide connectivity?

    Enter SD-WAN. 

    Combining the same service-level agreement (SLA) as a dependable WAN link with the affordability of a commodity link, SD-WAN specializes in handling brownout situations, rerouting link traffic to maintain network connectivity — ensuring users’ access to their must-have apps. 

    And by incorporating artificial intelligence for IT operations (AIOps), teams can harness the power of analytics to radically reduce their response time, ensuring that if problems do happen, they get fixed fast. In fact, 90% of IT professionals acknowledge that powering network management with AIOps will improve their companies’ business outcomes.

    Drowning in Data: Understanding Traditional Network Issue Investigation Headaches

    To determine the root cause of a service disruption, teams spend considerable time assessing the landscape of all the different apps that people use and the links they follow to use them. These apps could be located anywhere—forcing teams to find needles across vast haystacks of data to determine what went wrong.

    And this data is exploding at such an extraordinarily high rate that it’s practically impossible for anyone to keep track of it all. Only by leveraging an advanced analytics system like AIOps can teams gain the necessary insights to swiftly solve problems. 

    SD-WAN and AIOps: A Problem-Solving Dream Team

    How do SD-WAN and AIOps work together to enable a highly reliable, always-connected network?

    SD-WAN quickly addresses all potential local issues and optimizes performance and connectivity across multiple WAN links—or even just a single WAN link that may be unreliable. SD-WAN also ensures that business-critical apps are prioritized correctly and are offered the precise amount of error correction that avoids any issues or disruptions as much as possible over the WAN. 

    Where does AIOps come into play? It analyzes data at a far more granular level for extended periods over a global topology. What is it looking for? Everything. It’s examining emerging patterns that are impacting individual apps. It’s searching for systemic issues that affect multiple SD-WAN edge devices simultaneously, worldwide. It could also be investigating how edges or apps are being affected over a particular time period. 

    By surfacing all that data, AIOps analyzes much more than SD-WAN can. And by deriving insights from that data, it highlights issues affecting everything from a lone user on a single application to a systemic issue that’s impacting an app across a far broader cohort of users. In certain cases, AIOps can cue the SD-WAN to heal itself and automatically fix the problem, then verify that the issue was fixed. As a result, end-users remain connected to their network.

    Inside a Wholly Adopted SD-WAN/AIOps Environment

    Imagine you’re an IT admin, supporting an extremely complex enterprise network, spanning thousands of different sites that use diverse security services for connecting to apps. Even if you’re on a huge team that’s charged with monitoring the network, it would be nearly impossible to have full visibility across the network to solve problems as they occur.

    By adopting SD-WAN together with AIOps, you gain a superpower. No, you won’t be able to fly or become invisible, but AIOps will make you omniscient, enabling you to:

    • Harness powerful analytics firepower to find and fix network trouble spots instantly and precisely before users complain—in some cases before they even notice
    • Understand which sites perform better than others to pinpoint where improvements can be made
    • Propose configuration changes that help the network self-optimize and self-heal
    • Detect and optimize degraded links to boost user experience

    And if issues ever do arise, AIOps advises you immediately as it monitors the network 24/7, providing you with full situational awareness and insights, solving problems as they occur.

    SD-WAN & AIOps: Maintaining Connectivity by Self-Healing

    The Facebook event was unusual in that it globally impacted the world’s users, who couldn’t access the app through any Facebook data center. In that scenario, AIOps simply identified the app disruption and noted the root cause.

    But what if the situation had been different? For example, imagine if Facebook’s U.S. west coast datacenters were deemed unavailable but its east coast datacenters were operational. By combining SD-WAN with AIOps, a similarly affected company could have performed diagnosis and ensured app connectivity. 

    How could that have played out? 

    First, AIOps would have detected the issue and its root cause. It would have reported that the application was unreachable to users on the west coast but available to users on the east coast. Finally, SD-WAN would have self-healed by rerouting end-user application traffic to the eastern data centers to maintain app connectivity.

    Using the old-school method of manually monitoring and troubleshooting networks is nearly impossible for most IT teams. Instead, teams are relying on SD-WAN and AIOps. This combined capability gives teams an unbeatable superpower, enabling them to substantially improve connectivity across the enterprise and rapidly respond to issues as they occur.

  • BMC Looks to Advance ServiceOps Approach to Managing IT

    BMC Looks to Advance ServiceOps Approach to Managing IT

    During an online BMC Exchange event, BMC announced it expanded its portfolio of IT management offerings as part of an ongoing effort to unify traditional IT service management and DevOps processes under an integrated ServiceOps framework.

    The latest additions to the BMC portfolio includes an update to its Control-M workload automation platform that adds observability tools for monitoring application and data workflows along with additional integrations for cloud services such as AWS Glue, Google Cloud Dataflow and Function and Microsoft Azure Data Factory and platforms such the data lake from Databricks.

    BMC is also making available a preview of a Control M Python client that will extend the reach of the workload automation platform to the code data science teams employ to build artificial intelligence (AI) models.

    In addition, BMC is adding a BMC Helix Digital Workplace studio to make it easier for IT teams to customize the software-as-a-service (SaaS) platform it provides for IT service management (ITSM). Finally, BMC is also making available a version of BMC Helix that is optimized for communications service providers.

    BMC CEO Ayman Sayed said overall the current emphasis on digital business transformation is driving a need to converge the management of multiple IT platforms as organizations reinvent how core processes work across their organization. It’s not possible to drive that level of reengineering across multiple IT platforms without centralizing the management of those platforms using a ServiceOps framework that unifies DevOps and existing ITSM platforms, he noted.

    At the same time, Sayed said organizations need to invest more in artificial intelligence to automate IT operations, also known as AIOps, to augment IT staff that is hard-pressed by the need to manage both emerging and legacy IT platforms. The latest investment BMC has made in AIOps came earlier this month with the acquisition of StreamWeaver, a provider of a set of tools that make it easier to collect massive amounts of data that can be used to more easily train AI models.

    It’s too early to say to what degree IT management processes might become unified. In addition to centralized IT teams, most organizations have business units that often drive their own IT initiatives. However, as organizations realize how many platforms a digital process needs to access the need to centralize the management of IT becomes more apparent. The issue each organization will need to determine is the degree to which that centralization needs to occur.

    It’s clear in the wake of the COVID-19 pandemic that many IT organizations have never been more agile. In some cases, those IT teams are now able to drive a rate of change that is too fast for many businesses to even absorb. In fact, it’s arguable that entire businesses will need to adopt DevOps principles to manage that rate of change.

    In the meantime, IT teams will need to take a harder look at how much time is being spent on manual tasks that could be automated in a way that enables them to be lot more productive than they are today.

  • Learn a Bit About AI

    Learn a Bit About AI

    Every once in a while, a trend becomes broad enough that I feel the need to offer career advice. This year it will be simple: Know enough about AI/ML to be dangerous. We are past the point where not having at least a basic understanding is a good idea. Looking around the data center, AI is in use from the earliest stages of development through production network performance evaluation. Somewhere in your environment, you are using AI and ML. More to the point, somewhere in the places you may interview going forward, they are using AI and ML.

    I’m not suggesting you become an AI/ML expert—though you certainly could; I have a relative that is a data scientist and that’s all he does—design and tweak algorithms. He’s learning new things pretty much daily, so if you like that (as I do), it’s an option. But for those of us doing DevOps or SRE or even regular old operations, we’re only going to have to understand a bit.

    Understand what tools like Jupyter are, and what the most commonly used ML algorithms in your space are. Then, learn about how AI and ML help with what you are doing. Just as knowing more about the compiler/interpreter of your chosen language helps you solve more odd problems, AI is going to have that same issue. Think about how those folks who understand the guts of routing are the ones we want solving our routing problems. Again, same scenario.

    Moving forward, AIOps, NetOps, DevSecOps … All intend to increase the amount of work offloaded onto the AI in question. In a lot of cases that is great. We have enough to do; if you can paw through those logs and give me relevant summary info, go for it. But it doesn’t end there—some AI engines are making the calls and implementing changes. In both network and application security, a fair number of vendors will tell you there is no time to wait for humans to stop an attack, and they encourage their customers to turn on automated responses. That’s the first of many applications for AI/ML. We will see an increasing number of AI/MLOps tools being used for full-on automation. We all do difficult jobs with a complex—sometimes bewildering—number of inputs. But we use pattern recognition and experience to make determinations and do what’s best for our organization. Our software has reached the point where sometimes, it can do that for us on the easier tasks so we can focus on the more complex ones. When the AI/ML fails—and it will, because pattern recognition and historical trends are not always indicative of normal usage—knowing what is going on under the covers can help you determine if it did the right thing in the wrong way, or simply did the wrong thing. And it can help you to quickly recover. I can’t speak for every organization, but I’ve never seen one that lamented, “Sure wish that outage had lasted another couple of hours!”

    And keep rocking it. Somewhere in the distant future, an  AI may be able to do your job as well as you. But that isn’t today. Even if it can handle one bit, it certainly isn’t rocking the entire breadth of your responsibilities like you are. Thank you, from all the users that don’t realize you’re keeping them live.

  • BMC Acquires StreamWeaver to Advance AIOps Strategy

    BMC Acquires StreamWeaver to Advance AIOps Strategy

    BMC has acquired StreamWeaver, a provider of a platform that makes it simpler to capture streaming data in real-time, as part of an effort to extend the BMC Helix IT operations platform infused with artificial intelligence (AIOps). Financial terms of the transaction were not disclosed.

    Margaret Lee, general manager and senior vice president for digital service and operations management for BMC, said StreamWeaver brings with it a wide range of integrations with multiple data sources that BMC will use to both train AI models and improve overall observability. While connecting to data sources is relatively straightforward, the challenge now is collecting that data in real-time in a way that is easily repeatable for IT operations teams, noted Lee.

    In general, AIOps continues to gain traction as IT environments become increasingly complex. Most IT operations teams are not going to be able to manage highly distributed computing environments without augmenting their IT staff with some AI, said Lee.

    Naturally, many rote tasks that were once performed manually by IT operations professionals will become increasingly automated. However, there will always be a need for IT personnel, noted Lee.

    Arguably, there may come a day when IT professionals will not want to work for organizations that don’t have an AIOps platform in place simply because the tasks assigned to them would otherwise be considered archaic, especially if they involve a series of tedious tasks that not many IT professionals really want to perform.

    At the same time, AIOps should reduce overall risk by making it easier for IT operations teams to proactively identify issues before they disrupt an application service. That’s especially critical as IT environments—increasingly with microservices-based applications constructed using containers—and serverless computing frameworks are being deployed with greater frequency alongside legacy monolithic applications.

    BMC has been making a case for AIOps as far back as 2017. However, since then, a wave of startups and IT operations incumbents have either built an AIOps platform from the ground up or updated an existing platform with AI capabilities. It’s difficult to assess which vendors are now ahead in terms of the AIOps race, but one thing that is certain is that IT organizations now have plenty of options.

    From a cultural perspective, however, not every organization is entirely comfortable with AIOps. While some IT professionals remain skeptical, others view AI as an existential threat to their existence. Most IT teams are also concerned about the potential impact any flawed AI model might have on IT operations—it’s one thing for a human to make a mistake, but an AI model making a mistake at scale could be catastrophic.

    There are few IT organizations that are embracing AIOps across the board, but most do recognize that AIOps platforms, in one form or another, are inevitably here to stay. After all, machine learning algorithms never forget anything they learn. Nor do they ever take a day off or decide to leave when offered higher pay. The challenge is figuring out exactly where the interface between IT staff and algorithms is going to be—not just today, but also in the near future.