devopsy.kinsta.cloud

Category: Blogs

Using Chaos Engineering to Build Resilient Systems

Over the past decade, chaos engineering has become one of the most popular approaches in DevOps. It’s uniquely adapted to complex cloud-based systems and has the potential to succeed where more conventional approaches may not.

Chaos Engineering Explained

Traditionally, DevOps teams worked with systems hosted on single servers. They would analyze logs with a fine-toothed comb, looking for potential anomalies. They implemented automated tests to ensure the system worked to spec. They employed root cause analysis to debug applications.

Modern systems use cloud-based distributed architectures and are much too complex for this approach to be successful; they contain hundreds, and sometimes thousands of instances. In many cases, their behavior can be chaotic, making it nearly impossible to predict.

Because of this, modern systems need to be designed for resiliency from the outset. Chaos engineering was developed to foster resilient system design by deliberately anticipating failure modes.

Two Steps to Chaos

Here’s how it works. After characterizing a state of “health” for their system, DevOps teams perform a chaos experiment. Just like physics or chemistry experiments, chaos experiments involve setting up two versions of a system: the experimental version and the control.

The experimental system is then subjected to a barrage of virtual kicks and punches which could range from injecting faults to disrupting a data center. DevOps teams then pore through the resulting logging data, attempting to understand how the chaos experiment affected the state of the system.

Chaos engineering has two major differences from conventional DevOps practice. First, while conventional forms of testing supply a controlled input and test for a predefined output, chaos experiments are far more open-ended. They’re designed from the outset to expect the unexpected, requiring analysis of the whole system to give novel insights.

Second, they are performed on live systems. This makes chaos engineering controversial, as its experiments have the potential to affect real users. Still, it makes it much more powerful than alternative approaches. Chaos engineering encourages resilient design by taking genuine risks with systems in the wild.

Why Companies Embrace Chaos Engineering

Chaos engineering has been around for over a decade. In that time, it’s been adopted by several of the biggest names in tech. Netflix, the company that pioneered the approach, is still the most well-known for their use of fault injection.

In 2008, Netflix was moving from an in-house data center to the cloud. They knew that, as they expanded, their application would undergo massive horizontal scaling. Because any one of its thousands of instances could fail at any moment, it needed to be resilient.

They created the first chaos experiment, Chaos Monkey. This worked by randomly turning off instances in Netflix’s live application to check it was fault-tolerant. This practice sounds dangerous, but it encouraged Netflix engineers to design resilient system architectures.

Chaos Monkey sounds bananas, but it worked. In its first five years of operation, there was only one failure which was promptly fixed by an on-call engineer. That failure was caused by Chaos Monkey itself.

Facebook is another big company to adopt chaos engineering. They created Project Storm in the wake of 2012’s Hurricane Sandy. Although the hurricane itself didn’t affect Facebook, it served as a wake-up call. A storm hitting one of Facebook’s data centers could wreak havoc.

In 2014, Facebook took the step of turning off traffic to one of their data centers. Although no end users were affected, traffic loading went haywire on all sorts of different subsystems.

Once Facebook had successfully identified this weakness in their system’s resilience, they could preemptively design traffic management software that would be resilient in the face of a live data center shutdown.

As more and more companies use cloud computing and hyperscale data centers, chaos engineering is likely to become an essential feature of DevOps strategy.

Applying Chaos Engineering In Your Company

Chaos engineering is the ideal approach for complex, distributed systems. Chances are, your DevOps team either runs their application in the cloud or is planning to migrate to it.

Serverless computing, in the guise of systems like Kubernetes and Heroku, is popular with an increasing number of organizations. Now it’s common for systems to contain hundreds or even thousands of instances, hosted in massive data centers.

So far, we’ve only seen chaos engineering from 10,000 feet. It’s time to look at how you can make it work on the ground.

First, we need to understand the roles observability and monitoring play in chaos engineering. To successfully implement chaos experiments, there needs to be a high level of system visibility.

Dealing with Data

Marius Moscovici outlines four strategies for extracting valuable insights from data. He discusses the importance of detecting anomalies in your data stream, something a chaos engineer needs to be good at.

Whether it’s the haywire traffic in Facebook’s Project Storm or the deactivation of Netflix instances, chaos experiments rely on the ability to diagnose unusual patterns in data.

1. Profiling Normal System Behavior

In order to detect anomalies, you need to have a good idea of what your system normally does. In their book on chaos engineering, Netflix used the analogy of human vital signs. Just as doctors know that 98.6°F is a normal human body temperature, their IT counterparts need to characterize a “normal” state of “health” for their DevOps system.

Many log analysis and observability tools in the market offer machine learning capabilities to help achieve this.

2. Detecting Anomalies Across The Board

Anomaly detection should be integrated into every step of your data analysis. You need to be able to detect anomalies across the board, not just in places you happen to be looking at. Because chaos engineering, unlike more conventional forms of testing, is designed to reveal new and unexpected information it’s vitally important to have a completely open data strategy.

3. Data Scientists On the Lookout

Always be on the lookout for potentially anomalous behavior. Chaos engineering will require your company to take its DevOps strategy to a new level, both in terms of your team’s mindset and the sophistication of their logging tools.

Because chaos engineering requires you to analyze all available data without making assumptions, it’s necessary to leverage machine learning to extract patterns from your logs. Chaos engineering requires a scientific mindset and an appreciation of the power of data.

4. Create Alerts

Finally, make sure that anomalous behavior is being flagged and that the team is aware when it occurs. If possible, dynamic alerts should be used to avoid false positives and alert fatigue which is common with static thresholds.

Engineering with Chaos

More and more organizations are migrating to the cloud and building highly distributed application infrastructures. DevOps teams are being forced to confront chaos and complexity on a daily basis.

While the complexity of modern IT systems has necessitated chaos engineering, it has also enabled it. In a chaotic world, chaos engineering brings the order of an engineering approach.

June 18, 2021
Grafana Labs Acquires k6 to Add Open Source Load Testing Tool

During its online GrafanaCONline 2021 conference today, Grafana Labs revealed it has acquired k6, a provider of an open source load testing tool.

Raj Dutt, Grafana Labs CEO, said load testing will complement the observability platform Grafana Labs currently provides based on open source Prometheus software. Most of the output of any load testing processes are typically evaluated using a platform that analyzes metrics, he noted.

The k6 platform has been gaining traction among developers that require access to a tool that makes it simpler to dynamically scale load testing up and down as required versus having to guess how much infrastructure might be required to test an application workload, said Dutt. That capability not only makes it simpler to integrate testing within a DevOps workflow, Dutt added, but also serves to reduce testing costs.

In general, Dutt noted testing is shifting further left as developers assume more responsibility for quality assurance. The challenge is finding a way to enable developers to automatically incorporate testing within a continuous integration (CI) process that allows them to easily observe how synthetic tests are executing, added Dutt.

There’s no doubt developers are running more tests before promoting workloads into a production environment as part of an effort to reduce the number of issues that might be encountered when an application is deployed in a production environment. In many cases, the support calls that arise now go directly to developers who now have a vested interest in reducing the volume of those calls.

The issue that developers are contending with is as they embrace microservices, the dependencies that exist across multiple applications makes it more challenging to successfully deploy those applications. Grafana Labs has been making a case for a cloud platform that makes it simpler to centrally view data collected from instances of Prometheus distributed across multiple application development environments. Prometheus, now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF) that oversees the development of Kubernetes, is now being deployed more widely by IT teams to observe legacy monolithic applications alongside emerging microservices-based applications.

It’s not clear the degree to which organizations are also employing dedicated application testers. In many cases, those testers will surface issues that developers overlooked (or simply chose to ignore) in the face of tight application delivery deadlines.

In the longer term, however, it’s also apparent artificial intelligence (AI) will be playing a larger role in application testing. Someone will still need to review those tests, but many more rote tasks will increasingly be automated.

One way or another, the total cost of testing is declining, thanks in part to increased reliance on open source testing tools that, in many cases, developers are simply downloading themselves. Those tools are reducing much of the licensing friction and cost restrictions that developers might otherwise encounter.

In fact, the easier it becomes to test applications, the better overall application quality will become in the months and years ahead. At at time when organizations have never been more dependent on software, that potential improvement to the overall state of application quality can’t arrive soon enough.

June 17, 2021
PagerDuty Allies with JFrog to Modernize IT Incident Management

PagerDuty has integrated its IT incident management platform with the continuous integration/continuous delivery (CI/CD) platform from JFrog as part of an effort to streamline troubleshooting of application environments.

Steve Gross, senior director of strategic business development at PagerDuty, said the overall goal is to make it easier for IT operations teams and developers to collaboratively discover the root cause of issues by, for example, being able to identify builds within a JFrog pipeline that might be having a negative impact on a production environment.

At the same time, IT teams employing PagerDuty can now also receive notification from JFrog Xray, a tool that discovers both vulnerabilities in open source software and license compliance issues.

Organizations of all sizes are attempting to minimize the impact of an IT incident by providing deeper insights into how applications were constructed. The goal is to reduce the amount of time IT teams currently spend looking for the root cause of a problem.

Ideally, IT teams would discover those issue before they impact end users. The challenge they face is it can take weeks to discover the root cause of an issue that takes a developer a few hours, at most, to fix. Armed with insights from the CI/CD platform, Gross said it becomes easier for IT operations teams to proactively pinpoint the source of an issue in a way that provides the developer with enough insight to act.

That is critical because, more often than not, it’s the last build of application that was deployed in a production environment that is usually the source of an issue, especially when that IT environment was operating normally before that build was promoted into a production environment.

The data collected by PagerDuty from various DevOps platforms is fed into the PagerDuty tools via a Change Events application programming interface (API). Over time PagerDuty will also be able to apply machine learning algorithms and other forms of artificial intelligence (AI) to identify issues long before they reach a meaningful level of disruption, noted Gross.

In general, there’s a movement to modernize IT incident management in a way that aligns with DevOps best practices as part of an effort to identify issues before an application is actually deployed. The hope is that those efforts will reduce the number of “war room” meetings that need to be convened to determine the root cause of a performance or security issue after an application has been deployed.

It may be a while before IT incident management processes are modernized to that level. However, as IT environments become more complex—in part because of the number of microservices-based applications with multiple dependencies that are now being deployed—it’s apparent existing approaches to IT incident management don’t allow IT teams to respond fast enough to an issue. In fact, that’s more challenging than ever now that many of the members of an IT team are working from home more often.

The pressure to modernize IT incident management increases in almost direct proportion to IT teams’ stress levels. The challenge now is to modernize those processes before the members of an IT team are too burned out to appreciate the effort.

June 17, 2021
MediaOps Wins Two Gold Hermes Creative Awards for Video and Podcast
We are thrilled to announce that MediaOps is the recipient of two Hermes Creative Awards. TechStrong TV won the Gold Award in the video/informational category and DevOps Chats won the Gold Award for audio/podcast series.

The Hermes Creative Awards is an international competition for creative professionals involved in the concept, writing and design of traditional and emerging media. Hermes Creative Awards recognize outstanding work in the industry while promoting the philanthropic nature of marketing and communication professionals. Hermes Creative Awards are administered by the Association of Marketing and Communication Professionals (AMCP). Judges are industry professionals who look for companies and individuals whose talent exceeds a high standard of excellence and whose work serves as a benchmark for the industry.

Our main goal here at MediaOps is to create and produce high-quality, educational and informational content to our audience, covering IT industries and practices that are re-shaping the world of technology, including DevOps, cloud-native, cybersecurity and digital transformation. These awards are a testimony to all the hard work and dedication that goes into producing these programs. We are proud of our MediaOps team and honored to receive recognition for our work.

The two MediaOps broadcast programs recognized by the AMCP are:
- TechStrong TV: Gold Award winner in the video/informational category. TechStrong TV is a streamed broadcast featuring fresh video news and information three days a week and on-demand content from conferences and shows two days a week on the Digital Anarchist network. TechStrong TV features interviews with industry thought leaders, covering the latest news on DevOps, cybersecurity, cloud native and digital transformation.
- DevOps Chats: Gold Award winner in the audio/podcast series category. DevOps Chats is a “fireside chat” podcast featuring DevOps.com and Security Boulevard Editor-in-Chief, Alan Shimel, discussing relevant topics with luminaries and thought leaders in the DevOps and security industries.
The MediaOps team would like to thank the Hermes Creative Awards for recognizing the content we publish. Most of all, we would like to thank you, our audience, for all the support you have given us throughout the years.

TechStrong TV is available on the web at Techstrong.tv, with on-demand segments available on DigitalAnarchist.com and the MediaOps community sites (DevOps.com, Security Boulevard, Container Journal). TechStrong TV is also available on LinkedIn Live (DevOps.com page), Facebook Live (DevOps.com page), YouTube (DevOps TV channel), on the DevOps.com mobile app for iOS and Android, and on TV via DevOps.com app for OTT channels on Apple TV, Roku and Amazon Fire TV.

Episodes are available after broadcast on-demand at Digital Anarchist, the next-gen streaming service for tech communities, which offers in-depth coverage on DevOps, cybersecurity, cloud-native and digital transformation, adding high-quality content on a regular basis. DevOps Chats are published on SoundCloud and iTunes, and are also available on Digital Anarchist.
June 17, 2021
How Real-Time Debugging Improves Reliability

When designing and building software, service reliability is always at the top of the list of critical focus areas for development teams. Every team that builds software typically has, either directly or indirectly, service level agreements with their customers. These are, essentially, agreed-upon metrics or performance criteria that teams use to measure and ensure the reliability of a software system.

Organizations may measure reliability in different ways, looking at metrics like service availability, mean time to failure or mean time to repair. Regardless of how an organization chooses to measure reliability, you can often assess the health of an organization based on how reliable their applications are. Since we know that reliability issues happen regardless of how well we prepare for them, having tools to solve those issues quickly and efficiently becomes critical to maintaining smooth operations.

The Health of Your Organization

Most software development organizations today have key performance indicators (KPIs) which they measure to understand how well they’re doing and to be able to quickly and easily measure the health of their application development processes. If teams are finding that they are not meeting KPI’s, it’s often a sign that something fundamental needs to change or that new and improved processes need to be put in place. Teams that are constantly measuring what they’re doing well and where they are falling short with a data-driven approach are typically the teams that come out ahead.

In short, a team’s ability to measure and hit KPI’s and service level agreements with their customers directly influences the overall success and health of that team. If an organization isn’t producing applications that are reliable and available when their customers need them, it’s unlikely that company will have customers for very long. When reliability issues do occur, it’s critical that developers can immediately go to the source of the issue in their code to ensure that they can solve those customer issues as quickly and efficiently as possible. One such approach is to give developers the ability to do real-time debugging directly in the environment where those issues occur, without needing to create costly reproduction environments or having to figure out how to reproduce issues locally that are, many times, data dependent.

Reliability is Shifting Left

Historically, software reliability has fallen on the shoulders of operations or production support teams who manage the deployments and runtime of software after it’s been deployed to production (and after developers have thrown the code over the metaphorical wall). Things have changed in the recent years, in that many of today’s development teams are responsible for their applications from development all the way through to running and supporting those applications in production. Because of this, developers have more incentive than ever to ensure that their software has reliability built in, or at least the ability to quickly debug reliability issues when things go wrong.

Real-time debuggers are one such category of tools that development teams are choosing to package with their applications in order to solve those hard-to-address customer issues faster. Real-time debuggers are solutions which allow development teams to gather code-level debugging information such as snapshots of local variables, stack traces, tracing information and profiling data. All of this data (which traditionally relies on developers having the right logs in their code and, if not, deploying new debug builds) becomes instantly available, on-demand whenever an issue arises. This faster approach to identification of issues can drastically improve the reliability of applications by allowing developers to solve issues in a fraction of the time it would normally take.

Teams Focusing on Reliability Move Faster

Many development teams have shifted to an agile approach to delivering software; the goal being to respond quickly to business needs and to get new, valuable features to their customers as quickly as possible. Even though this is a helpful step, unplanned events can and will happen, which need to be planned for as much as possible ahead of time. In a recent Forbes article by Rookout CEO Shahar Fogel, he discusses how businesses should consider these software bugs as mini outages. Any time teams are dealing with software bugs or other unplanned issues, it costs the business revenue and can potentially leave customers frustrated. Focusing on metrics like mean time to repair (and giving developers the tools they need to reduce the mean time to repair) can give teams the ability to better predict how long unplanned issues will take to remediate once they occur.

In the book Accelerate by Nicole Forsgren, Jez Humble and Gene Kim, the authors discuss the notion that, by giving developers tools to fix problems when they occur, teams create an environment where developers accept responsibility for global outcomes such as quality, stability and reliability. Investments in technology are also an investment in people. So, it’s no wonder that by investing in technologies that can help improve reliability, teams will be able to move faster. This means happier customers, happier employees and a thriving business.

By focusing on service reliability, development teams are not only able to move more quickly, but are also directly contributing to the overall health and success of their organization. We’re seeing the onus of reliability shifted from operations teams to development teams, which adds newfound importance on ensuring development teams are building reliability into their product. Real-time debugging tools are one potential solution to ensuring that developers have the ability to quickly address unplanned customer issues which can directly impact the perceived reliability of an application.

June 17, 2021
Learning ‘The Third Way’ of DevOps – Continuous Improvement
I refer to the third way of DevOps–continuous experimentation and learning, as simply continuous improvement. In some sense, there is no way to learn the third way of DevOps because it is, by definition, a moving target. It does make sense to discuss what is needed to create a continuous learning organization for DevOps continuous improvement that builds on mastery of the first way and second way of DevOps.

As indicated in my blog Learning ‘The Second Way’ of DevOps – Continuous Feedback and my prior blog Learning ‘The First Way’ of DevOps – Continuous Flow, it is clear many organizations are struggling to realize well-engineered DevOps, even the first way of DevOps, continuous flow.

The first way of DevOps, continuous flow, is the foundation for DevOps. Once mastered, an organization will naturally want to master the second way of DevOps to achieve the higher level of benefits that continuous feedback offers, when engineered well. Once the second way of DevOps is achieved, then an organization will naturally want to master the third way of DevOps, to sustain and grow their DevOps capabilities.

DevOps is an evolving body of knowledge. There is no one-stop shop that covers all roles and the complete range of skills required for DevOps. A DevOps-competent workforce requires an ongoing commitment to DevOps skills development and mastery. Enterprises must approach DevOps training strategically to leverage training resources to their best advantage.

Learning Continuous Improvement

There are no training resources dedicated solely to the third way of DevOps—continuous improvement. Many of the same resources as the first way of DevOps and the second way of DevOps do have some relevant information, though. Analysis of DevOps results, third-party research sources, industry sources such as publications, articles, conferences and tool vendors roadmaps for DevOps and related frameworks including site reliability engineering (SRE) and AIOps are good sources of information that stimulate self-directed teams to seek the training information that is most relevant to the organization’s DevOps improvements.

Why Is Learning Continuous Improvement Important to Engineering DevOps?

The third way of DevOps—continuous improvement—requires an organizational commitment to continuous learning and mastery at the most advanced level. Typical skills needed include implementation and operation of advanced practices for microservices design and operations, advanced continuous testing, multi-pipelines, value stream management, SRE, observability, DevSecOps and AIOps. Retention and development of highly skilled and hard to find experienced DevOps professionals and SREs makes it important to develop skills in-house.

How Is Learning Continuous Improvement Accomplished?

The following are recommended engineering practices for learning continuous improvement:
- A continuous learning culture encourages individuals to continuously gain knowledge and exchange experiences and ideas for DevOps and SRE.
- Mentoring is an explicit program.
- Retrospectives are structured to thoroughly analyze significant, unexpected events and determine solutions that will prevent the events in the future.
- An active online library of DevOps and SRE training resources is maintained that tracks the latest and most innovative approaches to DevOps and SRE.
- DevOps and SRE teams evolve their skills and advance their careers by engaging with industry thought leaders to continuously update training content.
- Partner with trusted education and training partners ensure the highest-quality learning content and tools are made available to DevOps and SRE teams.
- A portion of DevOps and SRE teams time is allocated for training and education.
- Cross-team sharing of recommended engineering practices is proactively pursued using the concepts of Yokoten—proactive sharing of practices between teams.
What Is Needed to Implement Learning Continuous Improvement?

The following are recommended engineering practices for implementing a learning program for continuous improvement.
- Assign roles and responsibilities to experts for defining a training program for DevOps continuous feedback (e.g., DevOps sensei, training PM, evangelist).
- Conduct a continuous feedback DevOps skills gap assessment.
- Define a training strategy that best address the gaps and fits the needs and priorities of the organization.
- Define a training budget that includes expenses and time allocations per role and skill.
- Determine which skills and roles can be covered by inexpensive self-study resources.
- Determine which skills and roles warrant more costly commercial training.
- Identify training goals for each role.
- Ensure recruiting job profiles match skill requirements for DevOps skills that are most relevant to the organization’s strategic goals.
- Create a continuous learning culture that encourages individuals to continuously gain knowledge and exchange experiences and ideas for DevOps and SRE.
- Build an active online library of DevOps and SRE training resources that tracks the latest and most innovative approaches to DevOps and SRE.
- Engage with industry thought leaders to continuously update con tent for DevOps and SRE teams to evolve their skills and advance their careers.
- Partner with trusted education and training partners to ensure the highest-quality learning content and tools are made available to DevOps and SRE teams.
- Put in place means to make training resources visible and available.
- Set training goals for teams and individuals.
- Put in place incentives and recognition programs for training.
- Track training progress.
Some of the best sources of DevOps training resources are listed below:
1. The DevOps Institute
2. Partners of The DevOps Institute, such as Learning Tree International and Xellentro.
3. Udemy.com
4. Recorded talks and publications of IT Revolution
5. Books such as my own Engineering DevOps, by Marc Hornbeek.

What This Means

The third way of DevOps—continuous improvement—requires an organizational commitment to continuous learning and mastery at the most advanced level. Typical skills needed include implementation and operation of advanced practices for microservices design and operations, advanced continuous testing, multi-pipelines, value stream management, SRE, observability, DevSecOps and AIOps. Analysis of DevOps results, third-party research sources, industry sources such as publications, articles, conferences, and tool vendors roadmaps for DevOps and related frameworks including site reliability engineering (SRE) and AIOps are good sources of information that stimulate self-directed teams to seek the training information that is most relevant to the organization’s DevOps improvements.

Organizations must approach DevOps training strategically and continuously to leverage training resources to best advantage. To learn more about my blueprint for learning all things DevOps, refer to my book Engineering DevOps.
June 17, 2021
Milestones and Certifications

June 17, 2021
The Future of Cloud Testing

Roundtable 10: Cloud-native application development has helped organizations stay ahead by delivering faster time to market. The growth in cloud computing has also presented new testing challenges – traditional testing isn’t enough to address security holistically for cloud-native apps.

June 16, 2021
WSO2 Applies AI to API Management

WSO2 today unfurled a Choreo cloud platform through which DevOps teams can create and manage application programming interfaces (APIs) more efficiently by leveraging machine learning algorithms and other forms of artificial intelligence (AI).

Eric Newcomer, WSO2 CTO, said the integration platform-as-a-service (iPaaS) eliminates all the tasks associated with setting up a platform for and managing services based on APIs.

Choreo, available in beta, employs AI to learn historical activities and performance behaviors in order to anticipate the needs of a developer. As developers build an API, the platform will provide performance feedback, identify anomalies, map data and even complete code, said Newcomer.

The Choreo platform also comes with built-in observability tools for monitoring API usage and troubleshooting APIs, added Newcomer.

In addition, Choreo provides access to the open source Ballerina programming language the company developed to enable developers to create APIs using both low-code and procedural code as they see fit, noted Newcomer. The low-code diagram that developers create generates source code that developers can then run wherever they want, said Newcomer. Developers can close the entire repository Choreo provides, Newcomer added.

Developers can also trigger low-code integration logic to control hundreds of software-as-a-service (SaaS) APIs via pre-built connectors and templates provided by WSO2 or ones they create themselves.

WSO2 is also making it possible to deploy APIs via a single click in Kubernetes environments, which eliminates the need to manage cumbersome YAML files. That capability is based on technology that WSO2 gained via the acquisition of Performer that the company revealed today.

Finally, WSO2 is making available an API marketplace through which organizations can share APIs, event streams, data sources and templates.

With the rise of microservices-based applications, API management has become a lot more challenging. Each additional microservice adds an API that needs to be managed and maintained. As more APIs are employed, it becomes increasingly difficult for DevOps team to keep track of the dependencies that exist between all those APIs without the aid of machine learning algorithms.

It’s not likely machine learning algorithms will replace the need for DevOps professionals anytime soon. However, it’s becoming apparent that many of the manual tasks that DevOps teams routinely perform today are becoming automated by machine learning algorithms. In other cases. DevOps teams are gaining new capabilities that would have either been impossible for them to do on their own or would simply have taken too much time to complete.

Of course, it takes time for any AI platform to learn the DevOps environment. IT teams need to invest a significant amount of time training an AI model to recognize relevant issues in much the same way they might train any new member of a DevOps team. The only difference is, once a machine learning algorithm learns something, they never forget it. Nor do they take a day off or announce one day they are leaving the company because they got a better offer somewhere else.

It may take a while before DevOps teams are able to strike the right balance between man and machine, but one thing that is certain is that AI in DevOps environments is here to stay.

June 16, 2021
Harness Employs ML to Streamline DevOps Workflows

At the online {Unscripted} 2021 conference, Harness today announced an update to its namesake DevOps platform that includes support for feature flags that expose new capabilities to a select number of users for testing purposes along with the ability to prioritize the running of those tests based on the likelihood an application is likely to fail them.

In addition, Harness is adding a unified pipeline capability that enables DevOps teams to streamline the management of multiple tools that make up a workflow and has updated its cloud cost management module to enable DevOps teams to automatically stop idle workloads. That capability is based on technology the company gained with its recent acquisition of Lightwing.

Steve Burton, chief marketing officer for Harness, said Harness is now employing machine learning algorithms to identify which tests an application is most likely to fail. That test intelligence module enables DevOps teams to run those tests first versus testing an application only to discover close to the end of that effort that an application will need to be updated and then retested. The amount of time spent on testing is then sharply reduced, noted Burton.

Overall, Harness is reducing the amount of time and effort spent maintaining DevOps pipeline and workflows, said Burton.

The cost of maintaining a DevOps platform will naturally vary as the overall environment becomes more complex. It may not be clear how much time a DevOps team is spending on testing, but the faster that process becomes the more likely an application will be thoroughly tested before it is deployed in a production environment. Fixing applications after they are deployed in a production environment is, of course, considerably more expensive.

At the same time, as the number of applications being built simultaneously increases, the amount of cloud resources being consumed also increases. Developers, however, don’t typically pay close attention to what workloads are consuming which cloud resources, which can result in an unpleasant surprise when the monthly bill from the cloud service provider comes due.

In general, Burton says machine learning algorithms and other forms of artificial intelligence (AI) will soon become table stakes for any provider of a DevOps platform. The expectation is the platform will routinely employ AI to streamline processes and reduce total costs, noted Burton.

Many of the manual tasks that DevOps teams regularly perform to maintain a DevOps platform will increasingly become automated as usage of machine learning algorithms becomes more pervasive. It’s not clear to what degree organizations may abandon a legacy DevOps platform to take advantage of those capabilities. However, over time it will become apparent that some organizations are spending a lot more time building and deploying applications than others as the next generation of DevOps platforms, infused with machine learning algorithms, are installed.

In the meantime, DevOps teams might want to make a list of the manual tasks they perform today as part of an effort to determine how much time they are wasting that might otherwise be spent on tasks that are more valuable to the organization.

June 16, 2021