Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: SUSE announces a Red Hat Enterprise Linux fork after Oracle said a similar thing, and Sarah Silverman says OpenAI and Meta have stolen her words. (more…)
JFrog today announced it is adding a curation capability for open source software to its portfolio that will use metadata generated by binaries to identify malicious packages and software components that have licensing issues.
Sharan Hiremath, senior product manager for JFrog, said JFrog Curation is an extension of the JFrog Artifactory binary repository that the company provides to enable organizations to manage their software supply chains.
JFrog Curation also validates software packages against JFrog’s Security Research library of recorded Critical Vulnerabilities and Exposures (CVEs) and publicly available information to help establish a trusted repository of pre-approved, third-party software components.
That approach automatically employs the JFrog Xray scanning tool to eliminate the need to individually scan each package before developers employ them, said Hiremath.
Making sure open source software components are secure is often problematic because developers download them from public repositories that are frequently targeted by cybercriminals. There have now been more than 100,000 instances of malicious software packages discovered in repositories in the last year alone, noted Hiremath.
In addition, cybercriminals have become adept at creating fake repositories that mimic legitimate repositories; that can result in developers downloading software components infected with malware.
JFrog Curation takes the shift left concept of application security to the next level by automatically blocking the use of risky open source software packages, noted Hiremath. The JFrog platform will also surface advice on how to remediate any vulnerability discovered, in addition to creating an audit trail to enable organizations to comply with various regulatory mandates.
While open source software components reduce the cost of application development, from a cybersecurity perspective, it’s the wild, wild West, noted Hiremath. Organizations are trying to bring more discipline to the way software is constructed by adopting DevSecOps best practices, but they need an approach that doesn’t introduce any additional friction into the application development process, he added.
JFrog Curation addresses that issue by addressing cybersecurity and compliance issues before any open source component becomes part of the build, said Hiremath.
As regulations to hold organizations more accountable for the security of the software they deploy are being proposed around the world, it’s only a matter of time before organizations must revisit the security of their software supply chains. Cybercriminals have already proven how adept they are at injecting malware into software components that could be used in thousands of downstream applications. In the eyes of regulators, organizations that don’t scan open source software for malware and vulnerabilities are engaging in reckless behavior that will warrant stiff penalties.
The challenge is that, after giving developers free rein to develop software any way they see fit, organizations are now faced with a challenge that is as much cultural as it is technical. Automated processes will naturally play a key role in curbing aberrant behavior in application development, but there is, as always, no substitute for some good old-fashioned common sense.
Everyone thought AI would free us from toil. Unfortunately, that’s not true. What it will do, however, is lead to an explosion of technical debt. Some of the most alluring uses of LLMs have the potential to lead to some of the worst outcomes for enterprises and their DevOps and platform engineering teams.
SK Ventures and Adrian Cockcroft raised the alarm bells in their warning posts about the tipping points LLM introduced into work and society. And in my own Gluecon 2023 presentation, I challenged just how powerful and disruptive the new generation of AI/ML will be for ITOps and DevOps efforts.
There is no doubt that LLMs are powerful and will change how we think about expertise and knowledge work. While they are not truly innovating, their ability to draw on such a vast pool of human expertise makes it difficult to distinguish from true creative work. Since they are unbounded, they will cheerfully use the knowledge to churn out terabytes of functionally correct but bespoke code.
It’s easy to imagine a future where LLMs crank out DevOps scripts 10x faster. We will be supercharging our ability to produce complex, untested automation at a pace never seen before! On the surface, this seems like a huge productivity boost because we (mistakenly) see our job as focused on producing scripts instead of working systems.
But we already have an overabundance of duplicated and difficult-to-support automation. This ever-expanding surface of technical debt is one of the major reasons that ITOps teams are mired in complexity and are forever underwater. Generating even more scripts, even if they were better written, does not reverse the trend.
It depends on how we use the expertise in the models. Instead of asking it to generate new code, we could ask it to interpret and modify existing code. For the first time, we have tools to take down the “not invented here” barriers we’ve created because of the high cognitive load of understanding code. If we can help people work more effectively with existing code, then we can actually converge and reuse our systems. By helping us expand and operate within our working systems base, LLMs could actually help us maintain less code.
Imagine if the teams in your organization were invested in collaborating around shared systems! We haven’t done this well today because it takes significantly more time and effort. Today, LLMs have thrown out those calculations.
Taking this just one more step, we can see how improved reuse paves the way for reduction of the number of architectural patterns. If we improve our collaboration and investment in sharing code, then there is increased ROI in making shared patterns and platforms work. I see that as a tremendous opportunity for LLMs to improve operations in a meaningful way.
It comes down to whether we ignore complexity by replacing it or embrace complexity by understanding it. I see this as a choice between independence or collaboration, as illustrated by the table below.
|
LLM Driving Independence |
LLM Driving Collaboration |
|
Create new code |
Interpret existing code |
|
Replace existing code |
Maintain existing code |
|
Design new architectures |
Be a guide and mentor |
|
Build new stacks |
Evaluate alternatives |
|
Bypass documentation |
Improve documentation |
We keep taking the same hardware and software components and then building unique systems. That variation makes it impossible to leverage each other’s work. At its heart, the reusability problem lies in the difficulty of understanding systems, and that makes it nearly impossible for us to work together, collaborate, reuse and accelerate.
Breaking this cycle has been our mission at RackN for nearly a decade. We’ve seen tremendous ROI from availability, reliability and speed when customers are able to leverage standardized infrastructure pipelines. We’ve seen firsthand that it is possible to guide teams into collaboration with sufficient guidance, mentoring and training. Collaboration creates 10x outcomes even without AI, but it takes building expertise, deliberate effort and leadership discipline to implement.
Why does collaboration create 10x ROI? It works because true innovation actually comes from understanding and building on top of systems rather than rebuilding them from scratch over and over again. The key is being able to leverage LLMs to better understand existing automation and platforms instead of creating yet another bespoke stack.
I believe that the most effective enterprises will use these tools to elevate their IT leaders and architects to focus on collaboration and reuse. They must actually direct them to produce less bespoke automation; instead, they can use expertise in these tools to find and maintain existing systems. LLM presents options, but collaboration requires new thinking and strong leadership.
A funny thing happened on the way to being agile. We adopted a ton of tools to make us more responsive. They really do help us deliver software quickly and efficiently, and they have improved IT processes along with our specific process improvements.
They also created a toolchain that resembles areas of responsibility of old. That makes perfect sense; we still need to store source code, build applications, deploy somewhere, worry about security, etc. And we need to do all of that faster, which is where automation and the toolchain really shine.
But you need to keep in mind the overall impact of the toolchain upon the development process. Now the organization is dependent upon those tools. We went from slow and steady to fast and with a large series of steps along the toolchain. Things are better, we’re delivering quality software at a faster pace, but in some ways, the process risks becoming more fragile. If a tool in the chain shows signs of trouble, you need to be aware of it and on top of the issue almost immediately. The price of ignoring issues in the process can be catastrophic disruptions in software delivery. Losing a CI tool is akin to shutting down customs at an international airport: The source of the problem is big, and the downstream problems are show-stopping.
So while you are tooling your applications, make certain you are tooling the build chain, also. Know when a build that always takes ten minutes suddenly spikes to triple that. Know when a test suite delivers non-breaking errors at an increased rate. Know when space is low on a shared drive that the build tools use. Tool everything, because the software delivery toolchain is now the core of application development and deployment.
Tooling these systems can also offer a warning of problems in a given application. It is possible that the increase in build time is because there are a lot of previously unused libraries being included … and that those libraries have not been properly vetted by security. It is possible that the disk space on that shared drive has PII test data that needs to be cleansed. There is a lot of good that can come from over-tooling the build chain, and unless you have a bit of tooling that slows the process, there is no downside. It won’t take too long to build a list of what to watch, and modern DevOps tools have enough reporting and APIs that it won’t take long to begin watching those items, either.
That is where the people part comes in. Don’t ignore the tooling once it is in place. Build a process to regularly review results and take action when necessary. No tooling is useful if the results are ignored; sadly, we have a long history of spectacular failures that started with, “We saw indications, but didn’t think it was important at the time …” So act on results, even if that just means flagging them for review in the next reporting cycle—just don’t ignore them.
And keep rocking it. The organization thrives on the software you build, deploy and support. It is a massively complex system that has IT staff at its heart, so keep it thriving and watch for indicators of illness in the core toolchain.
Roadie today made generally available a tool that automatically tracks and reports how software development teams are performing against engineering standards defined by a DevOps team. The Tech Insights tool is an addition to Roadie’s implementation of the open source Backstage developer portal and is delivered via a software-as-a-service (SaaS) platform.
Roadie CEO David Tuite said Tech Insights is a proprietary tool that Roadie developed as an extension to its platform. Tech Insights provides scorecards that track software assets in the Backstage catalog and ensure they are meeting predefined quality and compliance targets.
Tech Insights comes with built-in, pre-defined data sources; alternatively, DevOps teams can define them using external application programming interfaces (APIs) or files in a repository. A rule known as a Check can then be created to validate whether a project complies with the engineering standards defined by the DevOps team. Each DevOps team can create an unlimited number of scorecards.
In addition, teams can create prompts and notifications to warn engineers when they are about to make a change that will violate an engineering standard. DevOps teams can also employ gamification techniques to show teams how their performance compares to others.
While Roadie is now adding its first proprietary tool to the platform, Tuite said the company remains committed to enabling organizations to migrate their instance of Backstage to another environment should they ever deem it necessary, he added.
Originally developed by Spotify, Backstage makes it possible to create and manage a catalog of blueprints that developers can readily consume rather than requiring them to repeatedly build these capabilities themselves. The goal is to create scaffolds that developers can consistently reuse across multiple application development projects.
Spotify has since donated Backstage to the Cloud Native Computing Foundation (CNCF). However, Backstage currently requires a significant amount of DevOps expertise to set up and maintain, an issue that Roadie is addressing by making it available as a SaaS platform that can be accessed by any application development team, said Tuite.
The concept of a self-service portal through which developers can address their own needs may not be new, but as an open source project, Backstage is lowering the cost of improving overall developer productivity. At the same time, many organizations are now embracing platform engineering as a means of centralizing the management of DevOps.
Ultimately, every minute developers spend on tasks other than writing code is one less minute they are spending writing the business logic that drives an application. Measuring developer productivity is, of course, an inexact science. No one knows for sure when inspiration might strike, but if developers are busy maintaining code that they created to provision infrastructure, it’s less likely they’ll be struck by a flash of brilliant insight that improves and application. The easier it is to spin up a development environment, the more likely that inspiration will one day turn into code that runs in an actual production environment.
As organizations digitally transform to support the needs of customers and help them win in their markets, IT leaders, software developers and DevOps teams have emerged as the central players in the transformation landscape. They are also key advocates for improving the developer experience and promoting technology adoption—both critical components of a successful digital transformation. They are also on the front lines of understanding operational and team challenges. Their understanding of the business objectives and the role people and technologies play in support of achieving these goals makes IT an indispensable driver of value creation.
Digital transformation has also fueled the swift adoption of Kubernetes and cloud-native technologies, which promise enterprise speed and scalability. As these technology adoptions accelerate, so too does the demand to make developer experience a priority. After all, developers are tasked with creating and delivering software to the market quickly to support business objectives.
As more organizations adopt cloud-native development methodologies, developers have taken on a more strategic role across the business. No longer just writing code, they have been asked to take on ownership of the entire application life cycle. In the cloud-native environment, this shift in developer responsibility comes with an evolving ecosystem of technologies and tools that can have enormous complexity and impact how dev teams collaborate as they code, ship, test and run apps to support business goals.
Asking developers to take on this level of responsibility is not ideal for productivity. Most developers want to know enough about the full software development life cycle (SDLC) and infrastructure to get their work done but do not want to become Kubernetes experts. Giving developers a good developer experience relies on understanding the importance of creating an experience that removes friction, fosters innovation and fuels the ability to move faster.
Here are four tips to ensure the developer experience is not your Achilles heel in fueling business performance:
Share the Why: Don’t just tell developers how to do something. Explain why they’re doing it. Storytelling and making the ‘why’ matter play an often hidden but important role in undoing developer frustration and, hopefully, avoiding it altogether.
With the pressure to do more with less, ensuring that developers understand the reasons behind their actions and activities is key to relieving their frustrations and getting to safer, faster deliveries that underpin business value. Developers don’t like black boxes, and the more you can share upfront about why and how the plan or strategy works, the better the developer experience.
The 2022 Stack Overflow Developer Survey found that well over half of the developers polled spent more than 30 minutes a day looking for solutions to problems. Not only is this a distraction from core development work and productivity, but it can also account for up to 650+ hours of lost time per week (in a team of 50 developers). Informing developers about the ‘why’ from the start—and then arming them with the right tools—can alleviate both developer and business challenges.
Reduce Friction and Preserve Existing Workflows: Developers want to focus on what they do best—writing code. They don’t want to be hunting down answers to problems created by a growing list of complex technologies, processes and tooling that impact their ability to write and ship software fast. Most developers don’t want to think about selecting or having to use a variety of different (or new tools) to support their work.
Clear the path for developers. Find infrastructure that puts your developers in the driver’s seat and allows them to continue using the toolchains they already know and love and the power to pick and choose which capabilities best support their requirements for fast, scalable app delivery.
Foster Team Collaboration and Communication: Faster, safer development is not just about developer experience, but includes the experience of the entire development team and its ability to collaborate. The entire DevOps team plays a critical role in getting software out the door. Site reliability engineers (SREs), for example, play a key role in guiding developers through the cloud-native learning curve to use platforms and ecosystems effectively. It’s more than firefighting—it is a trusted partner to create success in deploying, releasing and running services.
Encourage developers to share pain points and learn about tools and best practices from the SRE team. Rethink communication patterns to unify developer and SRE approaches to problem-solving, i.e., remove the “this is a developer problem” or “this is an SRE problem” to reframe them as “this is a business problem” and shared responsibility.
Let Developer Voices be Heard: Empathy cushions every good developer experience. This includes listening to and understanding the needs and wants of developers. Offering more training, having a robust onboarding experience, focusing on reducing cognitive load and having an ongoing dialog with developers shows empathy and reinforces your commitment to supporting developer skill sets and roles within the organization.
Organizations that empathize with their developers can eliminate friction and fuel a more collaborative, supportive onboarding experience that allows developers to provide input as they progress in their journey. Continuing that empathy throughout a developer’s time at an organization fosters an overarching culture of communication, collaboration and empathy.
Additionally, institute measurable and realistic key performance indicators (KPIs) as this will help organizations quantitatively understand the developer experiences and identify areas for improvement and celebration.
Many organizations that go cloud native to improve development speed and shipping frequency end up in a situation where they ask themselves, “Why is it taking hours to do what we used to do in minutes?” Here it is essential to return to the core question about developer experience. If developers are struggling with hours of repetitive, non-core work, a clear path is not paved for them. If they never got a solid onboarding, there’s no shared development environment, friction is everywhere and lines of communication are closed, it’s inevitable that the developer experience will suffer—and by extension, so will the business.
How you respond to the needs of the developer and the developer experience can make or break your business. Invest in the developer experience. It’s critical to fueling developer productivity and supporting a faster time-to-market for your business.
Stack Overflow has released its seminal yearly developer survey. The 2023 report, which surveyed over 90,000 developers, provides a comprehensive view of the current software developer experience.
Below, I’ll highlight a few important findings, namely the significant programming language and tooling preferences, the use of AI in the development workflow and what these trends could mean for the DevOps field. I also gathered insights from Joy Liuzzo, vice president of product marketing at Stack Overflow, to dive deeper into some of the findings.
Of note is that more and more developers are learning to code online. Learning to code using online resources increased from 70% in 2022 to 80% in 2023. Although many developers (47%) still have a Bachelor’s degree in computer science or the equivalent, these trends underscore the movement toward alternative knowledge solutions, especially for younger programmers. The topmost online resources include technical documentation, blogs, forums and how-to videos.
JavaScript is still king—it’s now in its eleventh year as the most common programming language. But it’s worth highlighting that Python has overtaken SQL as the third top language. “SQL was solidly in the top three (JavaScript, HTML/CSS, SQL) since 2015, so it’s a big deal for it to have fallen below Python,” said Liuzzo. “We’ve seen Python’s popularity increase based on the number of questions asked on our public site, so we’ve been expecting some shifts.”
TypeScript and Bash/Shell have also consistently grown in usage over the last few years. “These two languages are involved with the functions of other popular programming languages, explaining their prevalence among programmers,” Liuzzo said.
PostgreSQL also overtook MySQL as the most common database type. In terms of web frameworks, Node.js and React.js are the most dominant. Other frameworks, like jQuery and ASP.NET, are showing signs of going out of style, likely because they are older web frameworks.
The explosion of new AI innovations, such as large language models (LLMs) and chat-driven generative AI tools, had a major impact on this year’s technology findings. In fact, 83% of respondents have used ChatGPT over the past year. This is followed by Bing AI (20.6%), WolframAlpha (13.36%) and Google Bard AI (9.86%). GitHub Copilot was ranked as the top most-used AI developer tool.
There was a significant spike in using AI tools amongst those learning to code—these folks often see benefits like speeding up learning, increasing productivity and greater efficiency. “Pair this with how they are currently using AI tools to debug and get help (68%) and learn about a codebase (50%), and we see that the commonality is that AI tools enable learning,” explained Liuzzo.
That being said, there’s skepticism around AI’s accuracy, and only 13% see improved accuracy in coding as a benefit of using such tools. These pain points may resolve as LLMs evolve, but for now, there still needs to be human judgment to catch bugs and avoid hallucinations.
“AI has a complexity cliff. Much like Helmsman’s complexity cliff found in project management, after a certain point, the ability for AI to handle all the nuances and interdependencies of a solution drops off,” said Liuzzo. “That’s when humans, their ability to apply judgment and have original thoughts, saves the day.”
So what are some takeaways for those working in the DevOps field, namely DevOps tooling providers and platform engineers? Well, new tooling preferences and habits will continue to evolve in the DevOps field in new ways. Here are some potential examples:
Last but not least, generative AI is set to enhance programming in many areas and is already becoming embedded into many environments. Just as AI can be used for code generation and debugging, LLMs will likely assist the DevOps side of the equation, helping automate the CI/CD pipeline, optimize infrastructure, detect anomalies and provide more chat interfaces to initiate these capabilities.
“AI tools are empowering and enabling learning,” said Liuzzo. “Generative AI will democratize coding and grow the developer community by several folds—and that growing number of developers will be the ones using the tools and also verifying and validating the outputs as they learn—bringing the power of the developer community and the technological power of AI together.”
Loft Labs has made available an open source software project that enables DevOps teams to programmatically create developer environments using a tool that runs entirely on a local client.
Lukas Gentele, Loft Labs’ CEO, said DevPod provides an alternative to cloud-based approaches such as Codespaces from GitHub that lock organizations into a specific platform for building applications.
Instead, DevPod can automatically spin up developer environments as code on any public cloud or on-premises IT environment that can run a devcontainer.json file, he added.
In addition, DevPod will also automatically put developer environments to sleep when not being used to reduce costs, noted Gentele. Other capabilities include support for prebuilds, Git and Docker credential synchronization.
DevPod was created by Loft Labs to streamline the development of a virtualization platform for Kubernetes clusters. The company is now making DevPod available as open source software that any DevOps team can use to manage developer environments as code more easily than complex server-side alternatives, said Gentele.
It also makes it easier for developers to switch between cloud-hosted and local environments as needed seamlessly, he noted. That’s critical because it enables organizations to reduce dependencies on cloud computing environments for writing code by allowing developers to use their local machine whenever they prefer, noted Gentele.
Despite the ubiquitous availability of cloud services, most developers continue to prefer to write code on a local machine. The challenge is that replicating a developer environment elsewhere to foster collaboration has been problematic. DevPod provides a means for developers to collaborate with each other and with software engineers without giving up the inherent flexibility a local development platform provides.
At a time when organizations are more dependent on software than ever, there is much more focus on developer productivity. DevOps teams have historically tended to focus on DevOps Research and Assessment (DORA) metrics that focus more on deployment frequency and mean-time-to-recovery than they do on developer productivity.
Of course, measuring developer productivity is an inexact science. An organization might once have tracked lines of code written to gauge productivity, but that is not especially good metric. In more recent times, metrics have been developed that, among other things, keep track of how much time developers spend actually coding versus maintaining their software development environment.
The issue that most IT leaders will immediately encounter is that every minute developers spend on these tasks, however, is one less minute they spend writing the business logic that drives an application.
Obviously, there’s no way developers are going to write business logic daily for eight hours straight; there are maintenance windows. However, application development is as much art as it is science. No one knows for sure when inspiration might strike, but if developers are busy maintaining code created to provision infrastructure, it’s less likely they’ll have a flash of brilliant insight that allows them to make an application better. The easier it is to spin up a development environment, the more likely it becomes that inspiration will one day turn into code that runs in an actual production environment.
DevOps.com is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.
Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.
Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.
The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.
We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:
Harris Allied
New York, New York
DevOps Engineer
$160,000 to $175,000
CareerBuilder.com
MassMutual Financial Group
Remote
DevOps/Kubernetes Cloud Infrastructure Engineer
$137,800 to $180,800
Indeed.com
Lawrence Livermore Labs
Livermore, California
DevOps Engineer
$123,960 to $159,168
SimplyHired.com
CVS Health
Remote
DevOps Engineer
$90,000 to $180,000
Dice.com
Idea Entity
Herndon, Virginia
Azure DevOps Engineer
$80,000 to $100,000
In my blog Rapid Strategic SRE Assessments Accelerate IT Transformations published last year, I classified site reliability engineering (SRE) into nine pillars of SRE practices—a comprehensive framework that covers the full scope of SRE practices. This nine pillars SRE framework is consistent with the SRE blueprint published by the DevOps Institute. Each of these pillars represents a vital area of SRE implementation and, together, they ensure a holistic approach to aligning development and operations for effective collaboration, uneventful continuous deliveries and reliable, highly scalable, safe operations.
The current generation of AI-engineered tools is now being applied to each of the SRE pillars and has the potential to significantly increase efficiency, reliability and scalability for all SRE practices.
Just as SRE practices complement DevOps practices, these nine pillars of SRE complement the nine pillars of DevOps explained in my recent blog Revolutionizing the Nine Pillars of DevOps With AI-Engineered Tools.
The following briefly explains how the current generation of AI-engineered large language model (LLM) tools are being applied or could be practically applied to improve each of the nine pillars of SRE.
Culture: AI can help identify patterns and trends in system behavior that may not be immediately evident. This can assist in shifting the wisdom of production further to the left. Tools like Datadog’s Watchdog can be used to automatically detect and surface anomalies in your system behavior, for example.
Toil Reduction and Automation: AI can help reduce toil by automating more complex tasks that were previously challenging to automate. Machine learning models can predict system behavior, enabling proactive responses. AI operations platforms like Moogsoft or BigPanda can automate incident management, from detection to remediation.
SLAs/SLOs/SLIs and Error Budgets: AI can assist in refining SLAs, SLOs and SLIs by providing accurate predictions of system performance and identifying potential bottlenecks. Tools like Nobl9 can use machine learning to better predict and define SLOs based on historical performance data.
Measurements and Observability: AI can help analyze vast amounts of data from monitoring and observability systems, identifying patterns and correlations that may be difficult for humans to detect. Tools like Dynatrace and New Relic use AI to analyze monitoring data, detect anomalies and correlate events.
Anti-Fragility: AI can contribute to the development of more resilient systems by analyzing system behavior under stress and identifying vulnerabilities. The data gathered from chaos experiments using tools like Gremlin can be used in conjunction with machine learning models to predict system behavior.
Work-Sharing and Incremental Technical Debt: AI can help analyze code and infrastructure changes to assess their impact on technical debt. Tools like DeepCode and SonarQube use AI to provide guidance on how to manage this debt incrementally.
Deployments: AI can be used to optimize deployment strategies by predicting the impact of new releases. A tool like Harness can use machine learning to optimize deployment strategies and minimize risks.
Performance Management of Apps and Infrastructure: AI can analyze application and infrastructure performance data to identify optimization opportunities and predict capacity requirements. Tools like Turbonomic leverage AI to optimize resource allocation in real-time.
Incident Management: AI can assist with incident management by automating the detection and triage of incidents, helping to quickly identify the root cause. Tools like PagerDuty use AI to automate detection, triage and suggest potential remediation steps, leading to faster resolution times.
Applying AI to SRE is a complex process with certain challenges. Here are some potential pitfalls along with ways to address them:
Lack of Quality Data: AI and machine learning models are only as good as the data they are trained on. Inadequate or poor quality data can lead to inaccurate predictions and insights.
• Prioritize data hygiene and governance. Collect comprehensive and diverse data from your systems; ensure that it is well-structured and free of errors and store it in a way that’s easily accessible for training AI models.
Over-reliance on Automation: While AI can greatly enhance automation, relying on it too heavily without human oversight can lead to missed signals or overcorrections in response to false positives.
• Maintain a balance between automation and human oversight. Use AI to support decision-making, not replace it entirely. It’s important to have experienced SREs review AI outputs regularly to ensure they make sense and are beneficial.
Underestimating the Need for AI Expertise: Implementing AI is not just about buying and deploying a tool. It requires a deep understanding of AI and machine learning principles, as well as the ability to interpret the results correctly.
• Invest in training your team on AI and machine learning principles or consider hiring AI specialists. Having in-house AI expertise can help ensure that you’re using AI tools effectively and interpreting their outputs correctly.
Ignoring the Importance of Integration: AI tools need to work seamlessly with your existing infrastructure and processes to be effective. A lack of integration can lead to data silos and inefficiencies.
• When selecting AI tools, consider how easily they can be integrated into your existing infrastructure and workflows. Use APIs and other integration methods to ensure that your AI tools can access the data they need and that their insights are easily accessible to your SRE team.
Neglecting Ethics and Privacy: AI tools often need access to sensitive data to operate effectively, which can raise ethical and privacy concerns.
• Implement robust data privacy and security measures. Ensure that you’re following all relevant regulations and industry best practices for data privacy. Use anonymization and other data protection techniques to protect sensitive data.
Artificial intelligence is revolutionizing SRE practices by automating complex tasks, analyzing vast amounts of data and making proactive predictions. AI can reduce toil, enhance system understanding, and streamline incident management. It allows SREs to shift further left by integrating predictive insights into the development process, thereby enhancing the culture of proactive issue prevention. AI also aids in defining more precise service level objectives, error budgets, and optimizing deployment strategies based on predictive analytics. In the realm of incident management, AI aids in detecting, triaging, and mitigating incidents faster by identifying patterns and suggesting remediations. Overall, AI is fast becoming an essential tool for SREs, facilitating a shift from reactive problem-solving to proactive management, thereby increasing system reliability, performance, scalability and efficiency.