devopsy.kinsta.cloud

Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

  • Red Hat Adds Generative AI and Event-Driven Capabilities to Ansible

    Red Hat Adds Generative AI and Event-Driven Capabilities to Ansible

    Red Hat today unveiled a technical preview of Ansible Lightspeed with IBM Watson Code Assistant, a generative artificial intelligence (AI) service specifically trained on domain-specific data collected via the Ansible automation platform.

    At the same time, Red Hat is also making available an event-driven capability within version 2.4 of the Ansible platform that enables runbooks and other tasks to automatically execute when received from a third-party tool or application.

    Both capabilities were announced at the Red Hat Summit and are part of an ongoing Red Hat effort to democratize IT automation.

    First previewed within the Project Wisdom initiative launched by Red Hat and IBM Research last year, Ansible Lightspeed IBM Watson Code Assistant is based on a large language model (LLM) that was developed using Ansible data. Watson Code Assistant, currently in technical preview, is a service that automatically generates suggestions to improve code as developers create an application. IBM promised to make IBM Watson Code Assistant available later this year.

    Red Hat is now applying that generative AI capability to the YAML code that DevOps teams need to create Ansible playbooks. IT teams can create a simple natural language prompt that will translate their domain expertise into YAML code for creating or editing Ansible Playbooks. Red Hat and IBM, however, will not automatically use the code from an organization to fine-tune their AI model without express consent.

    Rich Henshall, director of product management for Red Hat Ansible, said as IT environments become more complex, it’s clear many are hitting a ceiling in terms of what can be automated using legacy custom scripts. Ansible Lightspeed IBM Watson Code Assistant, along with an event-driven capability, will make it much simpler to automate the management of IT environments at scale using a declarative framework, he added.

    Ansible, for example, can now receive events from third-party tools, determine next steps and consistently initiate an automated response using Ansible playbooks. Ansible can also consume events from third-party monitoring, observability and IT analytics tools from providers such as Cisco ThousandEyes, CyberArk, Dynatrace, F5, IBM Instana, IBM Turbonomic, Palo Alto Networks and Zabbix with more to come, noted Henshall. Red Hat is also making available supplementary content for Red Hat OpenShift, Red Hat Insights, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform and ServiceNow, in addition to enabling IT teams to create their own custom integrations.

    At this juncture, it’s not clear why more IT processes have not been automated using Ansible or other frameworks. Many DevOps teams continue to rely on custom scripts that typically don’t scale and are generally poorly documented. Any time the individual who wrote those scripts leaves an organization, it’s incredibly difficult—if not impossible—for the original scripts to be maintained, noted Henshall.

    One way or another, it’s clear that a more scalable approach to IT automation is required, especially with the rise of more complex cloud-native application environments. It’s no longer economically viable for most organizations to rely on legacy processes that require a small army of IT professionals to maintain.

  • 8 Hot Takes: Will We See a Monolithic Renaissance?

    8 Hot Takes: Will We See a Monolithic Renaissance?

    Is the DevOps world slowly backing away from microservices and to a monolithic renaissance? After years of proselytization about the benefits (Flexibility! Resilience! Support faster release speeds!) of microservices and serverless architecture, there are signs of a potential backlash against the singularly fragmented microservices architectural approach.

    The argument in favor of rolling back to monolithic architectures feels similar to the cloud repatriation debate. Just as with a lot of cloud-first transformations—often in lockstep with them, actually—many organizations moved headlong toward microservices without a case-by-case business or technical rationalization for the change. As the shiny newness of microservices has dulled under that approach’s daily exposure to real-world deployment and business challenges, the monolith isn’t looking so bad in a lot of situations.

    The monolithic versus microservices debate boiled over the last few weeks in the wake of the publication of a case study earlier this month from the Amazon Prime Video team. The analysis from Marcin Kolny, senior software development engineer for Prime Video, reported that his team was experiencing scaling bottlenecks in its audio/video quality inspection tooling, which was architected as a distributed system using serverless components. By rearchitecting the tool as a monolith, they were not only able to increase scaling capabilities but drastically reduced the infrastructure costs around that tool.

    “Moving our service to a monolith reduced our infrastructure cost by over 90%,” Kolny wrote.

    The announcement elicited a fair degree of hyperbole on the topic, but it also spurred a very organic airing of the pros and cons of monolithic, microservices and serverless architectures.

    The following are 10 different perspectives spurred on by this online debate from a range of CTO and engineering experts. Some of them are diametrically opposed to one another, many are overlapping in their opinions. But as a body, they point to the general (and very pragmatic) consensus that there simply will never be a one-size-fits-all approach to software architectures.

    Microservices a Siren Song for Complication

    “It’s clear that in practice, microservices pose perhaps the biggest siren song for needlessly complicating your system. And serverless only makes it worse. Replacing method calls and module separations with network invocations and service partitioning within a single, coherent team and application is madness in almost all cases.”

    —David Heinemeier Hansson, creator of Ruby on Rails and CTO at 37signals (maker of Basecamp and HEY)

    Architectures Aren’t Good or Bad

    “Application architectures are neither good nor bad, nor are they appropriate for every use case. Organizations need to step back and consider carefully what it is they’re trying to achieve and which architecture might best serve that need instead of choosing the most “modern” architecture because, well, it’s in vogue.”

    —Lori MacVittie, distinguished engineer in office of CTO at F5Networks

    Distributed Architectures Will Suit AI-Driven Future

    “In the ongoing debate between monolithic and serverless architectures, it is important to acknowledge that both approaches have their merits and considerations. Monolithic architectures have been prevalent in large human-centric organizations due to their management advantages and historical use cases. However, as the world continues its rapid progression towards a more AI-driven future, the landscape is shifting towards architectures that embrace fluidity, autonomy, and distribution.”

    — Reuven Cohen, veteran CTO and entrepreneur, mentor for TechStars

    Monolithic and Serverless Each Have Pros And Cons

    “Over the last 15 years, everyone has wanted to move to serverless microservices-based horizontally-scaled architecture – but it is not ideal for every application’s specific needs.  As with any choice in life and in business, there are always tradeoffs;  you must evaluate and choose the appropriate architecture based on your business and technology needs. Another important dimension to this is workforce skills for application architecture.

    —Hemang Davé, client technical leader and CTO at Kyndryl

    The Amazon Rearchitect Wasn’t Even Truly a Monolith

    “The problem is that they called this refactoring a microservice to monolith transition when it’s clearly a microservice refactoring step, and it is exactly what I recommend people do in my talks about serverless first. I don’t advocate “serverless only”, and I recommended that if you need sustained high traffic, low latency and higher efficiency, then you should re-implement your rapid prototype as a continuously running autoscaled container as part of a larger serverless event-driven architecture, which is what they did. If you built it as a microservice to start with, it would probably take longer (especially as you have to make lots of decisions about how to build and run it) and be less able to iterate as you figure out exactly what you are trying to build.”

    —Adrian Cockcroft, longtime DevOps advocate and veteran architect with stints at eBay, Netflix, Amazon, and Battery Ventures, among others

    One Tool as a Monolith, Maybe, But Not All of Amazon

    “Microservices, while duplicative and inefficient from a compute standpoint, allow you to release stuff really quickly, which is a tradeoff that most organizations at scale are willing to make.

    Microservices exist to reduce communication pathways—not to solve technical scaling challenges. At a certain point, teams stop functioning due to the communication overhead. A seven-person team has 21 communication pathways, whereas a 100-person team has 4,950. I work with large retailers selling billions of dollars a year online, and putting everything into a single monolith just doesn’t work. Could the entirety of Amazon.com be deployed as a single monolith? No way.”

    —Kelly Goetsch, chief strategy officer at commercetools

    Architectural Tension Always Exists Between Speed and Maintainability

    “Speed to market and speed to develop does not always equate to long-term scalability and maintainability. You must actively balance your investments across these two critical pillars to build a viable product. Active rebalancing and reconsideration of the stack and technology mix for your FinOps and BizOps context is both art and science.

    Cloud-native distributed systems paradigms are here to stay. The Prime Video folks have mistakenly labeled a properly factored, highly coupled data-intensive processing step a monolith—it’s at best a rocky outcropping in their distributed microservices forest. “

    —Ajish George, managing director of cyber data & analytics for State Street

    The Shiny Thing is Not Always the Best Thing

    “The news that AWS discovered that serverless and microservices aren’t for everything is surprising. Not because it didn’t work out for that use case; but because the wrong use cases have been so well-known for the serverless/microservices pattern and should have been obvious to Prime Video. It’s worth remembering: That shiny new thing isn’t necessarily a better way of trapping mice. Right tool for the right job.”

    —Doug Vanderweide, director of cloud architecture for Avanade

  • Despite DevOps, Software Supply Chain Security Challenges Persist

    Despite DevOps, Software Supply Chain Security Challenges Persist

    A survey of 397 IT, cybersecurity and application development professionals conducted by Enterprise Strategy Group (ESG) found that while most respondents work for organizations that have adopted DevOps practices, multiple software supply chain security issues and DevSecOps concerns have yet to be addressed.

    The survey, conducted on behalf of Data Theorem, a provider of a platform for securing application programming interfaces (APIs), the survey also found 82% of respondents have adopted DevOps, with 51% reporting they make extensive use of DevOps workflows (51%). Nearly three-quarters (73%) are deploying code weekly and 31% are deploying code daily.

    However, securing DevOps workflows remains a work in progress, with 41% reporting security teams lack visibility and control of development processes. Other issues include new builds deployed to production with misconfigurations, vulnerabilities and other security issues (40%), developers skipping security processes (39%), software released without going through security checks and/or testing (38%), lack of security process consistency across different development teams (38%), security team can’t keep pace with release cadences (34%) and developers’ reluctance to work with security (29%).

    API security is also becoming especially challenging, with 92% of respondents dealing with at least one API security incident in the last 12 months. Well over half (57%) have dealt with multiple API security issues in the last 12 months.

    On the plus side, the survey also found that more than half (54%) of teams responsible for securing APIs are involved with development as soon as or before the APIs are published. A full 97% said developers either have a good (22%) or high level (71%) of API security knowledge. A total of 89% reported their organization provides formal API security training to their development teams. More than three-quarters (78%) can remediate an API vulnerability within a day, and 39% reported they can do so within hours. A total of 60% reported a dedicated budget has been specifically allocated to API security, with another 30% identifying it as a discrete item within their cybersecurity budget. Two-thirds (66%) said spending on API security would significantly increase in the next 12 to 18 months.

    The use of APIs is only going to increase in the months and years ahead. Well over one-third (36%) of respondents are using APIs across all their applications, with 50% expecting this to be the case in two years. Another 44% reported most of their applications are using APIs., with 32% expecting this to be the case in two years. More than three-quarters (76%) of organizations reported that they have an average of 26 APIs per application and 53% have more than a quarter of their APIs facing the internet.

    More than a third (35%) are updating APIs daily while another 40% do so weekly. A full 61% are using APIs to connect microservices.

    Data Theorem COO Doug Dooley said those APIs are now being targeted by cybercriminals that routinely test the weaknesses of every API after each deployment and update. The most common types of API attacks reported by survey respondents included exposure of data (32%), account takeover (31%), denial of service (DoS) attack (31%), attack on misconfigured API (30%), fake account creation (28%), API injection attack (28%), ransomware (27%), data breach (26%) and content scraping (26%).

    Despite these issues, however, nearly three-quarters (74%) of survey respondents said they believe their organizations have a robust API security program in place, with 22% reporting they have some process in place. Organizations are using a mix of approaches including API security tools (59%), web application firewalls (WAFs) (57%) and API gateways (50%), distributed DoS mitigation (48%) and bot management solutions (42%).

    Challenges included managing multiple tools (29%), visibility into and control of APIs (28%), inventorying APIs (27%), inconsistent adoption of APIs (27%), API security testing (27%), data governance/exposure (26%), discovering and remediating misconfigured APIs (26%), keeping pace with threats (26%) and accurately tracking API usage (25%).

    Significant API security concerns included API authentication issues (41%), account takeovers (39%), identifying and tracking third-party APIs (38%), using tools not purpose-built for security (38%), shadow/undiscovered APIs (38%), abusive behavior bypassing app security tools (38%), denial of service attacks (38%), data governance and/or data exposure issues as a result of insecure APIs (37%), abusive behavior bypassing bot management (37%) and outdated/unneeded APIs, also known as zombie APIs. (33%).

    The type of vulnerabilities that are of the greatest concern are sensitive data exposure (34%), access control vulnerabilities (31%), API business logic flaws (31%) and DDoS attacks (30%).

    More than 79% of respondents, however, are relying on manual testing and review processes to ensure APIs don’t expose sensitive data (79%), with 63% having automated alerting capability for when sensitive data is exposed.

    Overall, it’s apparent that more resources as being applied to securing software supply chains in general and APIs specifically. However, given the amount of software being built and deployed, it may be a while before organizations see the return on those investments.

  • GitHub Adds Accessibility Improvements for Developers With Disabilities

    GitHub Adds Accessibility Improvements for Developers With Disabilities

    GitHub today revealed it added several capabilities to make it easier for developers with disabilities to access its code repository platform. The new capabilities were announced to coincide with Global Accessibility Awareness Day.

    Capabilities highlighted include a navigation tool and a search tool that can be used with a screen reader that GitHub provides.

    In addition, GitHub’s contribution graph has been updated to remove barriers for keyboard-only and screen reader users.

    Finally, GitHub has made site-wide color contrast improvements to the light and dark themes presented by Primer, the GitHub Design System.

    Ed Summers, head of accessibility for GitHub, said there are more than 1.3 billion individuals with a wide variety of disabilities that could, with access to the right tools, write code. That community today, however, is underrepresented as a percentage of the larger IT community, he noted.

    However, as tools such as GitHub Copilot—a generative artificial intelligence (AI) tool that helps developers write better, more secure code—is paired with GitHub Copilot Voice—a speech interface for GitHub Copilot that is available in technical preview—the GitHub platform becomes more accessible to developers with disabilities, he said.

    Generative AI coupled with a speech interface that eliminates the need for a keyboard is lowering the barrier for entry for every potential developer. But for individuals with disabilities, it represents a tremendous advance, noted Summers. Anyone who understands how code should be constructed can build an application, he added.

    It’s not clear what impact generative AI will have on the global shortage of developer talent that has persisted for decades but it’s clear the pace at which applications can be developed is about to dramatically increase. In addition, many organizations have embraced working from home in the wake of the COVID-19 pandemic. That shift makes it easier for many individuals with disabilities to become part of the workforce because they no longer need to commute to an office.

    Generative AI platforms are fundamentally changing the way humans interact with machines. Instead of requiring a developer to create a level of abstraction to communicate with a machine, it’s now possible for machines to understand the language humans use to communicate with each other. Via a natural language interface, developers will soon be asking generative AI platforms to not only surface suggestions but also debug applications and write the code used to create an application.

    At this point, like it or not, the generative AI genie is out of the proverbial bottle. Just about every job function imaginable will be impacted to varying degrees. In the case of DevOps teams, the ultimate impact should involve less drudgery as many of the manual tasks that conspire to make managing DevOps workflows tedious are eliminated.

    It may be a while before the impact of generative AI is fully appreciated, and there will certainly be more advances to come that will be applicable to individuals with disabilities. In the meantime, however, many members of a DevOps team may not even realize a member of the team has a disability as the writing of code continues to be increasingly automated.

  • Boomi Brings Generative AI to Integration Platform

    Boomi Brings Generative AI to Integration Platform

    Boomi announced today it is applying generative artificial intelligence (AI) to enable organizations to automate integrations via a natural language interface.

    Boomi CEO Steve Lucas said Boomi AI makes it much simpler for professional and citizen developers alike to create integrations at scale using generative AI platforms to invoke the low-code engine that Boomi already provides to create integrations between thousands of applications.

    Those generative AI capabilities will complement a wide range of AI capabilities based on machine learning algorithms that are already embedded in the Boomi platform, he added.

    Generative AI’s Impact on Integration and Automation – Steve Lucas, Boomi from Techstrong.TV on Vimeo.

    Boomi has been widely used to drive business process integration via an integration platform-as-a-service (iPaaS) environment it delivers as a managed service. The low-code platform the company created significantly reduces the need to rely on professional developers to write procedural code using Java, for example, to create those integrations.

    Generative AI capabilities make it possible for almost anyone of any skill level to create those integrations across multiple types of application programming interfaces (APIs), noted Lucas.

    That capability also makes it possible for Boomi to notify end users when, for example, a data attribute in an application has changed and ask them if they would like to apply that change across their entire application environment.

    In addition, Boomi AI can also interpret the intent of a prompt to better orchestrate processes in addition to providing an audit history in natural language that explains how decisions were made and what outcomes occurred as a result. As part of that effort, Boomi algorithms are also trained to avoid biases and unfairness with established ethical guidelines.

    Boomi’s AI engine is trained using anonymized data collected from approximately 20,000 customers that have created more than 200 million integrations using the Boomi platform. Those integrations have been exposed to third-party large language models (LLMs) that the company is using to train Boomi AI, said Lucas. Boomi is not storing any customer data to train its AI models.
    DevOps teams, of course, spend hours creating integrations that should be much simpler to create, test and validate with the rise of generative AI. In fact, the number of integrations being created to drive a wide range of digital business transformation initiatives should substantially increase.

    In the meantime, DevOps teams would be well-advised to start evaluating which tasks they manually perform today that could instead be automated by a generative AI platform. Many of the routine integrations that DevOps teams create today using low-level APIs will be increasingly automated by end users. The ultimate goal should be to identify more complex tasks that previously DevOps teams would not have attempted simply because there wasn’t enough time.

    It’s still too early to determine what impact generative AI will have on the way IT is managed, but DevOps teams that are typically committed to automating as many processes as possible should be at the forefront of adoption. The challenge is going to be simply keeping track of the rate of change that might soon be enabled by these platforms.

  • SapientAI Emerges From Stealth to Apply AI to App Testing

    SapientAI Emerges From Stealth to Apply AI to App Testing

    SapientAI emerged from stealth today to automate the writing of test code using a combination of generative artificial intelligence (AI), machine learning algorithms and data science.

    Fresh from raising $5 million in seed funding, SapientAI CEO Rishi Singh said the goal is to leverage AI to push application testing further left using an AI Test Coder the company built.

    The concept of AI Test Coder is similar to the co-pilot capabilities that GitHub and Google are providing to help developers write better code faster, but also goes beyond generative AI to add deeper analysis and contextual understanding to the tests it automatically generates, noted Singh.

    In the longer term, the ability to use AI to create tests will result in the deployment of more secure, higher-quality applications because there will be no need to short-shrift application testing processes in the interest of meeting deadlines, added Singh.

    Prior to the rise of AI, building application tests has historically been prone to error. AI Test Coder is designed to be a plug-in for an integrated development environment that will leverage the large language models (LLMs) created by OpenAI and cloud service providers to employ natural language to create tests based on an analysis of the application environment created using a combination of machine learning algorithms and data science techniques, said Singh.

    As part of the effort, AI Test Coder instantly inspects all software code to understand the software testing environment, including any incorrect assumptions that stem from errors and bugs, before generating any test code. It then leverages code intelligence to understand existing code and automatically identify and detect patterns to anticipate any new software issue that may arise. AI Test Coder also locates where code changes will have an effect before any test code is created. This eliminates time that would otherwise be wasted on manually debugging code.

    Once those steps are completed, Test Coder then generates more accurate tests without relying on the current trial-and-error approach to testing that can slow application development, said Singh.

    In effect, AI Test Coder provides each developer with their own assistant dedicated to managing testing tasks that ultimately improve quality assurance, including removing many of the vulnerabilities that create application security issues that have become the bane of software supply chains.

    It’s too early to determine precisely what impact AI will have on application development and deployment, but many of the manual tasks that create a sense of drudgery are about to be eliminated. That means the overall pace at which applications can be built and deployed is about to be greatly accelerated. DevOps teams are likely to find themselves pushing more code simultaneously through pipelines than ever.

    Naturally, AI will soon be applied to not just writing code and tests faster, but also DevOps workflows. The issue now is determining which of the tasks that today require manual intervention will be eliminated in a way that enables DevOps teams to focus more of their time and effort on more complex challenges.

  • Google Previews Generative AI Service to Improve Code

    Google Previews Generative AI Service to Improve Code

    Google is previewing an instance of a generative artificial intelligence (AI) service that surfaces code recommendations to make it possible to build high-quality applications faster.

    Specifically, Duet AI for Google Cloud surfaces code recommendations in real-time, generates full functions and code blocks and suggests fixes for vulnerabilities and errors in the code that it discovers. It also promises to enable DevOps teams to create custom large language models (LLM) based on their own code. In fact, GitLab and Tabnine are, respectively, already integrating LLMs running on Google Cloud with their continuous integration/continuous delivery (CI/CD) platform and testing tools.

    Chen Goldberg, vice president of engineering for Google, told attendees at the Google I/O 2023 conference that Duet AI for Google Cloud will fundamentally change how developers will be able to build applications, regardless of their skill level.

    Duet AI for Google Cloud was trained using data collected from Google Cloud using sample code and documentation to fine-tune behaviors and patterns.

    Duet AI for Google Cloud is available under an AI Trusted Tester Program and provides developers with many of the same capabilities that can be found in the GitHub Copilot tool that is based on the LLMs created by OpenAI.

    In addition to making Duet AI for Google Cloud accessible via both its tools and managed services, Google said it also plans to integrate the service with multiple integrated development environments (IDEs).

    It’s still early days as far as the use of generative AI within DevOps workflows is concerned, but it’s clear that the amount of code being created by developers is about to increase exponentially. Hopefully, AI technologies will also one day help software engineers find ways to manage the massive volume of code moving across their DevOps pipelines.

    Generative AI platforms will fundamentally change the way humans interact with machines. Instead of requiring a developer to create a level of abstraction to communicate with a machine, it’s now possible for machines to understand the language humans use to communicate with each other. Using a natural language interface, developers will soon be asking generative AI platforms to not only make suggestions but also debug applications and, eventually, write the code used to create an application.

    At this point, like it or not, the generative AI genie is out of the proverbial bottle. Just about every job function imaginable will be impacted to varying degrees. In the case of DevOps teams, the ultimate impact should involve less drudgery and toil as many of the manual tasks that conspire to make managing DevOps workflows tedious are eliminated.

    In the meantime, DevOps teams should spend some time identifying which tasks will become automated by generative AI and other technologies that employ machine and deep learning algorithms. It doesn’t make much sense to onboard new members to a DevOps team only to discover that the tasks assigned to that individual are about to be automated. Hopefully, when all these AI technologies are finally implemented, the entire DevOps experience will be better than ever.

  • Spotify Adds More Plugins for Backstage Developer Portal

    Spotify Adds More Plugins for Backstage Developer Portal

    Spotify this week added additional plugins for its open source Backstage platform that is used to build developer portals. The new plugins make it simpler to address role-based access and access Insights, a tool from Spotify that tracks Backstage usage trends.

    In addition, Spotify is also enhancing a Soundcheck plugin for Backstage that is used to visualize and track development of software components. Forthcoming capabilities include a no-code interface that makes it possible to programmatically create checks of code without writing any code.

    DevOps teams will also be able to view, export and understand how teams and components are doing compared to established best practices and monitor trends, graphs and historical views, and receive notifications when levels change.

    Finally, Spotify is committing to adding additional integrations with third-party tools and platforms such as Snyk, Sonarqube and the open source Argo continuous delivery (CD) platform.

    There are now five plugins available via a Spotify Plugins for Backstage subscription service, and Spotify said more are planned.

    Meg Watson, a group product manager at Spotify, said Backstage has been gaining traction in cloud-native application environments that are especially challenging to develop using multiple tools. Because of Backstage, developers are not only more productive but turnover has also been reduced because developers are provided with a better experience, she noted.

    There’s a lot more focus than ever on developer productivity, which Backstage addresses by creating a catalog of blueprints that developers can readily consume rather than requiring them to build these capabilities themselves multiple times over. The goal is to create scaffolds that developers can consistently reuse across multiple application development projects.

    Backstage was donated to the Cloud Native Computing Foundation (CNCF) and is now being advanced by contributions from multiple vendors. Much of that focus is on lowering the barrier of adoption for the platform, said Watson. Specifically, Spotify is working on a QuickStart for Backstage edition of the platform that is simpler to install, she said.

    In general, Spotify is trying to strike a balance between centralizing the management of DevOps workflows and the need to enable developers to define workflows that are natural to them versus ones that have been imposed on them, added Watson.

    It’s not clear to what degree Backstage will help fuel the adoption of a shift toward platform engineering to centralize the management of DevOps tools and platforms. The concept of a portal through which developers can self-service their own needs may not be new, but as an open source project, Backstage has made it simpler for many organizations to achieve that goal.

    Just about every DevOps team is now being asked to find ways to help improve developer productivity at a time when many organizations are trying to do more with fewer resources. Regardless of the motivation, however, it’s apparent that DevOps workflows are continuing to evolve and mature.

    In the meantime, DevOps teams would be well-advised to eliminate as many bottlenecks as possible before they negatively impact developer productivity.

  • CDF Report Reveals Lack of DevOps Maturity

    CDF Report Reveals Lack of DevOps Maturity

    A global survey of nearly 26,000 DevOps professionals published today by the Continuous Delivery Foundation (CDF) at the cdCon + GitOpsCon event found 84% of respondents are involved in some type of DevOps activity. However, the report also found just under half 49% are using a continuous integration (CI) or continuous delivery (CD) platform and only 22% are using CI and CD approaches to automate all building, testing and deployment of code in a production environment.

    The report, conducted in collaboration with SlashData, found the number of developers using CI or CD platforms only increased two percentage points year-over-year, with the use of self-hosted CI/CD tools dropping from 32% in the first quarter of 2022 to 23% in the same period this year.

    The survey also noted that only 7% of respondents this year described themselves as a DevOps engineer or specialist.

    Overall, medium-sized businesses have the highest involvement in DevOps practices (94%), followed by small businesses and large enterprises at just over 90%. More than three-quarters of developers (79%) are now using DevOps practices.

    The most common DevOps activity is monitoring software or infrastructure performance (47%), followed by application security testing and using CI to build and test code, tied at 37% each, and approving code deployments into production environments at 36%, the survey found.

    The average number of DevOps-related technologies used by developers (4.5) has, meanwhile, remained stable over the last year. The most widely used tools address issue tracking (39%), source code management (38%), test automation management (34%), collaboration and knowledge-sharing tools (32%) and application performance management/observability and agile project management tools tied at 31%.

    The survey also noted that the percentage of top performers—defined as those with lead times of less than one day—now stands at 15%. Survey results for top performers over the years continue to fluctuate between 13% and 17% of the survey total. On the plus side, 40% of developers have lead time changes of less than one week, the second-highest ranking for this survey.

    More than a third of all DevOps practitioners (34%), however, take longer than a week to restore service. Only 12% of DevOps practitioners take less than an hour to restore service.

    In general, the survey finds that organizations that use 10 or more DevOps tools are twice as likely to be high performers.

    Fatih Degirmenci, executive director for the CDF, told conference attendees the consortium, which now oversees nine projects, including the open source Jenkins CI/CD platform, continues to make progress as additional projects are added.

    However, while it’s clear that many organizations have adopted multiple DevOps tools and platforms, far too many of them still have a long DevOps journey ahead. Each organization will need to decide whether DevOps workflows need to be adopted to fit their needs, even as the pressure to build and deliver software faster continues to mount. In fact, at a time when more organizations than ever are dependent on software to drive mission-critical digital business transformation initiatives, time may be running out to adopt DevOps workflows at a deeper level if organizations hope to stay competitive.

  • Black Box SLIs

    Black Box SLIs

    This article is a preview of a talk by Stephan Lips for SLOconf 2023, on May 15 – 18. To watch this talk and many more like it, register for free at sloconf.com.

    SLOs are fast becoming the industry standard to measure reliability and help teams decide when to prioritize it. The first step in adopting a service level objective (SLO) culture is to identify the metrics that matter without drowning in noise and alert fatigue. This article explores how to apply the black box concept to aggregate granular metrics into service level indicators (SLIs) that focus on the user experience as an indicator of system reliability.

    To SLI or Not to SLI

    In general terms, SLOs define targets for the proper level of reliability of a given product, such as a service or a website. SLOs are applied to or informed by SLIs. An SLI is a measurement determined over a metric, or a piece of data, representing some property of a service. And this is where we, as engineers, can get lost in the details, since the perpetual proximity to the systems we build and support often leads us to think of system reliability in technical terms or metrics (e.g., response time, error rate, throughput). While these are certainly valuable metrics, the user experience may be compromised even if the error rate is zero and the duration is well within SLOs. Consider, for example, the response data. Even if well-formed, it may not be current, or flat-out wrong. An error-free and quick response is of no value to a user that expects current and correct data. Error rate and response time remain valuable metrics and SLIs, but focusing exclusively on them would leave higher-level issues undetected.

    We could add freshness and correctness SLIs, but by doing so, we increase the number of signals we monitor. And with each signal—or SLI and associated SLO and error budget—we increase alert frequency and make reliability reports unnecessarily complex. In other words, adding SLIs may address a particular aspect of system reliability, but it also introduces additional complexities.

    Tales of Black and White Boxes

    So, let’s take a step back and borrow a concept from a related discipline: Quality engineering—in particular, software testing. Tests commonly fall into one of two categories: Black box tests or white box tests.

    In systems theory, the black box is an abstraction representing a class of concrete open systems that can be viewed solely in terms of its stimuli inputs and output reactions, without any knowledge of its internal workings. A given input is expected to result in a particular output, without any consideration for the processing steps. Common examples include end-to-end tests.

    White box tests, on the contrary, are designed with knowledge of, and to test, internal structures and workings of an application. Common examples include unit and integration tests.

    User Journey as Black Box

    Now that we understand the concept of black box versus white box tests, let’s apply it to our SLIs. As mentioned above, a good SLI considers the entire user journey. Conceptually, a user journey aligns with the black box paradigm: For a given input, a particular output is expected. For example, requests to our API (the “input”) result in responses that provide fresh data to clients within a given time frame (the “output,” including success criteria). There are several aspects worth mentioning with this SLI:

    ● The SLI is applied at a system level
    ● The SLI aggregates lower-level metrics implicitly and explicitly
    ● The SLI is binary; it is either true or false.

    These aspects combine to inform an SLI that represents the user experience (system level), via measuring many indicators by measuring only a few and supporting pass/fail attribution to an SLO target (by being binary). In other words, the user journey is measured as a black box SLI.

    White Box to Black Box: An Example

    Let’s consider a concrete example. A user requests a new account for a website. After the request is processed successfully, the user receives a confirmation email with an activation link. The user follows the link to activate the new account and log in. This workflow is visualized in the following sequence diagram.

    Of particular interest are the account creation and user notification via email steps. Both steps occur asynchronously. In particular, the event processing engine where the request for account creation is queued offers several opportunities for insightful SLIs: Queue length, average processing time, etc. Those SLIs, however, fall into the white box category: They contribute to the user experience, yet are opaque to the user (black box). The user journey begins with the initial request for account creation (input) and ends with the email containing the activation link (output). Rephrasing the example from earlier—a user-focused (black box) SLI could be a request for a new account (the ‘input’) that results in sending an email with a valid activation link within 1 minute (the ‘output’, incl. success criteria). This single high-level SLI aggregates several lower-level metrics; it measures many things by measuring only a few.

    Let’s switch to the engineering mindset mentioned in the introduction and assume the processing queue is stuck. The high-level black box SLI does not capture queue-specific metrics, suggesting a more granular SLI specific to queue size may be needed. However, white box metrics like this will affect the error budget burn of the aggregate SLO associated with the high-level black box SLI. Monitoring and observability tools will allow engineers to diagnose and troubleshoot particular issues, such as a stuck queue, while understanding the impact on system reliability (via the higher level, black box SLO’s error budget and burn rate). The solution to the stuck processing queue used in this example is not an SLI dedicated to the queue, but reliability-focused work to diagnose and correct the root cause of the queue getting stuck.

    Summary

    This article introduces an SLI thought model that uses a common paradigm from quality engineering. This thought model offers a different way to think about SLIs. It supports implementing the fundamental objective of SLIs and the associated reliability stack they inform: Ensure a positive and reliable user experience by measuring reliability and providing quantitative support for decisions on prioritizing development efforts. Only a happy user is a continuous user.