devopsy.kinsta.cloud

Author: Dan Belcher

  • Three Trends That Will Transform DevOps in 2023

    Three Trends That Will Transform DevOps in 2023

    The onset of COVID-19 accelerated digital transformation-related initiatives like cloud migration, microservices, observability and CI/CD, giving long-delayed projects the budget and staffing they needed to build momentum. After almost three years of increased investment and accelerating progress, CTOs and software development teams are navigating how to enable the improvements in time-to-market and customer experience that they set out to achieve. In 2023, we’ll see enterprise DevOps mature and expand its business impact by placing a central focus on quality, delivering improved time-to-market, customer satisfaction and productivity.

    DevOps Matures as a DX Enabler

    The past three years have seen massive growth in cloud computing, observability and pipeline automation as enterprises raced to deliver new, high-quality digital experiences to consumers. Cloud spending alone increased from $270 billion in 2020 to 397.5 billion in 2022, according to Gartner. With new technologies in place, the focus shifts to processes that deliver business agility and efficiency.

    Under increased pressure to deliver faster, businesses need to revisit, evolve—and, in some cases, undo—decades of risk and change management practices built around brick-and-mortar experiences and long software release cycles.

    As digital engagement increases, so will expectations for software quality and scalability. We’ll need to invest in new tools and practices—from testing to observability and beyond—to ensure that we’re meeting or exceeding those expectations.

    The economic uncertainty will compel us to look for ways to improve productivity and decrease waste. Quality practices are central to both. We’ll continue to look for ways to shift left and catch defects early in our pipeline when they are the least expensive to fix. We’ll want to optimize our build, integration, testing and deployment processes to identify bottlenecks and causes of rework that drain the productivity of so many teams daily.

    DevOps, with its emphasis on automation and collaboration, will connect the dots between operational goals, technological investments and improved software. Established frameworks like DORA give software development organizations concrete ways to measure the efficiency and effectiveness of delivery pipelines. The key metrics outlined by DORA—deployment frequency, lead time for changes, change failure rate and time to restore service—enable teams to align on where they can improve development practices and quality practices.

    DevOps Will be Measured by Business Success

    The value of digital transformation and DevOps will also extend even further across the enterprise as CTOs evaluate DevOps practices with structured assessments such as DORA. With several years of accelerated digital transformation under our belts, the software development industry can build out a broader definition of DevOps performance over time. The DORA framework sets the stage for understanding how engineering practices impact the customer experience and, in turn, company growth. In one example, Liberty Mutual, a company prides itself on exemplary customer service, is already seeing how DevOps performance connects to business success.

    Architect Justin Robinson notes in their roundtable on data-driven decision-making and DORA metrics: “For organizations, this is improved organizational performance. If we want to keep investing in great tech and serving our customers as best as possible, [we need] behaviors and practices that we can adopt to continuously get better and hit on every front here. Happier engineers, a better culture and a better bottom line for our company.”

    In 2023, more enterprises will harness DORA to measure DevOps performance and its impact on company growth.

    Customers Will Reap DevOps Benefits

    We’re already seeing how metrics like deployment frequency can support an organization’s ability to attract and retain users. According to the 2022 Testing in DevOps Report, teams that increased deployments by 50%-100% were two-and-a-half times more likely to report high user satisfaction. The ability to innovate—without introducing defects to users—is the next frontier in DevOps success. Quality engineering will serve as a safeguard for DevOps teams so they can deliver new features faster and with confidence.

    The Testing in DevOps Report also illustrated the growing connection between digital transformation, DevOps and quality engineering. A full 88% of teams with highly automated pipelines and mature DevOps practices reported high test coverage, which in turn is connected to high customer satisfaction. Test coverage, a core metric for quality assurance teams, is becoming the connection between code and customers throughout development. Quality teams will expand their testing strategies to cover more aspects of the customer experience, including non-functional attributes like performance and accessibility. Automation, particularly automated testing embedded in CI/CD pipelines, will allow teams to adopt these additional quality factors without slowing deployments.

    Connecting Engineering Transformation to Business Success

    2023 is shaping up to be an unpredictable year in technology. But with the momentum propelling digital transformation to an enterprise-level priority, engineering leaders have the data and the tools to shift focus from DevOps processes to transformative outcomes. DevOps maturity, measured and improved with DORA, will help organizations extract more value from their digital transformations by supporting greater innovation and better customer experiences. Quality engineering, augmented with new testing strategies that measure more of the customer experience, will play a critical role in this evolution by enabling development teams to move faster with confidence.

  • Low-Code Automated Software Testing Drives DevOps

    Low-Code Automated Software Testing Drives DevOps

    The DevOps world has quietly turned its focus to solving an essential challenge: enabling software teams to innovate quickly while meeting high customer expectations for quality. In other words, to build useful things faster with fewer mistakes. This tension has been growing for years, but 2020 added the challenges of remote work and accelerated digital transformation to the continued disruption brought about by cloud, SaaS and DevOps. In 2021, we need a road map for taking test automation to the next level with quality engineering, a discipline that incorporates best practices and data-driven testing into the end-to-end customer experience and drives organizational growth.

    Rethinking the Quality Process for the DevOps Era

    Quality engineering (QE) has traditionally lacked the tools needed to keep pace with digital transformation. Many QE teams struggle with complex, brittle automation frameworks and slow manual testing which create serious bottlenecks to DevOps adoption. While development accelerates, QE is left with disparate or outdated workflows and toolsets.

    Rising to the challenges and opportunities presented by DevOps and digital transformation will require innovation in our approach to quality. More than ever, we need to shift our focus away from quality assurance as a task that you perform late in the software development life cycle (SDLC) in favor of quality engineering practices throughout the entire life cycle. It starts with broad, effective test coverage – beginning when your developer is working on a local branch and continuing until a high-quality customer experience is confirmed in production across the entire surface area of the application – web, API, mobile, email, PDF or otherwise. It also goes deep; you evaluate all aspects of quality, ensuring that you satisfy requirements related to functional correctness, accessibility, performance, user experience and more.

    Low-Code Test Automation Unlocks a Culture of Quality

    With low-code, genuinely effective testing is at the fingertips of every role, from manual testers to developers, automation engineers, product owners and customer support. Requiring all of these roles to develop specialized programming skills is impractical, so low-code tools serve to democratize quality engineering by enabling everyone to participate in authoring tests, updating tests, and gaining insight from test results.

    Low-code testing brings several advantages to quality engineering, particularly in the transition from manual testing to automated testing. While traditionally this process would create major disruptions, low-code platforms are smoothing the way. In their IT Automation predictions for 2021, Gartner expects that by 2024, improvements in automation capabilities will refocus 30% of IT operations efforts from support to continuous engineering. For manual testing teams, that shift would likely trigger talent changes and a long process of pilot projects, retraining, and gradual rollouts. Low-code is key to managing this shift, upskilling workers and ensuring quality throughout.

    Modern Automation Solutions Embrace Rapid Change

    A key challenge with traditional test automation tools is that they were designed for applications that didn’t change frequently and were intended only to validate code. They relied on static selectors to identify page elements. If the selector (say, an ID) changed or objects (the XPath) moved on your webpage, then your tests failed. As more enterprises accelerated development via DevOps, Agile and CI/CD, the burden of test maintenance resulted in the same bottlenecks as manual testing.

    2020 was a breakout year for low-code test automation. New, easy-to-use solutions surpassed the capabilities of legacy frameworks that have dominated the sector. These solutions incorporate machine learning, data analytics and other advanced technologies to greatly reduce the burden of test maintenance and overcome the test rigidity challenges of legacy frameworks.

    With AI and machine learning, test automation solutions are trained by a tester simply navigating their product. As the tester moves through common customer journeys, the solution builds an understanding of each element and its context so that if an element moves or a property changes, tests containing that element are updated without human intervention. This drastically reduces the amount of maintenance required to ensure your tests are current. When tests do need to be updated, low-code models make it easy for anyone to jump in and make the updates. Looking ahead, auto-healing and low-code editing are crucial to supporting quality engineering in high-velocity teams.

    Testing the Entire Customer Experience

    Traditional pass/fail validation stops short of offering real insight into the user journey, a critical step effective quality engineering. The next generation of test automation platforms expands focus from validating code to validating the entire user experience. This starts with broadening testing to cover non-functional requirements, such as application performance, accessibility, and user experience. For example, identifying trends in load time for key pages and monitoring visual changes are done autonomously and essential to managing the customer experience. From there, you can prioritize quality efforts toward the components of your application that are most heavily used or highest risk. This enables you to make data-driven decisions about test coverage — focusing your testing efforts on high-priority pages like the checkout page for an e-commerce site, or high-risk pages such as those with multiple interactive elements that are linked from other pages, such as signup forms.

    But even a fully data-driven testing strategy that only focuses on the app or website UI is unlikely to cover the full end-to-end user journey, resulting in test coverage gaps and potential pain points for customers. For example, a common user journey may include opening an email, clicking on a link, checking a web site, and downloading a PDF. Whereas the limitations of legacy tools relegates testing to the web portion of that journey, modern solutions allow you to validate the entire user or customer experience (CX): receiving and opening an email, clicking on the link, logging into the web site, downloading the PDF and verifying its content. It’s even possible to confirm values in the backend via API – all without writing a line of code. Developing the tools to seamlessly monitor, repair, and manage the expanded CX is central to our innovation and to our customers’ success.

    Quality Engineering as User-Centric as You

    Quality is no longer an afterthought or just a box to check before deployment; it’s an essential part of the user experience, and therefore an essential part of maintaining a competitive advantage in every industry that uses software to serve customers. Test automation is critical for high-velocity teams racing to innovate faster, shift to DevOps, and build quality products while working remotely. Thankfully, a growing number of quality assurance teams are getting a seat at the table as DevOps is implemented, elevating the profile of quality as a strategy and growing demand for test automation solutions that can keep pace with the demands of DevOps.

    In response to this demand, a new generation of intelligent test automation solutions is enabling teams to rise to the challenge by embracing low-code and machine learning to maintain quality. To build a culture of quality engineering in this environment, your team needs test automation solutions that are as user-centric as you are: solutions that provide coverage across the entire user journey. Solutions that not only verify functional correctness, but provide rich insight into all aspects of quality and enable continuous improvement. Perhaps most importantly, you need solutions that enable everyone on the team to participate in quality. This ensures that the entire SDLC actively participates in building user-friendly products that continue to set the standard for quality at the speed of DevOps.

  • The Era of Intelligent Testing

    The Era of Intelligent Testing

    The state of the art in software testing has not kept pace with advances in software development over the past 10 years. Modern quality assurance (QA) groups struggle to play their role in a world of continuous delivery, leading to more defects in production, low QA morale and excessive testing overhead. Thankfully, innovations in machine intelligence and test automation have laid the groundwork for a better model—one that will allow QA to keep pace with development, increase productivity, improve product quality, and raise customer satisfaction across the software industry.

    Why We Need a New QA Toolset

    Existing QA solutions were built for a world where software changed infrequently and functionality was highly specified and documented.  This is no longer the case.

    Products Are Evolving Faster Than Ever

    Several forces are converging to accelerate development. Software as a service (SaaS) has removed many of the issues around operating system compatibility, packaging, deployment and versioning that consumed significant development time during the installed software era. Consider that in 2016 Amazon Web Services delivered more than 1,000 new features and services—a pace that would be impossible with packaged software.

    “Development cycles are getting shorter and new features are coming faster than ever…There simply isn’t enough time to test.”

                                                 – QA Leader at Fortune 250 company

    The proliferation of open source and cloud-based features as a service has also reduced the time that developers spend building generic functionality. At mabl, for example, the thought of building our own authentication engine never crossed our minds; we use Auth0. Instead of building our own data processing pipeline, we use Apache Beam and Google Cloud Dataflow. Rather than building our own ML framework, we use Tensorflow. And the list continues; reusable components and cloud-based services save us years of engineering effort, allowing our developers to deliver product features rapidly.

    Continued adoption of agile methodologies has further increased the efficiency of development teams. A recent McKinsey study of 1500 software projects demonstrated that agile teams are 27 percent more productive than those that follow methodologies such as waterfall.

    Finally, continuous integration and continuous delivery are accelerating the pace of change in software. A recent survey by Atlassian suggests that more than 50 percent of software teams are already using continuous integration and/or delivery, and more than 32 percent have plans to move to CI/CD. The same survey indicates that more than 50 percent of organizations are able to push changes to production daily.

    In the end, improved developer productivity means that software products are evolving faster than ever. Unfortunately, this places increased pressure on the QA function. As one QA leader from a Fortune 250 company said in a recent interview with mabl, “Development cycles are getting shorter and new features are coming faster than ever. And we have five productive developers for every QA engineer. There simply isn’t enough time to test.”

    Existing Automation Frameworks Impose Too Much Overhead

    QA rightfully relies on automation frameworks to improve productivity; tools such as Selenium, Appium and JUnit now enjoy widespread adoption. More than 80 percent of the respondents to a recent mabl survey use Selenium today, and QA managers are under constant pressure to increase automation. Unfortunately, while Selenium and other frameworks have helped many companies increase their QA velocity as compared to manual testing, these frameworks have significant flaws:

    • They all require specialized scripting expertise, and given the lack of available talent, teams have very limited capacity for QA automation.
    • They provide limited context in test results, which leaves teams with very little information to help them triage and address issues.
    • At scale, they require their own infrastructure, which takes time to provision, operate and scale.
    • They are tightly coupled to attributes of the product under test that change frequently (xPath, CSS, etc.), resulting in constant maintenance and false positives.

    These flaws only become more painful as the pace of development and change accelerates, compelling many teams to limit their use or discard them altogether, resulting in lower product quality.

    mabl survey – satisfaction with existing testing tools and processes (n = 104)

    “Please indicate the extent to which you agree with the following statements”

    Time Pressures Lead to Unhealthy Team Dynamics

    People in testing roles are increasingly at odds with their peers in development. Unable to keep up with the continuous pace of development, QA becomes a bottleneck to product velocity, leading to tension with their peers in development. This constant pressure also starves QA of its ability to make the very investments in automation and process improvements that are needed to improve throughput. The feeling of always being behind with little hope for improvement often leads to low morale within the team.

    Gaining consensus on the standard of quality has also become difficult for teams, since agile teams often steer clear of detailed functional specifications for each release, preferring instead to focus on user goals and requirements (see: Agile Manifesto). Because user goals and requirements are open to more interpretation, QA and development can disagree on not only the severity of defects but also the validity of the tests themselves.

    Modern software teams are also more open to evolving requirements during a given release cycle (see: “Responding to change” in the Agile Manifesto). Evolving requirements should drive updated test cases, but this requires deep engagement and coordination for already stretched personnel. Rather than testing for quality versus specified behavior, we rely much more on ad hoc and manually intensive exploratory testing to look for defects.

    There is Hope for QA

    While faster feature development, shorter release cycles, evolving requirements and unclear accountability for end-to-end quality all present real challenges for QA, a new generation of QA tools are emerging to confront these challenges head on.

    It Will Be Easy to Create and Maintain Tests

    Unlike existing test automation frameworks, next-generation QA tools will not require specialized scripting expertise. Instead, they will offer intuitive interfaces that allow anyone to quickly create and manage tests. This will help teams expand their test coverage and evolve their tests quickly as their products evolve.

    Tests Will Adapt Seamlessly to Change

    Because they use much more sophisticated, durable methods of simulating user behavior in automated tests, next-generation QA tools will not suffer from the brittleness associated with existing automation frameworks. Tests will not be tightly bound to individual elements within the front-end codebase that change frequently. Rather, they will use machine intelligence to create and maintain sophisticated models of tests, enabling tests to adapt the tests rather than fail when xPaths and other locators change.

    QA Will Run in the Cloud

    Today, teams struggle with the performance, cost and operational overhead of on-premises testing systems. Next-generation QA tools will take advantage of on-demand cloud computing resources to execute tests faster and more efficiently. They will tap into powerful data processing and analytics services to analyze test results and deliver insights. They will be delivered completely as a service, offloading the management and operational burden to tool vendors.

    QA Will Be Part of the Delivery Pipeline

    Next-generation QA tools will be tightly integrated into automated CI/CD pipelines orchestrated by Jenkins, Spinnaker, CircleCI and CodeShip. Tests will trigger automatically whenever teams make changes to the product under test and on-demand. The tools will notify team members when there are potential issues so that they can address them before they affect users. It will be simple to run tests and compare results across builds and environments. Decisions around releases, deployments, promotions and rollbacks will be largely automated.

    The Definition of Quality Will Evolve

    Beyond validating specific assertions in tests, the next generation of QA tools will use machine intelligence to automatically detect and highlight potential regressions in applications. As a result, we’ll think less about tests “Passing” and “Failing” and more about the extent to which changes improve or degrade the user experience. We will focus less on code coverage and more on product coverage. We’ll look beyond basic counts of passing and failing tests in favor of a more on holistic approach to assess the risk associated with a given build or deployment.

    Tools Will Be Deeply Embedded Within Our Teams

    Next-generation QA tools will effectively serve as extensions to development teams. They will share test results and insights through Slack, Jira email and other common communications channels. They will take feedback and training from the entire team—including product, development, QA and customer success—and incorporate what they learn into their tests. Most importantly, next-generation tools will describe tests and results in plain language, and they will provide useful context to help developers reproduce and fix issues, just as an effective QA teammate would.

    The Era of Intelligent Testing

    Software quality assurance has never been easy, and the accelerating pace of software development in recent years has made it even more challenging. Thankfully, a new generation of tools are emerging to address this challenge. Incorporating cutting-edge machine intelligence, delivered from the cloud and embedded into the modern development workflow, these tools will dramatically improve the effectiveness of QA, ushering in a new era of efficiency and innovation across the software industry.

     

    mabl survey – user sentiment (n = 104.  For simplicity, combined “Agree” and “Strongly Agree” into “Agree” category,  combined “Disagree”, “Strongly Disagree” into “Disagree” category.)

    “How satisfied are you with the following aspects of the testing process at your company?”

    — Dan Belcher