devopsy.kinsta.cloud

Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

  • Higher Mobile App Stability Begets Higher User Ratings

    Higher Mobile App Stability Begets Higher User Ratings

    In the competitive mobile market, application stability is imperative. Downtime leads to poor user experiences, which stunts growth and revenue. Apps that have poor stability suffer the consequences. In fact, new studies show that just a 1% lower stability ranking can lead to nearly a one-star drop in App Store ratings.

    Bugsnag recently released its 2021 Application Stability Index (ASI) report, Characteristics of Leading Mobile Apps. The study took 260 top-performing mobile apps across industries and reviewed each’s stability and end-user satisfaction. The study found that apps with solid stability tend to rank much higher in the App Store and maintain increased user engagement. The report also demonstrated that developers are pivoting to weekly releases to quickly address bugs, progressing from the bi-weekly standard for software updates.

    Below, I’ll review the specific details of the ASI and the report and consider how stability varies throughout industries. Hopefully, the results here will help other software teams benchmark their app’s performance and release cadence against top-performers within their sector.

    Stability = User Satisfaction

    When measuring uptime and availability, five nines, or 99.999% uptime, is the golden target for most teams. This report measured stability as the percentage of apps sessions that were crash-free. The median across market-leading apps was 99.80%.

    What’s interesting is that apps with more nines in their stability score achieve markedly higher App Store ratings. Apps that scored less than 99.00% had a median rating of 4.15, whereas apps with 99.99% to 100% measured uptime had a median rating of 4.70. The distribution of ratings was extremely wide for low-performing apps, compared to higher average scores for higher stability.

    What do these results say? Users expect uninterrupted sessions. Frequently occurring stability issues can damage the consumer’s opinion of an app, leading to dissatisfaction and harsh reviews. The more stable an app is, the better chance it has to drive consumer success.

    Stability by Industry

    The study also exposed stability differences between industries. B2B SaaS apps ranked relatively high, at 99.85%. This is likely because B2B software must meet service-level agreements (SLAs). Such services have fewer (yet more high-paying) customers to support, the report speculated. Mobile apps directly tied to the exchange of money, such as e-commerce (99.86%), travel and hospitality (99.90%) and financial and banking (99.85%) also ranked consistently high in stability.

    Coming in second were mobile apps in the business-to-consumer (B2C) (99.82%), consumer goods (99.80%) and health and wellness (99.78%) categories. Though these areas require decent availability to remain viable, they may not need such stringent SLAs. In such scenarios, non-stop availability isn’t as directly tied to end revenue.

    Lastly, apps with the worst stability were found in the mobile games (99.71%) and media and entertainment (99.65%) categories. Mobile gaming had the worst ratings of all industries; mean stability equated to over a full day of outage per year. Increased downtime in both categories is perhaps due to more problems with frequent media updates. These areas also tend to have a lower average revenue per user (ARPU), which may affect how teams value ongoing support.

    Deployment Frequencies

    According to the report, highly stable mobile apps tend to release at least once per week, replacing the bi-weekly norm for updates. Nearly all industries are pushing at least four releases per month, aside from consumer goods and travel and hospitality, at 2.5 and 4 respectively. E-commerce takes the cake with a mean of seven releases per month.

    Nowadays, progressive delivery is made possible due to increased emphasis on CI/CD. Therefore, rapid deployment frequencies are the norm. Releasing new versions more quickly can help appease users with new features. It’s also helpful to quickly patch bugs and decrease mean time to recovery (MTTR) for major incidents.

    Users Love Stable Apps

    Throughout the pandemic, digital reliance soared—84% of consumers said applications were a lifeline to normality, found a recent AppDynamics report. In this climate, mobile applications are critical to delivering valuable interactions.

    Yet, user expectations have evolved. Now, it’s readily apparent that stability directly impacts consumer confidence. The ASI findings behoove engineers to edge closer to the five-nines goal. The more nines you have, the better off you are.

    “The results of our second ASI report highlight the need for better proactive error monitoring and stability management to help engineering teams build healthier apps that deliver superior customer experiences,” said James Smith, SVP of the Bugsnag Product Group at SmartBear.

    It’s good to reiterate that this study examined a handful of leading apps from each industry. So, these scores shouldn’t be construed as inclusive of the entire app marketplace, only as benchmarks for the top mobile app performers.

  • Puppet Survey Suggests DevOps Adoption is Stagnant

    Puppet Survey Suggests DevOps Adoption is Stagnant

    Puppet today published an annual State of DevOps report that suggests overall adoption of DevOps practices has stagnated.

    The survey of 2,650 IT, development and information security professionals finds 83% are implementing DevOps practices, to a degree. However, only 18% of respondents are working for organizations that have highly evolved, automated DevOps processes in place. More than three-quarters of respondents (78%) are somewhere along the path toward achieving that goal.

    A full 90% of highly evolved DevOps teams have automated most repetitive tasks compared to only 67% of mid-level organizations, the survey found. In addition, 91% of highly evolved teams report a clear understanding of their responsibilities.

    Not surprisingly, the biggest hurdles identified are more cultural than technical. The most common culture blockers at mid-level organizations include a culture that discourages risk (21%), unclear responsibilities (20%), deprioritizing fast flow optimization (18%) and insufficient feedback loops (17%). In contrast, less than 2% of highly evolved organizations report resistance to DevOps from the executive level, the Puppet report found.

    Nigel Kersten, a field CTO at Puppet, said during an online panel that, after more than a decade of DevOps advocacy, people continue to be the biggest limiting factor when it comes to adoption. Specifically, from a compensation perspective, many IT operations personnel tend to feel undervalued compared to developers, noted Kersten.

    Another challenge enterprises encounter is trying to balance the need to create a standardized IT environment that can be managed and secured consistently and the desire of DevOps teams to embrace new tools as they see fit. Senior IT executives often tend to make platform decisions based on cost criteria that don’t always account for the need to enable agile development methodologies. Most of those platforms are designed to be managed by an IT administrator using a graphical tool rather than via a command line interface (CLI) and application programming interfaces (APIs) that appeal to DevOps teams.

    As a result, individual pockets of DevOps teams that select and maintain a platform of their choosing are emerging within enterprise IT organizations. Those teams often view their chosen platform as a better alternative for advancing developer productivity. Finding and retaining site reliability engineers (SREs), for example, to manage those DevOps platforms is a major challenge for most enterprise IT organizations. Most IT administrators lack the programming skills needed to become an SRE, and it’s not clear how many IT administers are inclined to acquire those skills.

    Conversely, it’s not clear how many programmers want to programmatically manage IT infrastructure at the expense of time they would otherwise devote to writing code. Organizations that can only hire a limited number of developers may not want to shift responsibility for managing IT operations to the proverbial left simply because they need more code to be written faster. Finding and retaining developers, in many cases, is just as much a challenge as hiring and retaining SREs.

    Advances in automation and artificial intelligence (AI) should make it easier for organizations to embrace DevOps best practices in the months and years ahead. However, it should be apparent to all that adoption of those best practices across an entire enterprise still requires a significant amount of time and effort, and will for many years to come.

  • Why IT Should Focus on Quality Over Velocity

    Why IT Should Focus on Quality Over Velocity

    Quality assurance (QA) isn’t a new phrase for developers. Most know it as a staple part of the software process that is crucial for producing quality code. But throughout the pandemic, with extreme pressure for businesses to digitally transform quickly, it’s a discipline at risk of being dismissed by organizations and tech leaders as a last-minute, box-ticking exercise – but at what cost? Many have neglected QA in search of that quick app launch or new digital service, leading to an unwanted surprise or customer disappointment when the apps and features don’t work.

    We all know the old adage: ‘quality over quantity,’ but when considering software development, CIOs should have quality over velocity in mind. Although producing software at speed is clearly a necessity for business success–whether startups pivoting during a pandemic or enterprises keeping up with these nimbler competitors–QA must become more than a forgotten principle for internal IT teams. To achieve the necessary standards at the speed required, tech leadership must embed QA as a specified requirement of any software project, rather than an afterthought.

    Quality Assurance = Customer Satisfaction

    Quality assurance is about monitoring the software development process and method to ensure that the output adheres to the project requirements, as well as industry security and compliance standards. With QA at the core, businesses can gain a better understanding of the status of their product, which allows them to assess, improve and consistently enhance its output for better outcomes and know whether the output meets the business requirements.

    In order to draw out a strategy that ensures quality at each step, businesses need to understand their customers’ needs, user journey, expected target markets, and how they expect the software to perform. Without this, businesses will struggle to not only meet and ensure customer requirements but also business goals and expectations.

    Without QA, it is difficult for businesses to assess the functionality of the product or service and it becomes extremely difficult to track critical defects that become costly to address once found post-production. Poor QA can also lead to several other challenges, including delayed product releases, unhappy customers, reduced brand reputation and increased technical debt.

    Quality Shouldn’t be Assumed

    In its infancy, software development took on the best practices used in the hardware/manufacturing industry, which largely relies on assessing quality at the end of the product life cycle before release. As a result, much of the technology sector adopted outdated strategies which then became common practice. In fact, there are still hundreds of CIOs and CTOs I speak to that expect QA processes to be an add-on at the end of the software development process.

    Many technical leads believe it’s developers who drive quality in a software project throughout its life cycle. And this is true, to an extent – there are some fantastic, talented developers out there! But this perspective can teach bad habits and can mean QA is an expectation of individual developers rather than a specified requirement for the entire project.

    That’s because it’s easy to forget that developers aren’t quality engineers; those are, in fact, two entirely separate roles that should be working side-by-side throughout any digital project. A developer’s role involves writing and maintaining the source code, which usually includes implementing unit and integration testing to ensure that the code matches the requirements. However, they are not involved in performing functional and non-functional testing or reporting defects to the development team.

    It’s a People Problem

    Any business embarking on a digital project based on software needs both developers and QA testers—you can’t have one without the other. But tech leaders can struggle to hire both good developer talent and QA engineers—whether due to expense or lack of availability of skilled workers. Recruitment is a lengthy and costly process, especially when recruiting skills that are already in high demand.

    In light of this landscape and paired with the impacts of the COVID-19 pandemic, which has caused many businesses to slim down their in-house teams, IT departments are increasingly stretched. This means processes like QA, which many companies don’t view as a necessity or a priority, can be neglected. But putting quality assurance at the heart of the software life cycle can actually save time, helping to avoid clearing code with errors that will end up coming back to be updated.

    Ultimately, businesses don’t just need to build software, they need to improve it and make it work and evolve with its needs and goals. Quality assurance makes this possible.

  • Why You Should Embrace Incidents and Ditch MTTR

    Why You Should Embrace Incidents and Ditch MTTR

    The cliché is that everyone in IT hates incidents, and the natural reaction when assembling incident response metrics is to look for numbers that you can lower over time. Fewer incidents and shorter incident response times must be better, we think. You might already be familiar with the common metrics associated with these goals, including mean time-to-resolution (MTTR) and total incident count.

    As it turns out, however, these metrics actually work against your company’s resilience goals. To understand why, let’s look at the industry that birthed most of our modern incident management practices and has learned the most about how to properly apply them: the airline industry.

    What the Airlines Can Teach Us

    From the 1950s to the 1990s, the airline industry built and refined best practices for incident management that we have since simplified and that have been widely adopted across the tech industry. What’s often termed “modern incident management” simply means the predictable and repeatable approach to handling unexpected disasters and the learnings-driven remediation steps that are prized when building a DevOps or SRE practice.

    One of the most important findings to come out of this research is the somewhat surprising fact that incidents weren’t being declared often enough. Instead, incident reports were created and the carefully assembled response and cataloging process was only followed after catastrophic events, which meant earlier opportunities to catch the underlying problem during a less severe incident were being missed.

    So why did an industry with such incredibly high stakes take so long to realize that more incidents were actually a good thing? You guessed it: a misguided desire to decrease the incident count. In chasing this goal of “lowering the number,” pilots and mechanics were often reluctant to file incident reports for anything less than the most critical situations, which meant the powerful, learnings-focused response and remediation process was only triggered in response to relatively rare and critical situations.

    This is a textbook example of Goodhart’s Law in action. To paraphrase, Charles Goodhart posited that any metric created will ultimately become the goal. In other words, if the incentives are aligned to reduce incidents, no matter how good your response and learning process is, it will only be executed in the most dire and unavoidable circumstances. After discovering this, the industry rapidly moved their focus towards expanding the use of the incident process by working to expand the spectrum of situations that could be classified as incidents with the goal of triggering the formal incident response process more often, distributing learnings for more incidents of lower relative severity and, yes, ultimately resetting the goal to increase the number of incidents.

    This approach effectively turned Goodhart’s Law to the airlines’ advantage—any effort to game the metric(s) meant more learning, not less. Filing incidents became directly and heavily incentivized and these metrics were never used as a justification for punitive action. This resulted in the safest decade of commercial aviation on record, with over 12 billion passengers traveling and no fatal accidents.

    What IT Can Learn About Successful Incident Response

    As technology companies, there is much for us to learn from this exercise in achieving resilience versus chasing “lower numbers.” The primary learning is this: increasing your incident count by incentivizing the use of your incident management process for more non-critical incidents—likely your SEV2s and below—and focusing your metrics and incentives on that goal, is likely the best way to reduce your overall count of catastrophic failures.

    For most companies, this means putting MTTR on the back burner and instead focusing on making the incident process itself more accessible throughout your company and across your organizations. It means leaders should talk about impactful and recent postmortems publicly and that postmortem distribution should be a priority; it should also be something that’s measured. Most importantly, it means opening up the incident management process more widely. Ideally, everyone at your company should have access to systems that allow them to file incidents, read incidents in progress and review postmortems from previous incidents.

    I like to reference the “big red button” philosophy of factory floors, where the decision-making power to declare an incident is in the hands of every employee. An unsafe situation shouldn’t be up for discussion, debate or approval before being acted on, even if the situation is later determined to be less severe than originally thought. The context of why that incident was filed—why that big red button was pressed—may prevent larger, more catastrophic failures in the future.

    So if you’re just getting started building out an incident management practice at your company, or you’ve been chasing MTTR metrics and focusing on pushing your incident count down as a primary goal, consider taking these learnings to heart and creating a positive incident culture with positive metrics meant to encourage and reward incident responders and reporters. Your company’s future depends on it.

  • Aqua Security Acquires tfsec to Advance DevSecOps

    Aqua Security Acquires tfsec to Advance DevSecOps

    Aqua Security today announced it has acquired tfsec, an open source project that provides a static analysis scanner for infrastructure-as-code (IaC) that is designed to be integrated within a DevOps workflow.

    Amer Jerbi, Aqua Security CTO, said tfsec extends the security platform the company already provides in a way that advances adoption of DevSecOps best practices. In addition, to prevent developers from misconfiguring cloud infrastructure using tools such as Terraform, the scanner surfaces examples of code that provide developers with insights into how to properly configure infrastructure.

    Jerbi noted that tfsec is already integrated with Aqua Trivy, an open source tool that scans container images for vulnerabilities. Integrations with the rest of the Aqua Security portfolio will follow, added Jerbi.

    Tfsec co-founders Liam Galvin and Owen Rumney will join Aqua Security as cloud engineers. The challenge Aqua Security is trying to address is the need to enable organizations to proactively prevent cybersecurity issues from arising in production environments rather than only identifying vulnerabilities that need to be fixed, said Jerbi.

    Aqua Security tfsec

    The decision to acquire tfsec comes in the wake of a series of high-profile breaches when many organizations are scrambling to make sure software supply chains are secure. Cloud platforms are rife with misconfigurations mainly because developers often lack the expertise needed to ensure infrastructure is secure after they have provisioned it. Organizations are embracing DevSecOps best practices in the hopes of educating developers to assume more responsibility for application security. However, developers are not going to acquire that expertise overnight. The tfsec scanner provides a way to enforce security policies today while at the same time affording the time to teach developers how to avoid misconfigurations.

    It’s unclear how many cybersecurity events arise because of cloud misconfiguration issues. The cloud service providers have adopted a shared responsibility model that makes the organization employing the cloud responsible for both the applications deployed on those clouds and for how that infrastructure is provisioned. Many developers that provision cloud infrastructure themselves within the context of a DevOps workflow often assume the cloud service provider is doing more than it is to prevent misconfigurations. Cybercriminals, meanwhile, have become very adept at scanning for those misconfigurations.

    There are, of course, a lot of layers and nuances when it comes to DevSecOps. Arguably, Jerbi said, the most critical thing for most organizations is to simply get started. Tools that can be quickly and easily integrated within an existing DevOps workflow with minimal disruption are critical, Jerbi added.

    In the meantime, DevOps teams should expect cybersecurity teams to become more involved in their workflows, especially as government mandates aimed at securing software supply chains become stricter. Eventually, as organizations gain confidence in the ability of their DevOps teams to secure their application environments, those cybersecurity teams will refocus their efforts on hunting for threats. In the meantime, DevOps teams would be well-advised to make those cybersecurity professionals feel as welcome as possible, given the application security mandate most of them have been given.

  • 7 (More) Security Considerations for CI/CD

    7 (More) Security Considerations for CI/CD

    CI/CD orchestration has become increasingly popular in DevOps practices. It facilitates faster deployment of software and product delivery. Through the use of automation when building, testing and deploying applications, CI/CD bridges the gap between development, operation activities and teams. 

    However, there are many security considerations to keep in mind with frequent deployment. From quality automation to code history analysis, security must be baked into your CI/CD pipeline to ensure your product and processes remain safe from hackers.

    To gain insights on the top security considerations for CI/CD, I asked the speakers and sponsors for the upcoming SKILup Day as well as several DevOps Institute Ambassadors for their thoughts. In this two-part series, I will be sharing their insights. Did you miss part one? You can read it here. Here’s what they had to say in part two of this series:

    Speaker, Ravi Lachhman, Evangelist, Harness

    “CI/CD platforms are core to an organization’s engineering efficiency and typically have elevated privileges to sensitive environments such as production. Since CI/CD platforms typically will be deploying, this makes sense. Delving into the SolarWinds hack with binary artifact signing, etc., can separate security concerns into two parts. One concern would be the security around the actual CI/CD platform and one around the DevSecOps goals inside a CI/CD pipeline. With so many items shifting left, having proper role-based access control (RBAC) is crucial. With the separation of duties for regulated environments and the need for engineers to be efficient, controls that ensure only certain entities can deploy and certain entities can view pipelines is important. Around the DevSecOps movement, container/OSS scanning is still top of mind for many firms.”

    Sponsor, Basis Technologies, Eran Kriegshauser, Basis Technologies’ SAP DevOps Architect

    “The question you should be asking is ‘How do I maintain quality with the frequent deployment tempo demanded by business today?’ We need automation of quality and guaranteed rigor. For example, take promoting security configuration changes in systems, applications and products (SAP). Today, this is done by SAP transports where the change is created in an SAP development system, then imported into QA and, ultimately, into production. How do you ensure that a developer can’t give themselves extra security access through a role modification? Automated analysis and approval workflows must support segregation of duties requirements. Users can’t approve changes they create–the appropriate CI/CD process will not allow it.

    Another area is automatic peer review notification on changes to sensitive enterprise resource planning (ERP) objects. Once identified, the promotion of those object changes are halted until approved by a team lead or change and release manager.”

    Savinder Puri, DevOps Evangelist, Zensar Technologies

    “Infuse as many aspects of security as possible into your CI pipeline. CD is already too late for security—you’ve already deployed the application into an environment.

    Start from the basic static code analysis, security scanners, etc., and then keep maturing. Whatever tools you integrate into CI Server, ensure the corresponding integrated development environment (IDE) plugins are made available to developers. That way, devs can do the first-level validation within their IDE itself, and do it as they code, even before compiling. That’s the practical implementation of the often overused term ‘shift left.’”

    Vishnu Vasudevan, Head of Product Engineering and Development, Opsera

    “Now with cloud migration, emerging technologies and open source technology, companies will have to make sure they have the right tooling to measure and take action based on the security challenges born out of different development activities. 

    The focus will need to be on managing the risk that is tied to the business criticality. Each industry, domain, technology, etc., all have their own independent security goals, and you cannot have one security goal across the enterprise for different technologies and different domains. It is important to evaluate the technology being used and weigh the security requirements accordingly.”

    Stephen Walters, Sales Engineer, Everbridge

    “Inclusion! This is why the term DevSecOps came to be—because, erroneously, security was not included as part of a DevOps transformation, and the same is true for transformation of an SDLC to incorporate security.

    Security has still been treated as a non-functional requirement; an afterthought. If there is one thing we should learn, there is no such thing as a non-functional requirement. Security needs must be considered on an equal footing with all other requirements and built into the design, test, deployment and delivery of any feature.”

    Mark Peters, Technical Lead, Novetta

    “CI/CD pipelines should implement security at every step.  Tests can be automated in tooling to conduct tests of various functions at multiple levels for code accuracy, unit tests through fuzzing and integration or vulnerability testing. Most vulnerability testing in a pipeline is actually patch management, ensuring all commercial and proprietary holes are filled for release software. The first step should be using coding checks to make sure there are not any common problems with the code. While this may seem easy, if you are not managing that basic step, you cannot move ahead to the more complicated areas.”

    Pawel Piwosz, Lead Systems Engineer, EPAM Systems

    “At a high level, I would say that security is still outside the main scope of CI/CD. We should consider a few areas and different angles, here. Security should be not only implemented by CI/CD; CI/CD should be implemented in a secure way. Everything-as-code is a solution here: policy-as-code, security-as-code, drift-as-code, and if we talk about infrastructure, remediation-as-code.”

    If you’d like to learn more about the latest in CI/CD and similar topics, check out the upcoming SKILup Day schedule. Or, learn more about the benefits of becoming a DevOps Institute Member. 

  • Copado Acquires Qentinel to Gain AI Testing Tools

    Copado Acquires Qentinel to Gain AI Testing Tools

    Copado has acquired Qentinel as part of an effort to incorporate AI-infused application testing within a DevOps platform and build applications using low-code tools across multiple software-as-a-service (SaaS) platforms.

    The Copado continuous integration/continuous delivery (CI/CD) platform was originally created to accelerate the development of low-code applications on top of the Salesforce platform. Now, however, Copado is expanding the scope of its DevOps reach to include other SaaS platforms.

    Andrew Leigh, chief marketing officer for Copado, said as organizations increase the number of SaaS platforms they employ in the wake of the COVID-19 pandemic, it quickly became apparent there would soon be a need to build applications that enable organizations to unify business processes spanning multiple SaaS applications.

    As part of that effort, Copado’s acquisition of Qentinel aims to advance that goal by making it simpler for both developers and end users to participate in the application testing process, Leigh said.

    Qentinel is accessed mainly via the cloud to automate the creation of application tests. Rechristened Copado Robotic Testing, the core of the Qentinel platform is based on the open source Robot Framework that understands scripts, executes them and generates logs and reports. A set of built-in libraries comes packaged with Robot Framework and addresses a wide range of testing scenarios. Qentinel customers include Brunswick, Elisa and Konecranes.

    Copado AI testing

    Gartner is forecasting that, by 2024, 75% of large enterprises will be using AI-enabled test automation tools to advance continuous testing across different stages of the DevOps life cycle. Much of the interest in AI-infused application testing tools is being driven by necessity as the pace of low-code application development exceeds the pace at which manual testing processes can work. As application testing becomes a bottleneck, organizations will naturally look toward automation platforms that employ machine learning algorithms capable of testing applications at scale.

    It’s not clear to what degree those platforms might eliminate the need for humans to test applications. The expectation is humans will be able to test application functionality at a deeper level versus, for example, merely checking to see if compliance mandates have been met. Many smaller organizations building low-code applications don’t have any meaningful application testing capabilities at all. Testing platforms infused with AI will prove to be especially crucial for citizen developers that require guidance to build a secure application that can actually scale, noted Leigh. The more SaaS application environments those applications span, the more critical testing becomes, added Leigh.

    One way or another AI is coming to application testing. After all, machine learning algorithms don’t ever need to take a day off or suddenly leave to take a better offer elsewhere. In fact, soon, most organizations soon won’t even consider a test automation platform that doesn’t have AI capabilities as they attempt to reduce the level of manual, monotonous and repetitive testing tasks. It may take a while for everyone to gain confidence in the algorithms, but it’s now more a question of the degree to which AI will be employed rather than if.

  • Big Gains Are Made In The ‘Last Mile’ of DevOps

    Big Gains Are Made In The ‘Last Mile’ of DevOps

    A new study by mabl found a correlation between complete, fully automated test coverage and a significant increase in customer happiness and DevOps maturity. In contrast, those with moderate DevOps maturity and nearly automated testing coverage tend to experience a sharp decline in end-user satisfaction.

    Mabl recently released its 2021 State of Testing in DevOps Report, a study that highlights the state of DevOps adoption and quality testing throughout the software industry. According to Dan Belcher, co-founder, mabl, the study results clearly show that true business returns are made in the last mile of DevOps adoption.

    I recently met with Belcher to hear his viewpoint on the major takeaways. Below, we’ll explore these and other insights from the 2021 State of Testing in DevOps Report.

    State of DevOps Maturity

    Nearly one in four teams have a mostly mature DevOps methodology and automated process. Still, a significant portion, 52%, are still aspiring or striving to fully embrace a mature DevOps model. A small percentage, 11%, have a fully automated DevOps software development pipeline in place. The bulk of teams are still using manual deployment processes and disparate frameworks, limiting the potential for full automation.

    Interestingly, non-technical barriers remain the most common inhibitor to realizing a completely automated development pipeline, found the report. Out of these obstacles, reluctance to change was the most common barrier, cited by 29% of respondents. Other challenges to DevOps adoption include budgetary constraints, DevOps not being a company priority and internal politics.

    When added up, non-technical roadblocks account for 82% of the problems that hinder complete DevOps adoption. On the one hand, this depicts the cultural challenges associated with introducing DevOps in an organization. On the other, it could signify the wealth of DevOps packages and open source tooling available to manage areas like automated deployment and version control. The report found GitHub, Jenkins, Gitlab, Azure Pipelines and BitBucket among the five most commonly used DevOps Tools.

    Although it’s growing rapidly, continuous delivery is still an emerging trend; almost half of developers still deploy code less than once a month. Only 15% of teams are deploying code daily or multiple times per day. This echoes findings from CDF’s State of Continuous Delivery Report, which pegged this figure at 11%.

    Fully Automated Testing

    Testing is a critical component of the software life cycle. With the need to ensure functional applications and high-grade security, automated testing is helping teams discover more errors earlier in the development process. According to the report, the most popular test types include UI or functional tests, unit tests, regression tests, integration tests and end-to-end tests. Non-performance tests and visual testing ranked lower. Naturally, the teams that have implemented fully automated pipelines are automating more types of these tests.

    In general, it appears test automation aligns with DevOps maturity. The report found that those with fully automated testing experienced significant net customer experience gains. For example, 72% of teams that were automating regression testing indicated their customer happiness is pretty good or amazing, whereas this figure sank to 6% at companies not automating regression testing. And 80% of teams with high test coverage report high customer happiness, whereas only 30% of teams with low test coverage report high customer happiness. Nonetheless, much has yet to be fully automated—similar to the state of continuous deployment, the bulk of teams rely on a combination of automatic and manual testing approaches.

    In terms of tooling preferences, Selenium ranked as the most popular option for functional testing, with 41% of developers using it. The report also found migration toward Cypress, Smartbear and other functional testing tools. Interestingly, the report also uncovered a noticeable lack of adoption when it came to automated API testing—only about one out of three are automating their API tests. Postman, SoapUI, internal tools and ReadyAPI are among the most popular tools for running manual API tests.

    As I’ve covered before, APIs are surging in use across digital transformation initiatives. Web APIs are an increasingly common mechanism for delivering data to partners and consuming third-party software-as-a-service (SaaS). Full API testing coverage will be essential to audit the entire end-to-end digital experience. Low-code API testing is one way to reduce barriers to auditing the highly technical area of integration.

    Key Takeaways

    DevOps success is a ‘last mile’ problem. There is much variance in DevOps maturity and testing coverage. It’s pretty flat until you reach that last point, when breakthrough results occur. “Success in DevOps is a ‘last mile’ problem,” said Belcher. “The difference between 90% and 100% of the way there is dramatic.”

    Quality really matters. Though much tech discussion focuses on increasing delivery speed, code quality is just as important. “This isn’t about speed at all costs, it’s about innovation with velocity and quality,” said Belcher. “Quality engineers are just about as key to success as developers writing code.”

    DevOps success is less about technology and more about people. As the report found, the majority of the roadblocks to DevOps maturity are non-technical. “The things holding us back, as an industry, are more about leadership and figuring out our processing than introducing more technology,” said Belcher.

    Breaking down role silos is still fundamental to DevOps. Some see DevOps as instrumental to opening collaboration between roles. Simultaneously, there is the idea that new tooling dashboards will arm individual developers with complete ownership to develop, test and deploy software (known as full cycle development).

    Belcher believes that the full cycle momentum has been misinterpreted to mean developers alone have full accountability. Instead, he frames developers as a pitcher on a baseball team; leading the game, but utterly incapable without the rest of the team. Similarly, disparate roles should be working together, said Belcher. Reaching that ‘last mile’ of DevOps is not only about CI/CD or full cycle development; it’s about working together as a group.

    I’ve only scratched the surface of many findings from The State of Testing in DevOps. The report queried 600 quality engineers and software development professionals and offers loads of additional insights on the state of DevOps, testing, CI/CD and more. You can view the full results here.

  • Bringing Continuous Testing Into the Software Delivery Pipeline

    Bringing Continuous Testing Into the Software Delivery Pipeline

    For any highly competitive industry that relies on a smooth and stable user experience, it is vital that their development teams accelerate release cycles, lower operating costs and ultimately reduce risks. Continuous testing is one way companies can achieve these objectives.

    As a subset of software testing, continuous testing (CT) is valuable because it uncovers and addresses different problems at various stages of the development journey that other methods don’t. Most other testing methodologies require a large staff; CT doesn’t. CT leverages in-sprint automation, including end-to-end tests across the front and back end of the technology to cut costs and facilitate rapid delivery without adding more pressure on internal teams. Moreover, as part of a larger software delivery pipeline, CT is positioned closer to the beginning to execute tests early, often and comprehensively to realize cohesive release goals.

    CT introduces quality assurance with constant iterations of testing and actionable feedback at each stage of development to evaluate business risk coverage and ensure the best user experience and customer satisfaction. As the name implies, CT helps developers and product managers detect and fix critical bugs not only in the early stages of development, but continuously throughout the software development life cycle (SDLC), ensuring that errors in the code do not get unintentionally released into the wild.

    As software companies accelerate their release frequency, continuous testing sets a precedent for digital tools development and the quality assurance process.

    Benefits of Continuous Testing

    Continuous testing plays an integral role in the CI/CD pipeline, is part of the overall set of DevOps best practices and the information it provides is invaluable to development teams. The benefits of CT are also far-reaching, with positive impacts on the entire organization, including:

    Better Company-Wide Feedback

    Continuous testing facilitates feedback to all involved teams and stakeholders. For example, when a bug gets detected in code, ideally, the only team notified should be the team responsible for the part of the code with the bug. Without CT to automatically detect bugs and notify those responsible for fixing errors, triaging bugs could be a full team effort or, worse, a dedicated team’s main function. This approach is very inefficient and wastes the time of the engineers whose code is bug-free. The feedback provided from CT cuts down on these unproductive scenarios and allows stakeholders to make immediate and informed decisions.

    Higher Quality Code and User Experience

    A company that effectively uses CT should expect higher quality code and a more stable user experience. Stability measures application health and user experience; it is negatively affected by session-ending events like crashes. With CT, businesses will give their development teams better awareness of product quality and performance while also significantly improving overall business metrics by fixing broken apps faster.

    CT Breaks Down Siloed Teams

    Not only will CT enhance quality assurance, it similarly works to improve the development of digital tools through the breakdown of siloed teams. When teams become siloed, they inadvertently keep processes and knowledge to themselves, resist cross-functional collaboration and work in isolation toward their own goals. Predictably, separate groups fail to share insights or ideas, leading to stagnation and decreasing innovation. However, by leveraging CT, siloed teams can become more collaborative in development, testing and operations, increasing overall business efficiency.

    CT Enables Objective Evaluation

    Finally, CT enables an organization to pause and objectively assess if it is aligning efforts with its specific business goals. As mentioned before, CT uses automation to provide timely feedback on business risks, reducing the chances of a digital application or tool failing at launch. Likewise, dev teams and managers can more easily meet their business goals and optimize the value of a release. Additionally, objective assessments are helpful for critical decisions – as CT gives actionable data for decision-makers to support their choices.

    Today, consumers have come to expect error-free experiences within their software applications, with the most stable of these applications driving conversion, engagement and retention. Applying a continuous testing model to a software development pipeline will ultimately result in applications being delivered faster and at higher levels of quality than ever before, with an enhanced end-user experience. As companies combine agile development with continuous testing, they will optimize each aspect of the development life cycle and their larger business goals.

  • Survey Finds Biggest DevOps Obstacles are Organizational

    Survey Finds Biggest DevOps Obstacles are Organizational

    A survey of 600 IT professionals published this week by mabl, a provider of a test automation platform, finds the biggest barriers to adopting DevOps practices are more closely related to organizational issues rather than any technology.

    The survey identified non-technological issues as the main inhibitor to DevOps by a wide margin (82%). Only 11% of respondents said their organizations has fully implemented DevOps practices, while another 24% said they were most of the way toward achieving that goal.

    The top two barriers to adoption are slow processes and speed of adaptation (29%), followed by budget and funding (21%). Only 18% identified technology limitations as an issue.

    Nearly half of respondents (46%) said they have been able to implement continuous integration, while more than one-third (36%) said they have also been able to achieve continuous delivery. Just over a quarter (27%) said they have also achieved continuous deployment. However, only a third of respondents said they deploy new code at least once per week. Almost half still deploy less than once per month. The number of teams that deploy new code daily or multiple times a day (15%) is roughly equivalent to those that deploy code on a quarterly basis (16%).

    The most widely used DevOps tools are GitHub (34%) and Jenkins (33%), followed by GitLab (21%), Azure Pipelines (20%) and Bitbucket (19%).

    In terms of testing, the most widely automated tests are user interface (UI)/functional and unit tests at 42% each, followed by regression at 41% and integration at 35%, respectively. The most widely used functional testing tool is Selenium (41%), followed by Cypress at a distant second place (14%). The most widely employed tool for testing application programming interfaces (APIs) is Postman, followed by SoapUI (21%) and homegrown tools (17%).

    Dan Belcher, co-founder of mabl, said that while it’s clear most organizations still have a long way to go before they fully embrace DevOps best practices, it’s now more a question of degree rather than if. As organizations realize the extent to which they are dependent on software today, the focus on both quality and ensuring timely delivery will increase. The stakes involved in software delivery these days have never been higher, noted Belcher.

    Nevertheless, the survey makes it clear how resistant to change organizations are. Some organizations have been building software the same way for decades. The only thing more difficult than convincing IT and business leaders there is a fundamental need for change is driving that change all the way down to individual developers and IT operations teams.

    Competitive pressure, however, will inevitably force the issue. If rivals are delivering better application experiences that create a business advantage within the context of a digital business process, an organization will have to adapt. Otherwise, their fate will be, over time, to fade away or be acquired. Equating DevOps with that existential threat may be a challenge, but unless that connection is made it’s only a matter of time before organizations that aren’t embracing DevOps best practices will slide toward irrelevance.