devopsy.kinsta.cloud

Tag: test automation

How Test Automation Can Help Drive DevOps Success

Testing can be one of the more time-consuming aspects of the DevOps process. It can be exhaustive, frustrating, requires a lot of hands-on labor hours and people seldom enjoy it. But testing is vital, and no developer should simply skip it.

Recently, a strong trend has emerged wherein DevOps experts encouraged developers to automate their testing procedures. Test automation can bring your DevOps projects to the next level, save time and maximize pipeline capacity for continuous testing.

Test Automation: Definition and Benefits

Test automation minimizes the amount of human labor required in testing. In test automation, tools and frameworks write testing “scripts” that work to ensure the high quality of your application.

DevOps can use test automation in a variety of different ways–it can even take advantage of test cases and detect bugs in your code. This automatic testing procedure speeds up the process, getting your product to the market much more rapidly. The automatic quality checks can also practically eliminate the risk of human error in the process of continuous testing.

Test automation can also ease the burden of the repetitive nature of some DevOps testing, simplifying and speeding up the process significantly. Finally, automation promotes a high degree of reliability throughout your testing process. Test automation can also optimize your process.

Test Automation in the Context of DevOps

DevOps is all about continuous development—the product is created, deployed, tested, and sent out to be consumed by the end users.

Since it came on the scene, quality assurance (QA) has gained importance. The traditional approach to QA (wait until the product deployed to go through regression and functional testing) was much improved with the advent of DevOps. DevOps integrates the QA team into the whole of the development process, merging all players into a unified team. This integration requires a great deal of coordination across the different stakeholders.

This also allows for a high degree of agility and speed in the process, speed which the DevOps methodology then magnifies through a high amount of automation in continuous testing (which can then be configured to run automatically).

Deciding on the Best Tools for You

To build up an effective test automation process, you must find the right software workflow tools for you and your DevOps team. These tools should support and manage your test assets from a central location, passing your product through the testing process much more quickly and maximizing efficiency. That said, as helpful as having an automated testing process can be to enhance the efficiency and effectiveness of your software, using tools that don’t fit with your team and goals will present potentially significant problems for your organization. There is a staggering array of test automation tools available on the market, and they all have their strengths and drawbacks. No single tool will satisfy every need related to automated testing, so your goal needs to be to find the ones that will work the best with your unique needs and desires.

What to Keep in Mind When Choosing a New Tool

Cost (for both licensing and support): Budget is an eternal tyrant, and your team needs to go into this procurement work with a solid understanding of your financial limits. If you have stringent limits there, there are some types of open source software like Selenium that could be useful. These tend to require a substantial degree of technical skill, so they may not be as helpful for you—if that’s a weakness of your team.

Type: There are several different types of DevOps testing tools. Language is one of the more important factors when deciding on one–make sure the tool operates in either the same language as your application or at least one in which your team can efficiently operate. The intended platform for the tool is also notable, as is the intended audience (i.e. whether the product aims more towards developers or testers).

DevOps integration support: It’s always important to double-check that your tool supports both DevOps tool integration and continuous integration. If you don’t effectively coordinate your tools, it will undoubtedly slow down your process.

Training: Depending on the level of expertise within your DevOps team, you’ll need to ensure that there is an appropriate degree of training on how to use the team—for the people who will be using it.

Quality test reports: Test reports with both depth and breadth are crucial to success in continuous testing. These help notice defects and their root causes, then allow you to test the effectiveness of your analysis.

Excellent customer support: You will want to use a tool that will serve your needs well. That means a tool that understands your specific situation and can help you work through complications.

The Necessity of Automation Skills

All of this aside, you cannot just rely on the strength of your test automation tools to get you through the testing process. If you decide to implement test automation without having people on your team with the skills to manage it, you may end up in a challenging position riddled with inefficiency.

Here’s what you need from a test automation software engineer: They must have a broad base of knowledge that encompasses test automation tools. Beyond that, they need to understand how to dive deep into software testing functionality and the technical complexities that undergird the project. They should also know how to write scripts alongside development, collaborating with both operations and development teams. That collaboration between test automation, development and operations will have a deeply important role in writing test scripts and maximizing the coverage of your tests.

A test automation software engineer’s skills offer invaluable stability. Indeed, tools are often built on a foundation of skills. If you have a resourceful and capable software engineer on hand, you’ll be much more prepared for the fluctuations of the market than if you’re relying on a single tool. If you decide to focus on only one tool, your team risks limiting itself and potentially falling behind the times and needs of your customers.

The Dangers of Automation

Automation is one of the foundations of DevOps’ success. But even so, you must strike a delicate balance. If you automate too much, you may end up hurting your product.

When you start connecting too many different tools to your project, you begin to risk unwanted outcomes that practically cancel out the investment you made in DevOps in the first place. If you try to automatically test things that automation can’t effectively measure (general user experience, for example), you’ll end up with worthless data. Your various automation tools can start to create more work than they’re resolving if you need to spend a great deal of effort in managing them.

That said, if you under-automate, you’ll be losing out on workflow efficiencies that your competitors are using, making your team less competitive. Thus, it’s necessary to try and balance the two. The first step to striking this balance is having a thorough understanding of your requirements vis-a-vis testing, development and deployment. If you have that under your belt, you can find the automation tools that work best for your goals. Once you have the right combination of automation tools, you’re golden.

Also, test automation doesn’t negate the need for manual testing—and it isn’t designed to. Of course, certain kinds of manual testing can be eliminated by automation (especially the repetitive kinds).

But you still need manual testers behind the wheel examining your product more thoughtfully and at a deeper level than an automated program. That deep-level variety of manual testing can often discover issues that automatic testing failed to see. It can also provide a more thorough level of insight into the user experience that automatic testing will seldom be able to provide.

Benefits of Test Automation

We’ve covered a lot of deep ideas in this piece. But at the end of the day, test automation is beneficial for a few simple reasons: it reduces human error, simplifies your tests, allows you to deliver products much more quickly and leads to a much more reliable quality assurance structure.

That said, test automation won’t be a panacea for all your team’s ills. It will work, but only if you’ve chosen the right tools and you have people on your team with the skills to use them effectively. Once again, don’t rely on continuous automated testing as the be-all and end-all of your quality assurance practice. Several QA questions show that a human presence is much better at resolving certain types of problems and over-automating can cause significant inefficiencies.

Test automation can be a wildly useful tool if you use it well. These tips will help you take your organization to the next level of efficiency and excellence.

January 19, 2022
Survey Finds Biggest DevOps Obstacles are Organizational

A survey of 600 IT professionals published this week by mabl, a provider of a test automation platform, finds the biggest barriers to adopting DevOps practices are more closely related to organizational issues rather than any technology.

The survey identified non-technological issues as the main inhibitor to DevOps by a wide margin (82%). Only 11% of respondents said their organizations has fully implemented DevOps practices, while another 24% said they were most of the way toward achieving that goal.

The top two barriers to adoption are slow processes and speed of adaptation (29%), followed by budget and funding (21%). Only 18% identified technology limitations as an issue.

Nearly half of respondents (46%) said they have been able to implement continuous integration, while more than one-third (36%) said they have also been able to achieve continuous delivery. Just over a quarter (27%) said they have also achieved continuous deployment. However, only a third of respondents said they deploy new code at least once per week. Almost half still deploy less than once per month. The number of teams that deploy new code daily or multiple times a day (15%) is roughly equivalent to those that deploy code on a quarterly basis (16%).

The most widely used DevOps tools are GitHub (34%) and Jenkins (33%), followed by GitLab (21%), Azure Pipelines (20%) and Bitbucket (19%).

In terms of testing, the most widely automated tests are user interface (UI)/functional and unit tests at 42% each, followed by regression at 41% and integration at 35%, respectively. The most widely used functional testing tool is Selenium (41%), followed by Cypress at a distant second place (14%). The most widely employed tool for testing application programming interfaces (APIs) is Postman, followed by SoapUI (21%) and homegrown tools (17%).

Dan Belcher, co-founder of mabl, said that while it’s clear most organizations still have a long way to go before they fully embrace DevOps best practices, it’s now more a question of degree rather than if. As organizations realize the extent to which they are dependent on software today, the focus on both quality and ensuring timely delivery will increase. The stakes involved in software delivery these days have never been higher, noted Belcher.

Nevertheless, the survey makes it clear how resistant to change organizations are. Some organizations have been building software the same way for decades. The only thing more difficult than convincing IT and business leaders there is a fundamental need for change is driving that change all the way down to individual developers and IT operations teams.

Competitive pressure, however, will inevitably force the issue. If rivals are delivering better application experiences that create a business advantage within the context of a digital business process, an organization will have to adapt. Otherwise, their fate will be, over time, to fade away or be acquired. Equating DevOps with that existential threat may be a challenge, but unless that connection is made it’s only a matter of time before organizations that aren’t embracing DevOps best practices will slide toward irrelevance.

July 2, 2021
Make a Plan for Test Automation

The amount of testing that we could be doing is massive. Most of us don’t look at testing across the spectrum and all-inclusively, but let’s do that for a second.

We have functional testing at the code level, which is reasonably well automated already, if your shop is using such automation. Then we have integration testing, which is a superset collection of API testing, code integration testing, usability and regression testing. Integration testing is automated, and sometimes that’s done well, but this massive space is still growing. Next up would be security testing, which covers much of the same code that other testing is interested in from a strictly security focused perspective. Then, there is actual usability testing, which is more important for some areas than others—embedded needs far less usability testing than web apps, for example—and has different levels of automation at this point in time.

Really, when you think about it, the complexity of this environment is massive. We not only need to test an array of different functionalities from an array of different perspectives, but we have to do it on an array of platforms. Testing on Windows is different than testing on Linux; testing in the cloud can use different tools than testing in-house; clients can be on any given platform, depending on the application and target market … we really do have a beast, here.

And testers have always been behind, or have not covered things well. This is not a ding on testers, this is the result of the complexity at hand and the fact that, in many instances, one line of code can go wrong in a bunch of different ways.

Enter automation, which is growing rapidly in the world of continuous integration (CI). Automated testing tools range from broad to very focused, and all help facilitate testing. The DevOps increase in development means more testing, and testing was already behind. This has been discussed pretty thoroughly in the past, but it finally feels like we’re getting to a point where we have viable solutions to the problem. If testers become test developers that review results of automated tests, and through CI those automated tests are increasingly worked into the process of development, we’ll see testers getting better coverage and quality seeing a continued increase.

But that does mean you should keep your eyes on the market as it continues to grow, and pick tools that offer the most functionality that is most useful to your test team. And get them up and running. More testing means more work setting up tests. Even no-code test solutions for UI require someone to record the actions that are necessary to complete the test, and that takes someone’s time. But the return is long-term testing that doesn’t require a person until a red flag is raised by automation.

I’m all for it. Both testing and security have the most to gain from automation, because they’ve been the most behind historically, and corner-cutting has cost organizations dearly. So, let’s get with it; check out some new test tools (there are plenty listware articles giving you a place to start), and let’s make great software better!

June 9, 2021
How DevOps Tools Are Contributing to Automation Initiatives Across the IT Landscape

Prolific advancements in IT, specifically in agile methods and culture over the years, has led to burgeoning needs for a more holistic end-to-end software delivery approach. Agile development is an umbrella term that encompasses numerous incremental and iterative software development practices — including Kanban, SAFe (Scaled Agile Framework), Scrum, Extreme Programming (XP), Lean Development, among others.

Each methodology works on its own unique approach. However, the core commonalities they share lie mainly in their values and vision. The methodologies integrate continuous feedback from iteration to refine and deliver a successful software system. They include various forms of continuous evolution, such as continuous integration, planning and testing, of both software and project. Furthermore, one of the most significant aspects of agile methodologies is their keen focus on empowering workforces to work together and collaborate more efficiently, thus simplifying and speeding up decision-making processes.

Advancement in agile development is what has birthed the DevOps culture. The technology is born from the rising need to adapt to increasing software velocity and throughputs generated through agile methods.

Understanding DevOps

The DevOps market consists of a set of practices, designed to integrate and automate processes between IT and software development teams, enabling them to create, test and deploy software in a quicker and more efficient manner.

The term DevOps, which was coined in 2009 by Andrew Clay Shafer and Patrick Debois, is a combination of the words “development” and “operations.” It is also indicative of a cultural shift in the IT sphere, which bridges the gap between historically siloed teams, such as operations and development.

The DevOps framework essentially adopts a culture where the critical focus is on facilitating collaboration between the business, operations and development teams. Although tools are considered as pillars of the technology, DevOps is designed to deliver continuous value for customers, empower organizations to provide solutions of the highest quality at swift rates, and bring about the automation of the entire business process spectrum, from building to deployment of a product or application.

Shifting Trends Toward Automation in Software Development Processes and the Subsequent Impact on DevOps Adoption in the IT Space

The software development process consists of several integral parts, among which testing is a crucial task. It allows developers to monitor the functionality of the software, monitor and register performance, and detect any aspects that require remediation. However, with software projects becoming increasingly more complex as the years progress, and with project development life cycles accelerating, conventional methods of manual QA (quality assurance) testing fall short in meeting testing objectives within the desired time frames.

Consequently, software developers are shifting their focus toward automated workflows and testing tools, in order to expedite testing routines without compromising on consistency and completeness in QA processes.

DevOps practices are significantly dependent on automation to ensure frequent and timely deliveries across platforms. Studies have shown that nearly 85% of IT leaders claim that automation is a critical aspect of their DevOps strategies. This is attributed largely to the fact that automation empowers infrastructures to successfully field constant changes in codes that are associated with DevOps tools.

Furthermore, automation allows the effortless scaling of environments and unburdens skilled workforces from mundane and repetitive tasks, enabling them to focus on more integral aspects.

Considering a process perspective, automation in testing involves the addition of test automation actions and tools to regular software development regimens. For instance, new builds delivered to a repository can be automatically subjected to automated testing procedures using a number of prescribed tools, with little to no need of developer intervention. The results are recorded with care, then compared to previous test runs and eventually submitted for review to developers.

Depending on the outcome, the software can then be cycled back to the developers for further remediation or approved for deployment. This is especially relevant in terms of the DevOps framework, which relies on continuous integration/continuous delivery (CI/CD) pipelines.

CI/CD pipelines are a prominent outcome of developer operations, allowing developers to validate the quality of the software with minimal intervention, and deploy apps and software to customers in a quicker manner.

CI/CD platform CircleCI, for example, has recently raised over $100 million in Series E funding, in an effort to boost the development of their DevOps product suite. This move is indicative of CircleCI’s proficiency in delivering an automated approach to DevOps practices, including tools for companies to create, test and deliver software.

How Is the Current Pandemic Impacting Digital Development?

The COVID-19 pandemic has asserted a massive influence on the industrial landscape, driving workforces across the globe into remote-only work set-ups. With millions of people working from home for a seemingly indefinite period, enterprises across the globe are rallying to accelerate automation efforts in order to more efficiently deliver digital business applications to engage their employees from remote workspaces.

According to various studies, nearly 73% of IT operations and DevOps teams are working toward maintaining and expediting initiatives for digital transformation throughout the current crisis, holding the belief that the value of digital offerings is set to witness a positive growth trajectory in the social distancing era.

DevOps industry players such as JFrog are working toward providing enterprises and teams with access to cloud-based DevOps technologies, free of cost, particularly those which are undertaking active research and response efforts to combat the COVID-19 pandemic. These DevOps technologies include Pipelines, Xray and Artifactory, among others.

While the exact impact of the global crisis on the DevOps environment in particular and the IT space in general is yet unclear, it is becoming more prominent that the strong focus on the prevention of such crises in the future — both in and out of the IT realm — is likely to set a great precedent for the growth of the DevOps industry in the years ahead.

June 16, 2020
Quality at Speed During the COVID-19 Pandemic

Think back to 2015. The U.S. economy was in the early phases of a growth spurt that would continue for years. Ninety-five percent of organizations were practicing agile, to some extent (State of Agile, VersionOne). And the proportion of the IT budget allocated to QA and testing had just increased from 18% to 26% (World Quality Report, Capgemini).

“Quality at speed” became the catchphrase of the year in the testing world. With all the buzz around DevOps and agile, it was clear that speed mattered. In reality, most teams weren’t quite there yet. Agile adoption was still low (less than 50% of teams within each organization actually practiced it) and maturity was even lower (only 17% of those practicing it reported mature adoptions). Still, the writing on the wall was clear. Leaders wanted more application development, faster — so testers should at least prepare for the pending storm.

From that point forward, organizations across almost every industry all progressed to a sweet spot where time-to-market pressure was relatively high and cost pressure was moderate. By the end of 2019, quality at speed had graduated from buzzword to business as usual at most organizations.

But then came COVID-19. Now, we’re in the midst of a global pandemic. Every business, every non-profit and every governmental agency is impacted — but in different ways.

Speed expectations suddenly shifted into overdrive for an unlikely assortment of organizations, including government agencies and non-profits, healthcare and pharmaceutical companies, online retailers, online conferencing providers and telecoms. Yes, everyone had already been working to accelerate their software delivery processes. And many enterprise organizations had made great strides toward scaling the adoption of agile and DevOps. But, very few of these organizations were previously delivering applications at the extreme speeds required right now.

For another group, cost reduction suddenly became the top priority. Airlines, hotels, as well as other travel and leisure organizations experienced such dramatic changes in fortune that speed became a non-issue. The focus shifted to increasing cost efficiency in an effort to offset some of these staggering losses.

Assuming that high quality cannot be compromised, this leaves both of these groups with distinctly different — yet equally daunting — challenges from a quality perspective.

The organizations in the first group suddenly needed to build, test and release applications at the speed of a startup — but with the added burden of legacy systems, complex application landscapes and intense regulatory scrutiny. Quality at speed remains the goal, but the “speed” part is much, much faster.

The organizations in the second group need to determine how to continue meeting strict quality expectations with a significantly slimmer testing budget. Quality at speed isn’t so important right now. Speed gets replaced by the other corner of the classic cost-quality-speed triangle, making the new mantra “quality at cost.”

It’s hard to find good news in the current situation. But, from the software testing perspective at least, there’s some consolation. Whether you’re facing unprecedented time-to-market pressure or extreme cost pressure, the same core “test efficiency” strategies will help you either way.

Here’s a quick overview of ways you can test more efficiently at different steps of the software testing process.

Selecting and Designing Test Cases

I’m addressing these two steps together because they both face a common challenge: intuition. Most modern business systems are insanely complex — so complex that it’s just not feasible for most testers to intuitively identify the best set of tests to cover a specific test need. Having a more systematic way to find the smallest — yet most powerful — set of tests will save you both time and money.

For designing test cases for progression testing, methodologies like linear expansion and pairwise testing can help, and augmented intelligence can dramatically simplify them for you.

For selecting what test cases should be run for regression testing, consider change impact analysis. Some approaches identify which objects are impacted by a given change, then help you narrow the test scope to the impacted objects. This often reduces the testing scope by 15%. Not bad, but you can do much better.

AI-powered change impact analysis pinpoints exactly what needs to be tested in a given update, based on whether the changes put an object at risk. With this approach, you can reduce testing efforts by 85% or more. Not only is this approach faster and cheaper, it’s much more efficient at uncovering defects. The number of defects that hit production are reduced by 75% to 95%, sometimes even 100% (no defects in production).

Automation and Execution

Here, technical scripting is the top time and cost drain. Some level of maintenance is required to keep scripts running, and this maintenance will cut into your test efficiency. One way to combat this challenge is to take a model-based approach to test automation. Such approaches have been field-tested for years, and proven to solve the test maintenance nightmare that’s doomed all-too-many test automation initiatives.

The new kid on the block is AI-driven UI test automation. This is a big topic that warrants a much deeper discussion — but here’s a quick overview. Specify tests in natural language. Build the tests even before the actual UI is ready. Use the same tests as the technical implementation changes. Understand dynamic behavior like a human tester would. Test absolutely any technology — faster than a human eye. Curious? Stay tuned.

Analysis of Results

When it comes to results analysis, false positives are the vampire sucking your resources. False positives are tests that fail even though the application is behaving correctly. They will really kill you in terms of cost and speed. First, someone needs to figure out why the test failed. This might involve manually checking the related application functionality, inspecting the test automation, reviewing the test data, checking whether every element of the test environment is up and working as expected — maybe even all of the above. Once the exact problem is found, it needs to be fixed. Then, the test needs to be rerun. If it still fails, rinse and repeat.

How do you stop false positives? To start, stabilize the automation with model-based test automation or the AI-driven testing I mentioned in the previous section. As you move into more advanced automation scenarios, stateful test data management and service virtualization can help you stabilize tests to the point that they’re fit for CI/CD. Then, your final challenge is to deal with the flaky tests resulting from things like asynchronous application behavior, concurrency issues and test order dependencies. Again, AI and ML can offer tremendous assistance here—in flagging potential false positives, and even in helping you identify the cause and optimal resolution.

Final Thoughts

COVID-19 split up the original quality at speed sweet spot into the extreme speed camp and the extreme cost-savings camp. In both cases, the same core strategies to increase your test efficiency will help you increase speed as well as reduce testing costs. If you’re under extreme time pressure, showing that you also reduced costs while meeting speed expectations certainly won’t hurt. If you’re currently reducing costs because your industry is essentially paused, taking these steps now will prepare you to speed it back up when the world becomes more normal again.

June 4, 2020
Why You Can’t Have DevOps Without Test Automation

The rapid adoption of digital technologies by enterprises to stay nimble, competitive and quality-driven has given digital transformation wide currency. The process facilitates decision-making and allows organizations to innovate, accelerate the time to market and be responsive to customer needs.

The world of today is for those enterprises that can quickly turn an idea into a consumer product or service, absorb feedback from the market and customers to refine the product/service and relaunch it. This is where DevOps can help such organizations in terms of offering continuous development, innovation, integration and deployment.

Fixing Glitches with DevOps Testing Strategy

To stay on the right side of customer preference, enterprises must roll out software updates on a trot, which is not easy by any stretch of imagination. This is because every update needs to be tested across devices, operating platforms, frameworks, geographies and networks. If the DevOps approach is not right in executing the process of development, testing and deployment, the inherent glitches can prove to be costly. In many cases, in a bid to be ahead of the curve, enterprises do not give testing its due.

In other words, products or their updates are often launched without letting them pass through a stringent QA pipeline. The consequences of such quick deployment without adequate testing may manifest themselves in various ways. These may include the quality of product taking a beating, regulatory bodies subjecting such organizations to censure and penalties, and customers taking a flight to the competitors. With business reputation at stake, enterprises are well-advised to follow a strong DevOps approach and identify/fix glitches quickly in the pipeline.

DevOps and Test Automation

When it comes to DevOps specialists discussing key elements that go into a continuous integration (CI) or continuous delivery (CD) pipeline, they mostly talk about code repositories or CI servers. Interestingly, DevOps test automation finds less mention in the rarefied circles where strategies are planned. This is despite the DevOps test automation being considered an integral part or medium to achieve outcomes like CI and CD.

DevOps specialists mostly focus on aspects such as integration and deployment of the software, thereby giving test automation a miss. This can be self-defeating and invariably goes against the grain of DevOps philosophy where QA forms an integral part along with development and operations. Does this mean these specialists are more tuned into the manual testing aspect while leaving test automation altogether? The probable answer would be the ingrained belief in some DevOps experts about the efficacy of manual testing. This would not have raised any hackles until the advent of continuous delivery chains.

Why Test Automation for DevOps?

The two outcomes of DevOps, CI and CD, mean the development pipeline should deliver quality products quickly and consistently. Besides, the products should be upgraded periodically based on market feedback. This means products should be tested for glitches early in the SDLC to save cost and time. What better way to achieve the same than by implementing test automation in DevOps QA.

Faster test outcomes: Test automation can be implemented by writing a test script on a tool. The code in development is put through that script and checked for expected outcomes. Any deviation is noted and investigated for possible errors. This is way faster than manual testing, where QA specialists find repetitive testing frustrating and have a tendency to miss errors.

Consistent results: Consistency in QA means the software behaves on expected lines throughout the delivery pipeline. Test automation can validate the code for variables at different stages and mitigate the chances of human errors.

Ease with regression testing: When a major update is implemented, certain types of tests, such as usability testing, are performed manually. However, when there are more variables and the software needs to be checked across devices, operating systems, browsers, geographies and networks, then automatic testing takes the cake. It can work non-stop and check for deviations or non-compliance.

Human resources utilization: During manual testing, scarce human testers are deployed in tasks that require constant attention. Further, to avoid boredom arising out of conducting repeated testing, QA testers can give certain tests a miss. Imagine the type or number of glitches escaping the test dragnet and later ruining the user experience. However, with test automation, repetitive testing can go unhindered wherein scarce human resources can be deployed for better activities.

Conclusion

In today’s age of fast software releases, quality can be ensured through a smooth DevOps-led pipeline ensuring CI and CD. However, achieving consistency in testing and pre-empting issues like cybercrime are underpinned on implementing DevSecOps and enabling a culture of quality control at every step of the way.

April 30, 2020
Software Testing: The Comeback Kid of the 2020s

The past 15 years have been rough on software testers. The rise and fall of outsourcing, Agile growing from a manifesto to a movement and ever-shrinking release cycles have all taken a toll. By 2011, Albert Savoia walked on stage at the Google Test Automation Conference (GTAC) dressed as the Grim Reaper and declared testing dead. Since then, the emphasis on the “automation” part of “test automation” has fueled demand for Software Development Engineers in Test (SDETs) and left many testers wondering if they must learn programming to save their jobs.

If you look at how frequently people are searching for “software testing” on Google, the trends seem to suggest that the profession is, if not dead, at least flatlining.

However, I think the times are a-changin’. The pendulum will be swinging back—to the point where software testing could be the comeback kid of the 2020s. Sounds crazy, I know. But consider some developments that could turn things around.

Developers Don’t Want to Bear the Testing Burden

On Agile development teams, the party line is that unit testing is essential. It’s the most efficient way to expose implementation errors as they’re introduced. You can feasibly cover each function/method when you test at this level, plus when a test fails, you know exactly what function is broken. Moreover, having a suite of effective unit tests establishes a safety net that alerts the team whenever changes break previously-verified functionality. Those are just a few of the reasons why unit testing forms the foundation of the Agile test pyramid.

But, if you talk to developers in a casual setting, the truth starts to come out. Unit testing requires a lot of work, and they’re skeptical about whether it pays off. Just skim through a few of the 2.5 billion results returned by a Google search for “Is unit testing really worth it?” to get an idea of some of the issues.

Unfortunately, a seemingly endless amount of work is required to ensure that unit testing delivers long-term benefits as a regression testing safety net. Developers are committed to progression testing, not regression testing. Although developers on Agile teams usually write unit tests to check new functionality, few keep them up to date as the code base evolves. As the team moves on to another sprint, these tests become regression tests. Then, little by little, they start failing—but developers have moved on. The number of working unit tests at the base of the Agile test pyramid erodes, as does the level of confidence the test suite once provided.

In fact, data from our internal studies on financial instructions across EMEA shows that as an iteration ends, unit tests cover about 70% of the new code. After a few iterations of code getting extended, refactored and repaired, the original unit tests provide only about 50% coverage of that original functionality. This drops to 35% after several more iterations, and it typically degrades to 25% after six months.

Is Unit Testing Doomed?

Ultimately, developers don’t have the time or desire to keep these tests current over the long term. Unit testing has been a best practice for more than 20 years, yet despite waves of unit test automation tools (including one created by Albert Savoia not long before he declared testing dead), unit testing remains a thorn in developers’ sides. Does that mean we give up the benefits of unit testing altogether? Not necessarily.

In order to take on unit testing per se, testers would need to understand the developers’ code as well as write their own code. That’s not going to happen. But, you could have testers compensate for lost unit test coverage through resilient tests they can create and control. Professional testers recognize that designing and maintaining tests is their primary job and that they are ultimately evaluated by the success and effectiveness of the test suite. Let’s be honest, who’s more likely to keep tests current, the developers who are pressured to deliver more code faster, or the testers who are rewarded for finding major issues (or blamed for overlooking them)?

Another option is to consider new testing approaches that achieve unit testing benefits for today’s most popular development architectures—but without requiring knowledge of the code. Enter microservice contract testing.

Microservice Contract Testing: The New Unit Testing

Interest in microservices (the blue line on the graph below) has been rising rapidly since 2015. In fact, it seems to be on the exact opposite path from software testing (red line).

This rise is hardly surprising. From the architecture perspective, there are significant benefits to microservices. Initial forays into service-oriented architectures (SOA) were a huge step forward in enabling reuse and flexibility. Business logic formerly implemented in monolithic applications was freed and released as isolated independent services, which were loosely-coupled and highly-distributed.

Microservices take the decoupling and granularity to new levels. Business logic is now implemented in the smallest, most discrete units possible. Functionality that would have once been a single service is now distributed across tens, or potentially hundreds, of microservices, each of which can be independently developed and deployed.

In addition to enabling unprecedented agility and scalability, this extreme decoupling opens up promising new options for testing. With the initial waves of services and APIs, API tests had little choice but to pass through a fairly broad set of business logic, which was likely implemented through multiple units of code (e.g., functions).

If an API test failed, there was no clear pointer to what specific code function needed to be fixed. That required unit testing and/or additional debugging.

Now, with each microservice implementing an extremely focused unit of business logic, you know exactly what code is broken when an API test fails. Why spend the time and effort on unit testing if you’re already getting the benefits from something that’s considerably less painful?

You might be thinking, “Testing hundreds of independently-evolving microservices is just going to present a different pain.” Correct. Each integration point is a potential point of failure, and testing all the possible integrations is a logistical nightmare—so don’t. Instead of testing how each microservice interacts with all the other microservices it touches, perform contract testing instead.

Very basically, contract testing helps both microservice consumers and providers ensure that they’re meeting expectations in terms of message format and data. The microservice consumer adds a contract that specifies expected request and response pairs for its various interactions with other microservices. Each contract is used to create a virtual service that the consumer can then test against. If the consumer’s changes result in incorrect/unacceptable request messages, they’ll be alerted immediately by a contract failure message triggered from the virtual services.

On the flip side, the microservice provider can use the same consumer-specified contracts to ensure that their own changes don’t break the functionality which consumers depend on. If the actual provider response suddenly fails to meet consumer expectations, they, too, get a rather clear and telling contract failure message.

The beauty of this approach—even beyond the fact that neither consumer nor provider ever has to stand up the whole spider web of interconnected and independently-evolving microservices—is that you get very specific unit-level failure information without burdening developers. As soon as testing moves away from the code level to the API level, it doesn’t need developers doing it. You certainly don’t need developers to deal with the maintenance, and you really don’t even need them to set up the test. Instead, the task can be delegated to people who view testing—not development—as their profession. We call them testers.

Looking Beyond Unit Testing Is Essential for Ensuring a Positive User Experience

Nevertheless, hyperfocusing solely on unit testing of any flavor can leave you blind to many critical issues. For example: Does every step of the end-to-end transaction work as expected—from whatever mobile or browser interface the user happens to have, through the backend (which might interact with custom/legacy applications, packaged applications, mainframes, databases, third-party services, etc.) and back to the user? Will these same transactions perform acceptably under a sudden increase in demand? If the user exercises the application in ways that the developers didn’t anticipate, will the application respond in a reasonable manner, or at least fail gracefully?

The answers to such questions lie beyond the horizon of the testing performed by the typical Agile team, which rarely extends beyond unit testing and a little integration testing.

However, as organizations invest millions in digital transformation initiatives, this doesn’t cut it. Maximizing that investment requires a flawless user experience from start to finish, 24/7. Professional testers thrive on exposing the glitches that developers tend to overlook but that would cause a CIO to cringe.

I’m confident that software testing will surge in importance in the 2020s if we commit to expanding its scope and effectiveness, which will ultimately elevate its business value. As the pendulum swings back, testers can’t expect to just hang on for the ride. Willingness to jump on the momentum of this decade’s trends—for example, stepping in to solidify the foundation of the Agile test pyramid and embracing new practices such as microservice contract testing—will be essential for navigating the turnaround.

April 24, 2020
SauceCon 2020 Speaker Profile: Nikolay Advolodkin

This is the fifth in a series of articles highlighting speakers at SauceCon 2020. SauceCon brings together the global community of Sauce Labs users and continuous testing experts to learn from each other and level up their automated testing and continuous delivery skills. This year’s event will be held April 27-29 in Austin, Texas, and devopsy.kinsta.cloud will be there, broadcasting live.

SauceCon speaker and automation expert Nikolay Advolodkin says that test automation is insanely hard, and that there are thousands of things we can do to make it really good—or really bad.

“Learning best practices is a great way to create a solid foundation for test automation success,” said Nikolay. “I want to share all of the things that I’ve discovered through my career to make everyone successful at test automation.”

You’d be hard-pressed to find someone more qualified than Nikolay to do that. As a senior solution architect at Sauce Labs and founder of UltimateQA.com, Nikolay educates customers and testing professionals worldwide on testing. Most recently, he was a contributing author to Continuous Testing for DevOps Professionals.

He was named one of 33 Test Automation Leaders to follow in 2020 by TechBeacon.com. And according to Udemy.com, he is one of the top Selenium WebDriver instructors across the globe, educating more than 50,000 students on the ins and outs of test automation from more than 120 different countries.

At SauceCon, Nikolay will host a full-day “Best Practices in Automated and Parallel Testing” workshop that takes attendees on a journey through a really poor test that follows multiple anti-patterns. Along the way, Nikolay will help attendees transform this test into a production-ready suite of tests that follows automation best practices: making them extremely robust, fast, atomic and ideal for parallelization. He plans to make this workshop session fun with games that will push the limits of test automation beliefs.

When he isn’t teaching people how to be automated software testing masters or revolutionizing the test automation world as we know it, Nikolay is into physical fitness. He said it helps him stay calm and rock in test automation. He also loves animals and the environment and supports both every day with his plant-based lifestyle.

You can hear from Nikolay and more than 30 other expert speakers at SauceCon 2020, the fourth annual Sauce Labs user conference. To see a full list of presentations or to register, visit the SauceCon website.

— Rebecca Cramer

March 9, 2020
SauceCon 2020 Speaker Profile: Hugh McCamphill

This is the fourth in a series of articles highlighting speakers at SauceCon 2020. SauceCon brings together the global community of Sauce Labs users and continuous testing experts to learn from each other and level up their automated testing and continuous delivery skills. This year’s event will be held April 27-29 in Austin, Texas, and devopsy.kinsta.cloud will be there, broadcasting live.

As test engineering manager at ShopKeep, a point of sale solution provider, Hugh McCamphill leads the company’s internal tools and test improvement team and works to further the test and quality focus of the company as a whole.

Every year, thousands of small businesses rely on ShopKeep to track inventory and sales, and ring up millions of customers. Because of that demand, testing is critical.

With over 15 years of testing experience, Hugh is up to the task. He started out on an agile team, working closely with developers doing testing and writing end-to-end UI tests.

“People have always struggled with how long end-to-end tests take to run,” said Hugh. “It’s become less of a problem in the desktop space, but tests created in Appium can be slower and harder to distribute.”

That’s what drove Hugh to his presentation topic for SauceCon 2020: “Not Just Accessibility IDs: Speed Up Your Appium Tests for Faster Feedback.”

“I wanted to share what we have done with improving the speed of our own Appium tests and suites so that we get more feedback sooner. We no longer rely solely on running tests overnight, we have Appium tests running on every pull request to the app,” he said.

During his presentation, Hugh will help SauceCon attendees better understand the time-consuming aspects of a test and demonstrate some options to reduce test time when interacting and asserting against elements. He will also explain why it’s important to understand the abstractions that you are using (as they might be slowing you down), why to be careful when querying multiple elements and show some of the capabilities that exist to speed up Appium tests.

Throughout his career, Hugh has developed deep expertise in test automation across a range of languages, APIs and tools, and loves sharing this knowledge with others at speaking engagements. He has also been the organizer of the Belfast Selenium Meetup for the last several years.

When he’s not talking about testing, he loves traveling with his family and two young children, playing squash and watching football—or what we in the U.S. would call soccer.

You can hear from Hugh and more than 30 other expert speakers at SauceCon 2020, the fourth annual Sauce Labs user conference. To see a full list of presentations or to register, visit the SauceCon website.

— Rebecca Cramer

February 21, 2020
Leadership Role in Adding Test Automation to Existing Environments

Leadership is the foundation for all agility. I’ve written elsewhere extensively on a model that works well for the new environment in which we find ourselves: the need for tens or hundreds of very smart people, doing things that have never been done before with new tools, working with people they may not be familiar with. These creative, expert people make dozens of decisions a day that few others see in real-time, turning ideas into code that must run flawlessly and execute complex business processes. Quite a leadership challenge.

The leadership model I’ve articulated is simple. Leaders at all levels should focus on rigor (good decisions based on options and data), alignment (everyone rowing in same direction) and efficiency (respecting everyone’s valuable time). Getting the right people, with the right skills and the right kinds of interactions, is more valuable than having the right process and tools.

For software agility, we know reducing the friction between development and productive use is a powerful leverage point. This general area we now call DevOps. Coming from the application space as I do, my primary focus has been on test automation. How can we at very low cost continually ensure that our evolving software is ready to be deployed to production (or production staging, if we are doing periodic bundled software deploys)?

I’ve recently watched a team introduce test automation into an existing application environment. Great leadership was demonstrated throughout by multiple team members. Let’s examine the steps taken and point out the leadership that made it happen.

Set the Target

Team and organizational leaders laid out a clear vision of the business benefits and their support for the initiative. This was the culmination of several months of learning, communication and cost/benefit analysis. Leaders gained alignment based on rigorous analysis.

Ensure Adequate Leadership and Technical Skills Are in Place

In the run up to the inception of the work, leaders hired an experienced testing leader, who in turn hired a couple engineers to do the work. Because the environment was a common one in the industry (Salesforce was a big part of it), the leaders sought the best expertise they could find, and wound up engaging an external specialist firm to get them going. People over process.

Define the Technical Architecture

The new team leveraged the knowledge of the consultants and some of their engineering partners to examine options and select an approach. They shared the options and analysis with their team and leaders and gained consensus and understanding. In the process the approach was fine-tuned. The team demonstrated rigor and efficiency and gained alignment. One key of their success was the “towering technical expertise,” a lean product development term, brought by the consultants.

Deliver Incrementally

For this item, I will describe the solution chosen, since the specifics will be of interest to this audience. The vision was to train the product owner/systems analysts to write acceptance criteria for each user story (which was the unit of development here) in the stylized language Gherkin. As the development was done during a sprint period, the test team would take the Gherkin and write automated tests for the test tool Cucumber, which would drive execution in Selenium. The execution would be automatically connected to the deployment of new code using Jenkins.

The team wanted to avoid disrupting the existing flow until the risk was reduced based on proven results. Accordingly, they began by building automation for existing software, proving that the vision worked in practice, and gradually completed the regression suite and started to catch up to work in process that hadn’t been released. Excellent leadership for agility was demonstrated again, driving rigor through proof of the concept and laying the groundwork to get the rest of the team aligned.

Get the Rest of the Team to Change

Once the test team caught up to being able to do in-sprint automation, they needed to tackle the training and acceptance of a new way of writing acceptance tests (in Gherkin). The great results they demonstrated helped influence the product owners and systems analysts to give it a try.

Continuously Improve

As the creative period of designing the test automation approach and getting into production came to an end, a new period of measurement and continuous improvement began. Here we have another foundational element of leadership for agility: understanding where we have repeatable flows where process really is the key, and where we have a creative endeavor in which the focus on people and interactions must rule. In a sense, the team transitioned from a product development effort to a manufacturing-like flow, and they accordingly changed their approach.

Conclusion

Sometimes we make the mistake of thinking that getting to efficient DevOps is all about tools and processes. My experience is that to the contrary, leadership is the key. As the first value of the 2001 Manifesto for Agile Software Development states, “People and interactions over process and tools.”

— Michael K. Levine

January 3, 2020