devopsy.kinsta.cloud

Tag: Automated testing

TCoE as a Way to Level Up Test Automation

A testing center of excellence (TCoE) can help you leverage automated testing efforts.

The “World Quality Report 2018-19” claims that test automation is almost a heal-all concept in QA and one of the most booming trends for the near future. Still, the general percentage of automated testing in overall test efforts is quite modest. Apart from technical and financial impediments, test automation can be hampered by a number of organizational challenges. Here, testing centers of excellence (TCoEs) may be of practical value.

Let’s find out what TCoE is, why test automation is a must and how TCoE may come in handy on your way to higher test automation levels.

TCoE: Capsule Review

A TCoE provides QA optimization and control due to consolidated testing best practices, tools, standardized methodologies and testing metrics. It aims to centralize QA resources dispersed around the whole enterprise, thus being more relevant for large-scale organizations with a number of departments or even subsidiaries. Capgemini research results prove that TCoE can bring such tangible benefits as optimized QA resources (people, budgets and tools), reduced testing time (by 30 percent) and costs (by 35 percent in a three-year time frame) and increased test automation rate (up to 50 percent).

Why Level Up Test Automation

Test automation is not just hype, but a proven way to get more from your testing. The ever-growing popularity of Agile and DevOps/continuous delivery projects necessitates that testing be performed within a shorter time frame. Test automation tools are highly efficient in performing the most time-consuming types of testing and the ones comprising large amounts of data, such as regression, performance, data warehouse or load testing. With its ability to execute thousands of test cases during every run, test automation provides coverage unachievable with manual testing resources.

Test Automation Challenges That TCoE Can Solve

The promotion of test automation is acute as it can result in tangible benefits. For now, the average level of automated test activities is only around 16 percent, as reported by the leading CIOs from around the world. However, it is pointless to automate all testing activities. Test automation is not a panacea for every testing project or testing type. Each project’s automated testing efforts require a careful consideration and ROI assessment, as test automation has a number of limitations. For example, it can save the testing budget only in the long run. So, there’s no need to introduce automation for small-scale testing projects. Besides, even in projects with a large testing scope, test cases that are unique for each release or source code modification, rarely repeated test cases and those with difficult-to-measure or not purely objective output shouldn’t be automated.

However, the low rate of test automation comes not only from its natural technical impediments but also because of the organizational problems bound. Luckily, TCoE can handle some of test automation organizational challenges.

Test Strategy Lacking Test Automation

If the test strategy doesn’t fully correspond to software requirements and its peculiarities, it may lack testing types appropriate for automation. TCoE may be of help here as TCoE managers can assist test leads in building a more effective test strategy with respect to high-level QA priorities and help BAs to come up with multilevel software requirements, if a software requirements specification turns out to be incomplete.

Besides, the initial part of test automation requires a significant share of testing time (choosing the repeatable test cases and scenarios, scripting and maintaining them) and budget. Without TCoE, some test leads may try to economize and strike automated testing off the test strategy, sure that the moves will help them meet the rigid deadline and budget. If it’s truly impossible to implement relevant test automation and ensure acceptable testing quality stays within time and budget, TCoE managers can suggest revising the testing time and financial limits for the project.. The alternative, of course, often results in even more expenditures because of the release of heavily bugged software.

The Lack of Skilled Test Automation Engineers

QA teams consist mostly of manual test engineers without code writing skills needed to script and maintain automated test scenarios, while developers don’t have the knowledge of relevant automation testing tools or the experience in creating appropriate test cases. So, neither manual test engineers nor developers can fill the gap without sufficient training.

TCoE can help, with its managers knowing each test engineer and having an established knowledge-transfer process. TCoE managers can find test team members (preferably, manual test engineers) who are enthusiastic about test automation and provide training to them on a broadly applicable programming language (for example, Java or Python) and the most adjustable test automation tools (Selenium, Ranorex, TestComplete, etc.).

Silo Testing Processes

Uncoordinated and varied testing processes can hinder test automation. In different QA teams, test leads may choose disparate test management tools and test automation engineers are likely to use diverse test automation frameworks. Besides, with QA resources being dispersed around the company, it’s complicated to create a common base of automated scripts that can be reused in different testing projects.

Efficient test automation requires unified testing processes around the enterprise, which can be attained with TCoE. From the earliest stages of its maturity, TCoE focuses on aggregation, analysis and standardization of testing processes and test automation tools and frameworks. In addition, the TCoE team captures test patterns, creates modular automated test scripts and gathers them in a library to be fully or partially used through the same testing types or similar testing projects.

Excessive Test Automation Spend

Test automation may be pricey due to automation software license costs. Software license purchase for almost every testing project anew can turn test automation into a nightmare for the finance department and lead to excessive testing costs. TCoE handles this challenge by providing conventional licensed test automation software tools that can be used for a number of projects.

Also, test automation requires ongoing engineering resources for maintenance. Application under test (AUT) changes from iteration to iteration and all the automated scenarios should be checked for their validity. Only a few of them can remain unchanged while the majority needs restructuring and correction. A number of TCoE’s qualified test engineers ready to cater to continuous automated testing can be relocated to the project where they are most needed, not being bound to a distinct one. Besides, common test cases and scripts that were automated once can be reused through different projects.

Wrong Time for Test Automation

Another test automation pitfall is the difficulty of choosing the right time to introduce it during a development life cycle. If a test strategy implies a too-early start with scripting test cases, a test automation engineer risks finding the prepared test scripts and suites outdated in the next iteration due to requirements changes. At the same time, starting with test automation too late may result in excessive testing time and costs.

TCoE consolidates test automation best practices and QA professionals with the deep knowledge of software and its requirements. Thus, it’s much easier for TCoE managers than for each test lead separately to evaluate a certain project and decide when it’s the safest time to develop and implement automated tests.

On a Final Note

The relevance of test automation is here and is predicted to gradually grow in the future. Still, automated testing has a number of constraints and trying to reach higher levels of its implementation is bound to organizational challenges. TCoE can assist in solving such organizational test automation problems as an unbalanced test strategy, the lack of test automation engineers, uncoordinated testing processes, excessive test automation costs and inappropriate time for test automation.

— Andrei Mikhailau

February 5, 2019
How Automated Testing Can Help the Latest C-Level Issue: Quality Software
Every year developers gather in San Jose for Apple’s annual WWDC, where software quality is typically a key point of discussion. This year, the company progressed from its 2017 buggy software debacle in full force—it’s a heads-on effort, marked by the industry leader naming 2018 its year of software quality.

This raises some larger questions around the software industry itself. Has modern IT become so complex that the concept of “bug-free” software is no longer realistic, particularly for a company the size and scale of Apple? Or did Apple recognize this pain point and shift its focus to software quality in an effort to be the harbinger of change?

Of course Apple isn’t alone when it comes to the struggle with quality. Buggy software from organizations across the board persists today, but it simply cannot be ignored. Poor quality software creates long-term effects of waning customer confidence, brand erosion and revenue loss. These are very real concerns for the entire C-suite. Quality is no longer an issue limited to Dev or Ops—it’s critical to every organization’s success.

For example, consider the fact that 88 percent of consumers say they’d abandon an app if they encountered bugs or glitches (source: QualiTest). Nearly half those people are likely to discontinue using an app entirely if they experienced a bug every day. And 32 percent say they would abandon an app the moment they encountered a glitch.

The cost of acquiring a new customer is 5 to 25 times higher than retaining an existing one. Getting software right the first time through a culture of continuous testing is critical to every business, as it translates directly to retaining customers.

Budgets for QA and testing are expected to rise to 40 percent in 2019 from 31 percent this year. However, one-third of developers still admit they don’t have enough time to properly test before software deployments. As everyone in this business knows, developers run fast. It is of the utmost importance to ensure software actually works well and will provide the expected customer experience. The garbage in, garbage out model is not an option. To fill the gap, more and more organizations are turning to automated testing.

Any repetitive, labor-intensive task is typically a good candidate for automation, including those associated with testing. Automated testing involves execution of prescripted tests on a web or mobile app, with test data accessible in detailed reports to help development teams quickly identify issues in the code, as well as any issues in the automated scripts themselves.

Performance of the automation framework and its scripts is crucial, and should be a high area of focus. Prescripted tests fall into different categories that comprise the test pyramid. They include:
1. Unit Testing
2. Code Coverage
3. Code Syntax Checks
4. Security Tests and Checks in Code
5. Integration Tests
6. UI Tests for Web, Hybrid and Mobile Apps – on multiple cross browsers on various platforms and OS versions
Once created, automated tests can be run again and again at no additional cost—which is particularly helpful when source code is changed. Since it is much faster compared to manual testing, automated testing translates into considerable time and cost savings.

Automated testing is especially valuable given the breadth of browsers, operating systems and devices in the marketplace. Consumers today run apps on a countless number of these, so developers must have access to every combination imaginable to deliver the highest level of performance. What good is your app if it doesn’t run (and, by extension, monetize) properly when it gains popularity in some part of the world with users on a different platform from the one you tested on?

Ensuring that software works on as many options as possible allows a business to reach a maximum number of users. This is where automated testing can help, by executing thousands of different complex test cases during every test run. Testing efficiency can be achieved by executing tests in parallel or grid, and executing CI/CD jobs in parallel. For example, one code is deployed in staging, teams can simultaneously start jobs to test against browsers and cross-browsers, apps and localizations, etc. The result is attaining coverage that would be impossible through manual tests. Teams can scale apps fast, maximize CI/CD workflows, and ship higher quality software faster.

Attaining CI/CD and automating all the tasks involved in creating a real-time test system is one of the arenas in which the next big software battle will be fought. More importantly, as software continues to eat the world, the C-Suite can no longer ignore the impact of poor quality software devouring your business. Apple’s bold step of identifying 2018 as the year of software quality is a call for other organizations to do the same.

Automated testing is a key piece of the solution. How is your organization meeting the software quality challenge, and by extension, the evolving standards of customer experience?
— Ritesh Arora
July 6, 2018
Continuous Testing and Business Assurance

As development moves ever faster, continuous testing can help further reduce the risk of introducing errors

In IT circles, “risk management” implies information security. But in the business as a whole, risk management is a function that spans the organization. The risk that an application will negatively impact corporate image, sales or employee morale is very real in the age of automating everything.

Once properly identified, these types of risks can be mitigated with continuous testing processes. The ability to set up a test to prove the login system works as expected keeps users able to get to the application.

Increasing use of agile and DevOps offers the opportunity to receive rapid feedback. Among other things, that includes business (as opposed to InfoSec) risk management. Requirements from the business side can clearly delineate items that create risk for the business unit and the requirements to reach desired outcomes. Continuous testing can turn those requirements into tests that validate the application is not contributing to business risks.

After all, there are enough risks in the business environment: competition, suppliers, coordination of marketing, legal, etc. Making application development less of an issue is good news for business owners and IT alike.

By translating business risk management requirements into automated tests early in the development process, stability and risk aversion are built into the application from (or nearly from) inception. This offers stability and provable correctness in the application development cycle.

As the pace of development continues to accelerate, there is a risk of negatively impacting the user experience. Whether that negative impact be through failures of critical parts of the system or degradation of performance, it harms the application and the business it supports. Continuous testing can help alleviate fears by testing for these scenarios and pointing developers at fixing them early.

Test automation tools are becoming more prevalent, and the next step is to include business requirements designed to mitigate business risks at every step in the process. It is simple to say and test “valid users must be able to log in,” but as development progresses, if the login prompt takes more than 10 seconds to return, that adds the “in a timely manner” business requirement that is then refined by IT to a number, and a new test is run. Only by having the inputs from the business at each stage can such refinements be identified and implemented.

Interested in this topic? Check out the ebook here to hear from experts on balancing speed of development with risks to the business.

And consider it. The purpose of any given piece of software is to serve the business. Speed that introduces unwarranted risk to the business cycle is not the best answer. Continuous testing, and attendant business owner involvement, can mitigate those risks.

— Don Macvittie

May 30, 2018
Moving from ‘Tester’ to ‘Quality Enabler’
Speed versus quality. Can you really have one while the other exists?

When Agile came on to the scene, it pushed to get development teams to create code as quickly as possible. Then DevOps came on as a practice that helped these Dev teams and the Ops side of the house to interact and collaborate more on best practices, tools to use and pinpointing issues in operations.

But the testing practices and QA teams have had challenges keeping up with all this code that is getting thrown their way at an avalanche pace. That is probably why nearly two-thirds of self-proclaimed DevOps practitioners report that their biggest bottleneck occurred in the QA/Test phase. In fact, this cited bottleneck was more than 2 times greater than the No. 2 proclaimed bottleneck, planning (29 percent).

It’s been said that “the consumer appetite to engage via digital channels is matched only by intolerance for any defects that might have an impact on their user experience and business outcomes.” If organizations are going to speed their delivery pipeline, they need to remember that the success of continuous delivery hinges on the effective employment of continuous testing.

Part of the problem remains with manual testing practices. You can’t manually test millions of lines of code, then try to figure out the customer experience during integration testing with other internal or third-party systems and API, with hundreds of microservices or varying mobile devices. The complexities of today’s composite apps make it impossible.

And traditional, legacy testing tools don’t help, either. Gone are the days of waiting for the UI to be complete before QA teams begin their testing. And gone are the days of developers throwing code over the wall to the testing team and washing their hands of it.

A few other facts:
1. Ninety-one percent of QA professionals agree that the role of testing must change to meet the demands of today’s agile development needs.
2. Only 1 in 3 organizations involve QA in planning requirements.
3. Fewer than 1 in 5 organizations write new tests each time new code is checked in.
The idea of continuous testing helps to solve that. Continuous testing is the practice of testing across every activity in the software development life cycle (SDLC) to uncover and fix unexpected behaviors as soon as they are injected. That can’t happen in the above environments.

A new ebook from CA Technologies and devopsy.kinsta.cloud notes that to continuously deliver software, organizations need to make sure tests, procedures and processes around quality are designed to fit in line with the flow of development, integration and deployment. While continuous testing is dependent upon strategic automation, it is not synonymous with automated testing—a continuous testing process encompasses other important QA activities that impact overall product quality.

As such, continuous testing depends on four major pillars:
1. Code Quality – Did developers create the application code correctly?
2. Pipeline Automation – Can the application code work across environments without manual intervention?
3. Application Quality – Did developers create the correct features?
4. Customer Experience – Are users perceiving value in the application delivered?
These four pillars are discussed in more detail in this continuous testing ebook. You’ll learn the evolutionary journey you can take to shift testing practices left, and really make testing and quality everyone’s responsibility. You’ll even read how GM Financial was able to transform its load practices by implementing some of these practices. See how you can start using these pillars in your QA efforts.

Now is the time to elevate QA’s new role from that of tester to quality enabler.

— Scott Edwards
July 27, 2017
SauceCon Speaker Profile: Shivani Sharma, Slack

Just Say No to Flaky Tests: Releasing Code with Confidence

Shivani Sharma doesn’t like anything flaky—unless it’s baked goods.

Shivani is senior engineering manager at Slack, working on test automation for Slack’s iOS, Android and web applications. An avid baker, Shivani thinks flakiness should stay in the kitchen.

Although “flaky” is a common adjective describing automated tests, the Slack test automation team wanted to be better and create value by delivering deterministic results. The team’s mission is “to make our software development process simpler, more productive and our product more pleasant to use.”

Worthy goals, to be sure. To that end, Shivani will present on, “Releasing Code with Confidence,” at the upcoming SauceCon conference in San Francisco June 6-8. Her presentation will discuss how Slack’s automation team approached the common problem of flaky tests to build a reliable feedback system into their continuous integration pipeline for code hundreds of developers write and release every day. This talk will provide actionable takeaways that can be used in other organizations, too.

In addition to baking and “crème brûlée-ing,” as she describes it, Shivani is an avid runner and has run eight half-marathons. To give her knees a break, she’s excited to participate in her first Olympic distance triathlon this summer. She has spent 10 years working in Silicon Valley, holding positions at Google, BigFix and IBM, as well as Slack. Shivani holds a BS in Electrical Engineering from UCLA.

SauceCon, the first-ever Sauce Labs user conference, is a three-day event filled with training, workshops, best practices and visionary content from the leading minds in automated testing. For more on upcoming SauceCon presentations or to register, visit the SauceCon website.

About the Author / Rebecca Cramer

Rebecca Cramer is a writer and communications strategist specializing in technology. With more than 20 years of experience across disciplines including marketing, internal and executive communications at companies such as Dell and Quest Software, Rebecca is passionate about creating and sharing good content of all kinds. In her spare time, she enjoys spending time at the beach and hiking in her native Southern California.

June 1, 2017
Testing What’s Important: Keep Growing Your Test Base

This project has gone on soooo long…

In most human endeavors that take a stretch of time to complete, the idea is latched onto and people become excited about being involved. It doesn’t matter if it is building a skyscraper, inventorying a massive warehouse or even going to war. People are intrigued by the possibilities and want to get started doing the thing. After a time, though, this enthusiasm wanes, and most often is replaced by a weary determination to get the thing done. The nature of the endeavor doesn’t have to change; the reality of its implementation just becomes the dominant factor.

IT projects in general—and continuous integration (CI) projects in particular—are no different. Whether a new project or implementing comprehensive CI on an existing project, the beginning is the most exciting part. Soon the excitement wanes and achieving the desired level of test coverage and a consistent build success rate becomes the goal that is trudged toward. It is a rare team that reaches for 100 percent code coverage with testing on initial implementation. The key becomes “testing what’s important.” And that varies by team and project.

But testing at all levels—unit, integration, smoke, system, regression—could be better on almost every project. That improvement can be done incrementally while running through sprints. While some of the testing is managed by QA, some is managed by the developers—and in a CI/CD environment, arguably more is managed by developers. They are more involved in the testing process in CI/CD, so are more likely to be more involved in test development/management.

The point is, good enough coverage is just that. And there is a hidden cost in getting too carried away with testing every line of code for every possible input, but improving tests is more than simply cranking out more permutations. Improving existing tests, or writing tests for functionality currently poorly tested, is increasing testing.

So, as you strive to make your code get better, strive to make your testing get better, too. Testing after the baseline is established is one of the first things to get minimum attention, even in automated test environments. But improving it reduces the risk of releases, and that’s a benefit to the team and to the company. So it’s worth staying up on.

Unless you have my friend, who we will only call Pete, on your staff. Then, no matter the quality of your testing, no matter the number of tools and layers of testing, he’ll find a way to break your product within 5 minutes. That made him a good tester, made him a better developer and, now, a better architect. But most of us don’t have Pete, so work on those tests.

— Don Macvittie

June 1, 2017
SauceCon Profile: Greg Sypolt, Gannett

Using Design Patterns to Improve Automated Testing

Test automation architect Greg Sypolt of Gannett | USA Today Network fell into a testing career by accident. As an on-call software engineer who spent many late nights early in his career discovering that many bugs were caused by a lack of test coverage, he developed a passion for optimizing software and building quality into every stage of the continuous integration pipeline.

At the upcoming SauceCon conference, Greg will present on, “Using Design Patterns to Improve Automated Testing.” Attendees will learn how to transform their new or existing automated tests to speed development and reduce maintenance impact.

A self-proclaimed technology geek, Greg is passionate about a few things: DIY projects (he recently finished his basement single-handedly), John Denver songs (one in particular, see below), outdoor sports and speeding up development with design patterns. In Greg’s view, a design pattern is a general reusable solution to reduce commonly occurring problems in software design. A design pattern is not a finished design that can be transformed directly into code. It is a description or template on how to solve a problem that can be used in many different situations.

At Gannett | USA Today Network, Greg is responsible for leading the test automation architecture team to build and maintain testing solutions, evaluate test coverage, and champion continuous integration across 40+ Gannett products. The test automation organization at Gannett | USA Today Network has built a platform that enables teams to move at their desired pace through a balance of autonomy and collaboration. Their goal is to ensure every team is successful when using the self-service model. The vision is clear for test automation: they believe it starts with everyone on the team owning quality. Making this shift requires daily collaboration and planning to build quality into every stage of the pipeline.

Greg grew up in a small town in West Virginia (Take Me Home, Country Roads!) and now calls Northern Virginia home. He is a husband, father of three kids and an avid do-it-yourselfer. Greg enjoys presenting at conferences and blogging about automated testing strategies, testing frameworks, standards and continuous integration.

Join Greg and other presenters at SauceCon, the first-ever Sauce Labs user conference. SauceCon is a three-day event filled with training, workshops, best practices, and visionary content from the leading minds in automated testing. For more on upcoming SauceCon presentations or to register, visit the SauceCon website.

About the Author / Rebecca Cramer

Rebecca Cramer is a writer and communications strategist specializing in technology. With more than 20 years of experience across disciplines including marketing, internal and executive communications at companies such as Dell and Quest Software, Rebecca is passionate about creating and sharing good content of all kinds. In her spare time, she enjoys spending time at the beach and hiking in her native Southern California.

May 25, 2017
SauceCon Speaker Profile: Angie Jones, LexisNexis

Automated tests are an important part of continuous integration, but if they’re not reliable, they can cause more problems than they solve.

This problem is not new to Angie Jones, who works as a consultant automation engineer advising scrum teams on automation strategies and developing automation frameworks for countless software products.

Jones currently works for LexisNexis, an information provider for legal and regulatory businesses. LexisNexis recently began embracing agile and test automation companywide, and she was brought aboard to help the company master the change. She is consulting with multiple teams to help define their automation strategy, building automation frameworks and training their testers on automation techniques.

The biggest challenge so far has been finding different implementation strategies for different teams. “Automation strategies are not one-size-fits-all and should be customized for a given context,” she says.

Jones will bring her expertise to the upcoming SauceCon event and plans to present a talk titled, “The Build That Cried Broken: Building Trust in Your Continuous Integration Tests.” Her presentation will focus on automated tests that are part of a continuous integration process. These tests aren’t run privately on someone’s local machine, and the person running them simply gives a final verdict to the team and spares everyone the glitches and hiccups. In CI, the unreliability of these tests are exposed for all to see, and they are detrimental to the process.

In her talk, Jones will share how her team managed to turn their CI builds from an untrustworthy mess to a credible source of information. Since this isn’t a quick fix with a lot of cultural and mindset changes, she will discuss that as well.

She is very excited to give this talk at SauceCon because “people attending this conference are the ones who are invested in ensuring that they get CI right, and I want them to know that automated tests don’t have to be their downfall. Besides that, it’s a really fun talk that I think everyone will enjoy,” she says.

As a Master Inventor, Jones is known for her innovative and out-of-the-box thinking style which has resulted in 22 patented inventions in the United States and China. Angie shares her wealth of knowledge by blogging at angiejones.tech, speaking and teaching internationally at software conferences, serving as an adjunct college professor of Computer Programming and teaching tech workshops to young girls through TechGirlz and Black Girls Code. When she is not inventing something or helping people with test automation, she is reading urban fiction or soul line dancing!

Join her and other presenters at SauceCon, the first-ever Sauce Labs user conference. SauceCon is a three-day event filled with training, workshops, best practices and visionary content from the leading minds in automated testing. For more on upcoming SauceCon presentations or to register, visit the SauceCon website.

About the Author / Elizabeth Hechtel

Elizabeth is a passionate tech writer at TestObject. As a Junior Marketing Manager she is focusing on delivering the best quality content on mobile testing. Personally, she loves learning new languages and reading up on recent advancement of modern technology.

May 4, 2017
Automation Killed QA – It’s Time to Deal With It

Quality assurance (QA) had a good run.

From the time it was considered an afterthought by organizations and software vendors alike to the era of massive outsourcing contracts, followed by significant vendor investments in related tooling, and, finally, its recognition as a critical success factor among high-performing organizations, quality has evolved to become even more important than functionality to end users.

Customer experience, today, is defined by speed and quality above all else. Namely, can I access the information and products that I want in the blink of an eye and get expected feedback based on input? As a result, customer experience has also shifted: It’s all about application experience, not the friendliness of a checkout cashier or sales/service rep.

With this massive shift toward customer experience platforms (think omni-channel) and increasingly lofty expectations, the operative question becomes, How are we enabling velocity in our testing practices to ensure the highest quality while keeping up?

We’ve tried everything from outsourcing to insourcing, investing in tooling, investing in people and even building testing architecture teams to oversee the automation, quality and consistency of testing results. Roughly seven years ago, my top priority for testing teams was to automate 30 percent of our tests. Today, if you are not 100 percent automated, or close to it, you are severely lagging in output.

But what’s next? What if I’ve automated all of my tests and my manual testers have been replaced with automation engineers, but I’m still having quality issues and being perceived as a bottleneck?

The answer is simple. Automate everything.

Makes sense, right? Isn’t everyone telling you that? But what are we defining as “everything” from a testing perspective? Therein lies the problem; we are thinking about testing as its own factory within the larger software development life cycle.

While we believe we are changing this perception by going agile and moving most QA functions into scrum teams, the reality is we still look at testing as a standalone function. Centralized or decentralized, this is the case, and we need to think differently.

Automating for quality begins with defining requirements. How am I ensuring that I’m covering all cases? How do I ensure I’m translating the business requirements properly? These questions, the time it takes to answer them and the risk of human miscalculation all impact quality and velocity.

We must start our automation for quality at the beginning, during establishment of business requirements.

Let me introduce you to next-generation QA. Intelligent “continuous testing” platforms are replacing humans for testing, automating tests, creating data and building test cases. These platforms are able to translate model-based business requirements into complete automation—test cases, test data and test script automation.

Orchestration across your software development life cycle creates your environments, lays down your test data, deploys your software, runs your test scripts and reports results. No more human error, no costly resources and guaranteed test coverage that ensures quality.

Integrating your existing automation around continuous integration, portfolio management reporting, and execution reporting to this framework now gives you the ability to report and automate your ideas all the way through to outcomes.

Early adopters are already doing this. QA as a functional organization is gone. If you aren’t already automating to this end, you are severely behind.

Questions? Comments? Perspective? Have at it in the comments below.

— Ron DeCanio

January 27, 2017
DevOps Chat: Steve Hazel, Sauce Labs

I recently had the chance to sit down and speak with Steve Hazel, CTO and co-founder of Sauce Labs. With so much being written and talked about the role of quality assurance (QA) and testing in a continuous integration/continuous delivery (CI/CD) world, it is good to hear from someone like Steve who is an actual thought leader in the space. Steve tells us what he thinks the limits of automated testing are, the future of QA and more. Have a listen for yourself to find out, or follow along with the printed transcript below!

Alan Shimel: Hi, everyone. Alan Shimel, devopsy.kinsta.cloud here, for another DevOps Chat. Today’s DevOps Chat’s guest—that was a tough one—is Steven Hazel, co-founder, CTO at Sauce Labs. Steven, welcome to DevOps chat.

Steve Hazel: Hi, Alan. Glad to be here.

Shimel: Thank you. Glad to have you here, actually. So Steven, there might be some people listening who maybe aren’t familiar with Sauce Labs. So, before we start, why don’t we, if you don’t mind, give our listeners a quick overview? Tell us about Sauce Labs.

Hazel: Yes. At Sauce Labs, we do automated testing for both web and mobile apps. We run a cloud service that gives our customers access to a wide range of platforms, whether that’s operating systems and browsers for web testing, or mobile devices and emulators and simulators for mobile testing. You can get access to all the different platforms you need to test on, and you can get access to a lot of platforms at once to run your tests in parallel to get through a large test suite rapidly, which is really what people need for today’s continuous integration and continuous delivery workflows.

Shimel: Got it. Of course, Steven, Sauce Labs was closely linked with around Selenium and some of the open-source testing tools that are out there today, right?

Hazel: That’s right. Selenium is the dominant web-testing technology. In mobile testing, there’s a variety of technologies, but we support Appium, which also uses the web driver protocol that underlies Selenium. We really think that fitting in with those open-source tools is the smart way to go. It fits with what people are doing.

It gives people a lot of choices about how they want to do their testing, what kind of languages and frameworks they want to use on their end. So, we just plug right into Selenium tests, if you’ve already got Selenium tests, or Appium tests. You can just change the config and point them at Sauce Labs, and they’ll run unmodified on our systems.

Shimel: Got it. Steven, we were talking before we started recording today, I was telling you about a recent article we had over on devopsy.kinsta.cloud. The gist of it was basically that DevOps was the death knell for QA. I told you I didn’t agree with that and you agreed with me. You said you didn’t agree with it, either. But for our audience—they’ll respect you a lot more than they respect me—why don’t you agree with that, Steven?

Hazel: Yeah. I think that to suggest that there’s no longer a role for manual QA when you take on a DevOps process is just not in line with the facts. If you look at what we see large organizations doing, they’re not dropping manual QA for automation. They’re changing the mixture of what they have manual testers doing by quite a lot, but they still have manual testing teams. You still need a manual testing team.

So, what I see happening, as you start to take on automation, your testing coverage improves dramatically because you can afford to do a lot more testing when you’re automating than was ever possible with a manual testing team. But automated testing, the classic example is exploratory testing. You’re just not gonna get the kind of creative, exploratory walk-through of your app with an automated test that a manual tester can do. I think there’s still a role for QA organizations, beyond just manual testing, in being the other side of the conversation, right?

A lot of people have the concern that if developers are writing tests, it’s the inmates running the asylum. The exact developers writing the software are the ones deciding whether it’s high quality, and you want another perspective in there. You want a perspective that’s maybe closer to your end user. So, I think that the other side, beyond just manual testing, that you see with QA teams continuing to have a role is in deciding what cases should be tested, whether you have good test coverage, encouraging good testing practices, even when you are automating your tests.

Shimel: Absolutely. Really, Steven, I guess the boogeyman in the room around the friction with DevOps and QA is automation. There’s some schools of thought that says, “Well, we can automate everything. The more we automate, the better we are.” There’s another school of thought that says, “Well, you can automate as much as you can and automation’s not a bad thing, but there are just gonna be some things that are not automatable, if that’s a word now, or may never be automated. Where do you go in on that continuum?

Hazel: Well, being someone who’s pushing forward automated testing, I want to see automation go as far as we possibly can. I think that that’s very far. That said, I think that the way the mixture of tests changes as you adopt automation is maybe a little bit counterintuitive. You might imagine, well, we’ll automate some percentage of our manual test cases, maybe 80 percent or 90 percent. Maybe we could get to 100 percent if you’re a zealot for automation.

But actually, what you see is, it’s not that you’re automating a percentage of what was there in manual testing. You’ll actually have more automated test cases than you were able to do with manual testing. So, if you just looked at the number of test cases, your automated cases are gonna way outnumber the original count of manual test cases just because you can, right? You can now cover a lot more edge cases, a lot more scenarios because of the power of test automation. So, that’s a real advantage to test automation.

Shimel: Sure.

Hazel: At the same time, certainly there are things, visual testing, there’s some interesting approaches out there, but I don’t think we’ve yet seen a way of making sure that the app looks right, especially as it changes. You really need a human being involved when there are deliberate visual changes to make sure that they came out right. I think it’s those things where automation really can’t give you an answer about, does it look right? Does it act in a way that’s gonna be acceptable to somebody?

I think you still need a human being in there at some point, checking those things, but it’s a different process. It’s no longer this phase gate before every change is released that you used to have. It’s now a process that can be made asynchronous from the bulk of your development and release process.

Shimel: Absolutely. If you don’t mind, Steven, a little bit of time on, you mentioned a little bit, but what do you think of the kinds of tests that lend themselves well to automation? Certainly, a visual test of an interface, different arts. What are the sweet spots for automated testing?

Hazel: I think certainly at the bottom of the testing pyramid, unit testing brings tremendous value. It’s something that’s very easy to automate in a way that’s completely deterministic, very fast. Gives you really quick feedback as a developer as you’re making changes to the code on whether you’ve broken something. But the coverage with unit testing, the coverage of functionality that may be important to your end users, it’s really hard to achieve. It’s really hard to achieve good, solid feedback from your unit tests that you haven’t broken anything anywhere.

Functional testing, the next layer up, which is what really we focus on, gives you that complete end-to-end check. You’re actually testing the features of your product and you’re making sure that end to end, with all the various components involved, they work properly. That’s more of a challenge for automation, but we’re seeing, I think, a majority of development organizations out there seeing success automating functional tests and real value from having that be something where you can run through hundreds of test cases in minutes and get feedback to your team. So, that works quite well for automation.

I think it’s that next level up, as you start to get into the types of tests where you’re looking at, what’s an acceptable experience for the end user? User acceptance testing as distinguished from functional testing, where you’re really starting to look for, does it look right? Does it act right? Is the response time throughout the entire process acceptable?

You can zoom in and measure a lot of these things very specifically, but getting the answer to, “Overall, though, was it a good experience?” That’s the kind of thing where it’s really tough to cover that with automation.

Shimel: Yeah, agreed. Steven, if you don’t mind, I’m an old security guy prior to DevOps. One of the things within the security community that we’ve really pushed around is the idea of automated security testing shifting left in a DevOps environment and allowing security testing to be done in an automated fashion at a CI/CD-type of level. Where do you see that in this whole discussion?

Hazel: Security testing isn’t what we focus on at Sauce Labs, so –

Shimel: Right, I know.

Hazel: I’m not an expert in this, but it is something that I follow and care a lot about. I think I see a lot going on with static analysis, with some of these tools that can help to constrain the types of mistakes that you could have in a program, type systems and things along those lines. I think really that’s very badly needed in making code secure. Some of the issues we’ve seen come up over the last few years, things like Heartbleed, the Glibc bug, really those kinds of things aren’t acceptable, aren’t something that we can continue to have happen, and I think the solution really is fully automated tools, like static analysis tools and type systems, that can lock down the code and just constrain the types of mistakes that could possibly happen.

Shimel: Sure. You know, Steven, I see it as very analogous to the QA situation, which is I think a lot of security people get very territorial around job security because they’re afraid somehow that doing automated security testing earlier in the process—shifting left, so to speak—is somehow gonna affect their livelihoods, or make them obsolete.

Hazel: I can see that. We see that among QA folks, for sure. To me, that’s a shortsighted view. If the world is getting more intense security problems, more intense quality problems, we’re shipping more software, you just can’t solve it 100 percent by throwing bodies at the problem. I think that what’s gonna happen is not that those people’s careers are in jeopardy; instead, as a society, we’re just gonna do more and better security and quality testing than we’ve been able to do in the past.

Shimel: Yeah, agreed. Steven, we’re running a little low on time, so let me get just one or two more questions. Sauce Labs has been, in my opinion anyway, for what it’s worth, somewhat of a DevOps darling in terms of really encompassing—you’re living the DevOps model. You’re working with open-source tools, testing and automation, enabling automation. Sauce Labs is a player in the DevOps ecosystem. Where do you see it going?

Hazel: Well, toward the enterprise is one big trend we’re seeing. Earlier on in the history of Sauce Labs, a lot of our customers were essentially software companies, companies that their primary focus is technology. Now, a lot of our customers are retailers, financial institutions—companies that you don’t think of as primarily a software company, but all of these companies have apps, they have complex web experiences. They’re really adopting DevOps in a big way. They’re moving to what is, for them, a very different process than what they’ve been doing and allows them to ship better software faster, with more complexity than they were able to handle before.

Shimel: Absolutely. Steven, we are low on time. I’m gonna ask you the final question, which I ask most all of my guests here on DevOps Chats. That is, if you had to recommend one book to our audience that they should read, a must-read for them, over the next six months or a year, what would that book be?

Hazel: Oh, wow. A book recommendation. I think the obvious choice in a conversation about DevOps is “The Phoenix Project.” If you haven’t read it, you’ve got to read it. It’s a quick, enjoyable read, but it really gets the concept across better than probably anything else is going to. It’s an obvious recommendation, but I still think it’s worth it.

Shimel: Sometimes the obvious is good. I will just add to that, if you already have read “The Phoenix Project,” Gene Kim’s next book, I think co-written with John Willis and Jez Humble and someone else that I’m blanking on – oh, darn it, the guy who started DevOps, who coined the phrase. Patrick. Patrick Debois. Their new book, I think it’s called “The DevOps Handbook,” will be out, I want to say, in the fall. Not a sequel to “The Phoenix Project,” but another good book to look at.

Anyway, Steven Hazel, CTO, co-founder of Sauce Labs, thanks for being this episode’s guest on DevOps Chat, and continued success with Sauce Labs.

Hazel: Thanks a bunch. Thanks for having me.

July 25, 2016