devopsy.kinsta.cloud

Tag: Quality Assurance

The Role of QA in the DevOps World

The need for faster time to market and superior quality is getting intense in the software development process. Software trends for 2017 suggest that DevOps will grow as a trend and become more widely accepted across enterprises. Organizations are reaping the benefits of DevOps approach and are incorporating the rules within every aspect of software testing. A case in point: DevSecOps—security testing integrated with DevOps—is an emerging concept and is being considered by enterprises to execute application security testing.

The underlying objectives of adopting DevOps are to make the testing and development process more collaborative, ensure software quality and to help launch the product faster to the market. This makes quality assurance (QA) a key component in the DevOps methodology. New-age applications need real-time changes and updates to meet the growing demands of the consumer. This requires continuous testing, continuous development and continuous delivery/deployment. DevOps enables teams to constantly upgrade and deliver.

In 2016 a number of enterprises adopted DevOps. According to industry experts, 2017 will see more defining moments for DevOps. The idea of accepting DevOps is to consider a cultural shift and not just a technical shift. As digital transformation expert Jeff Sussna says about DevOps, “The key to following the continuous delivery path is to continually question your own assumptions about what’s possible.” Currently, DevOps is on the route of being questioned and defined at every level. It will result in bringing more value to the methodology.

Every enterprise and its style of working differs. The way they would adopt DevOps will differ, too. But digital transformation and the need to be available for customers all the time with new updates is transforming the way enterprises adopt new practices and methodologies. Every enterprise has to evolve in a sustainable manner, so quality is at the core of every practice, including DevOps.

Why is QA Integral to DevOps?

In the DevOps scenario, QA integrates into the testing and development process and takes the collaborative approach. Quality is ensured throughout the testing and delivery cycle and both the testing and development teams are responsible for it. In the traditional pattern, quality creeps in toward the end, but with DevOps it comes in at every level.

In terms of code development, the QA team pushes the new development code into production and DevOps facilitates to ensure a seamless deployment of infrastructure and tools. Thus, it helps in releasing a code on a consistent basis and the responsibility to ensure quality is not just limited to killing the bug; it is built in the overall application release.

In the current scenario, hourly releases are required to meet the demands of the consumers and deal with competition. Web/mobile applications such as an ecommerce or financial services application need to be updated frequently. It is a key business requirement. DevOps helps to make this possible, but with QA involved, the quality of the release is ensured. Moreover, with DevOps, the reality check can happen in real time rather than taking the code offline and testing it.

On the whole, the popularity of DevOps over the years has intertwined quality into every engineer’s responsibility, which is a positive sign. This helps in addressing the needs of the market and ensure that customers do not suffer. QA further enables and directs teams to check the application for performance and quality before it is delivered.

The role of QA in the current software development process is changing, where the customer comes into focus and the needs of the customer are brought into perspective while delivering the application. Releasing new and functional builds at any point through the application’s life cycle can become a disaster if quality is not ensured constantly. Automation of tests plays a critical role to guarantee quality and bring speed.

Building a QA strategy with the necessary best practices is essential for organizations that are embracing the DevOps movement. This will help in delivering effective software development and operations to achieve user experience. DevOps practices are adopted to improve the regularity of the releases and reduce the defects. It is a methodology that boosts rapid, short, iterative development and deployment of applications.

So, the idea of incorporating QA is to ensure continuous improvement in the delivery process. DevOps helps digital enterprises to meet the needs of the competition and enable overall business growth. As a result, enterprises are able to streamline and pace up the delivery of applications with changes and innovation.

Best Practices for Incorporating QA in DevOps

The key aspect to consider while merging QA with DevOps is that the QA and testing teams must team up with the technical teams. In this way QA becomes an enabler for ensuring quality in the software development cycle. In addition, it is important to define tangible aspects of the quality that are expected from the project/application. This helps in achieving results within stipulated time frames.

Putting down the quality metrics is absolutely critical in DevOps, which helps teams to identify bugs early in the development cycle. Defining the metrics also enables teams to integrate them within the organizational culture at various levels. In this way, quality is ensured at every level and development teams also get enabled.

The idea of DevOps is to make the developer think in line with testers and constantly check the code for potential errors. On the other hand, a tester who holds the responsibility to test the application/software with various testing tools and methodologies also needs to fix the code. In this way, enterprises with an inadequate QA or testing strategy cannot ensure continuous testing and continuous delivery.

In Conclusion

The application scenario is changing and it is getting very complex. DevOps can speed up the development, but what can you expect without a robust QA strategy? QA is absolutely integral to the DevOps process. Continuous development and delivery is impossible without a comprehensive QA/testing strategy.

This blog has been drafted in collaboration with Cigniti Technologies, an independent software testing company. Click here to read more blogs on DevOps.

About the Author / Komal Lopez

Komal Lopez works with Cigniti Technologies and is instrumental in helping enterprises take better decisions related to Quality Assurance products, tools and services by leveraging research and content. She specializes in writing about technology trends, testing trends and has been in the Marketing and Communications industry for over a decade. Connect with her at komal.lopez@cigniti.com

July 28, 2017
From ‘Water-Scrum-Fall’ to DevSecOps
As organizations abandon the waterfall method of software development for Agile, many are stuck in what Hasan Yasar terms Water-Scrum-Fall. That is, the organization has not effectively embraced Agile and DevOps principles and remains in silos with no links to business goals. Enter DevOps, an extension of Agile thinking. While Agile embraces constant change and embeds the customer into the process, DevOps embraces constant testing and delivery and embeds operations into the team to internalize expertise on deployment and maintenance.

This is how Hasan started his talk, “Multi Security Checkpoints on DevOps Platform,” at the All Day DevOps conference.

In his talk, Hasan lays out a plan to get organizations to DevSecOps. Really, DevOps is a risk mitigation strategy, built on situational awareness, automation, and repetition. But, security is where a lot of DevOps implementations fall down. The goals for each organization should be:
- Protecting private user data
- Restricting access to data/systems
- Protecting company data/intellectual property
- Standards compliance
- Safeguarding disposition/transition
But, how do organizations get there?

First, integration and communication. Every point of the product development life cycle should be integrated and communicating, including among the tools. Once this is achieved, you can automate many, if not most, of the tasks. The automated steps are the ones that require less human actions/input to the software development process. This allows everyone to focus on innovation and better code and less on tasks that can be automated by autonomous systems. Also, tasks that can be automated are less susceptible to errors.

Of course, it is the team that ultimately designs, develops and delivers the software. Your team consists of development, IT operations, quality assurance and security. Each has its own skill set and focus, and the overlap is Secure DevOps.

The team is in place, processes are automated and development has started. Development in this day and age has evolved tremendously from even just a few years ago. Previously, software was limited to size, function and audience, and the supply chain was practically non-existent. Your team built each component. Now, development has grown beyond the ability of an organization to develop outside of its core competencies. The supply chain now involves many sources for the code. It is more like plug and play, and this creates lots of vulnerabilities.

Hasan notes the software supply chain risk factors:
- Supplier capability – Does the supplier follows practices that reduce supply chain risks?
- Product security – Is the delivered or updated product acceptably secure?
- Product distribution – Does the method of transmitting the product to the purchaser guard against tampering?
- Operational product control – Is the product used in a secure manner?
To reduce your supply chain risk, Hasan recommends:
- Ensure supplier security commitment;
- Evaluate a product’s threat resistance;
- Create a centralized private repository of vetted third-party components for all developers;
- Establish good product distribution practices; and
- Minimize variation of components to make things easier.
Finally, as you transition to DevSecOps, remember that security must be addressed without breaking the rapid delivery, continuous feedback model.

You can watch Hasan’s entire talk online here for more details and other tips.

If you missed any of the other 30-minute long presentations from All Day DevOps, they are easy to find and available free-of-charge here. Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here. This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed). It’s all free and online Oct. 24.

— Derek E. Weeks
July 27, 2017
DevOps Assessment: Assess, Analyze, Implement, Achieve!
DevOps has taken hold in IT. A growing number of organizations are adopting DevOps to deliver applications and services at higher speed; evolve and improve products at a faster pace than in a traditional software development and infrastructure management environment; and better serve their customers. After realizing the benefits of development and operations teams working together instead of in siloed environments, organizations are fast embracing the idea of DevOps.

To move forward in the DevOps journey, companies first should outline, identify and map the areas where they need improvement. Having a clear idea of maturity of your current IT infrastructure will help you greatly in that process.

What is a DevOps Assessment?

Adopting DevOps practices is not a one-day process. And, if it’s not done correctly, companies can do more harm than benefit. So, it is necessary for any organization to know whether the practices they are following are in line with the DevOps initiative. A DevOps assessment is nothing but a set of questions that personnel from different organizational areas can answer. Based on the analysis of the answers, the organization can see whether it is moving in the right direction. DevOps assessment helps outline current maturity and identify a road map of improvement areas.

Why go for a DevOps Assessment?

An organization with continuously changing IT and performance needs might be thinking of going for—or may have already started—a DevOps initiative. It’s important to ask the following questions:
- Do I really need DevOps?
- Is my infrastructure flexible enough for DevOps?
- What value is DevOps going to add to my business?
- Are my DevOps strategies right?
If yes, then the DevOps assessment is totally built for you.

What Are the Typical Questions in a DevOps Assessment?

A DevOps assessment comprises of three stages: Dev and Test, Release and Deploy, and Manage and Monitor—all of which are part of any product development cycle. The assessment includes questions mostly about production test cases and environment, test cases and test data, the level of automation, complexity, visibility, proficiency in the dev-test-release-deploy-manage processes, etc.

What Can You Achieve with a DevOps Assessment?

Once a DevOps assessment has been completed, organizations can get a clear idea of functional areas which are underperforming, performing as expected or overperforming.

A typical DevOps assessment report provides an evaluation of an organization’s current DevOps ability and recommendations for improvements based on the responses. Not only that, the organization also can get an idea of where it is compared to its competition, the expertise it needs to cement its position, the most valuable talent assets in the organization and the talent pool the organization needs to recruit. A DevOps assessment can help the organization align its software development and IT operations processes with business goals, and ship products faster without compromising with quality and with more reliability because of increased agility and ability to adapt to changes.

In short, it will help:
- Technical teams to baseline their performance and understand strengths as well as areas of improvement
- Recruitment team to develop capabilities and recruit related talent
- Management team to target the investment areas
- Operations team to understand the most critical operations
While DevOps has proven its accelerative impacts on development processes and timelines and quality assurance, it also imposes demands that every organization must be ready to address. DevOps Assessment is a real necessity to get an idea of where you are and where you can be on your DevOps journey.

About the Author / Chaitanya Jawale

Chaitanya Jawale is founder and CEO of Opcito Technologies, a leader in microservice-based product engineering and automation. Chaitanya comes with strong background in product engineering, datacenter automation, DevOps and multi-cloud orchestration and management.
July 24, 2017
How Much QA is Enough in Software Development?
People working in quality assurance as testers, managers or engineers know what it’s like having to justify their existence; it’s been part of the job for many years. QA experts have had to fight the perception that what they do is a bottleneck, overkill and it slows everyone down.

Finally, however, QA is getting the attention that it deserves. Companies are gradually investing more in QA skills and tools. This began with the onset of Agile and DevOps—methodologies which have now infiltrated mainstream software development practices everywhere. Since those practices focus on speed, efficiency and quality, engineering leaders have grown to understand that great products are not achievable without adequate QA checks and balances. Another risk is the unflinching nature of high user expectations: Thanks to social media, one user complaint can quickly escalate into others across the web, creating an uncomfortable perception problem for the software provider.

Robust testing practices can prevent egregious flaws in the software that result in poor user experience and performance—or, worse, a breach. True, testing can sometimes get in the way of speed. Yet no matter how fast you release a product, if it doesn’t work well, no one’s going to buy it, much less use it.

Nevertheless, we can continually improve the QA practice to fit into modern development teams’ workflow. Testers need to work with developers and master automated testing, while developers need to sometimes engage in testing; everyone should think about the appropriate level and amount of testing. You can have too much of a good thing.

The Right QA Balance

The below ideas can help your organization reduce conflict between QA aficionados and those who would rather code and release as fast as possible.

1: Assess, analyze and report. Developers and testers alike are highly technical; they like numbers. Therefore, the more that the QA team can quantify the risk of an application, provide evidence as to the effectiveness of current QA practices and use metrics to show the need for more QA resources when warranted, the faster that everyone will get on board.

There are plenty of tools that can deliver common quality metrics such as total number of defects, defect leakage, test coverage, test pass/fail rates, test completion rates, code coverage and application uptime. It’s important to share and discuss these metrics throughout the development team and with product and business managers. What constitutes high quality for the development team may not be the case with executives who are closer to the end user/customer.

2: The gut check. After metrics, take into consideration the comfort level of your team and company concerning product flaws. In regulated spaces, or for applications that collect sensitive private data, the tolerance for bad code and security gaps is usually low. Gather feedback from different perspectives and roles in the organization for a balanced perspective on the QA practice and areas for improvement.

3: Get people involved. Quality assurance is no longer a separate activity from development; especially when you are practicing Agile and DevOps, the responsibility for quality is shared. Therefore, developers, business analysts and operations managers all should be doing their work with an eye to quality, delivering feedback regularly to the QA team. Provide opportunities and incentives for any team member to attend events and training related to software quality.

4: Streamline testing. Test automation tools are helping product teams be more comprehensive in testing, by automating low-risk test cases so that testers can focus their exploratory and manual methods on high-risk or new features. This is a positive development, although testers must take time to learn how to use the tools. Some other ways that can help teams be more efficient in testing include:
- Conducting more testing early in the sprint. This avoids the 11^th hour disaster of finding a major flaw right before the release deadline.
- Per Agile methods, dividing development tasks into smaller chunks with team members from different disciplines to ensure testing is being done all the time and with visibility by all.
- Using test driven development (TDD) or behavioral driven development (BDD), which requires that QA, developers and business stakeholders meet at the beginning of a project to agree upon requirements and test cases. Developers then write code against the test, until it passes. New tools are helping bridge the gap by writing requirements and user stories in plain English.
- Outsourcing part of the testing load during a major project, which can help increase coverage and reduce stress on your team. However, understand that too much outsourcing will undermine any efforts to develop QA as an internal best practice and “mindset” for software teams.
If your team is committed to QA as a best practice, not an afterthought, those at the top will need to invest properly in the right tools, training and people to get results. Misconceptions about QA still exist; contrary to the view of some DevOps experts, we can’t (and shouldn’t) automate every test. As well, even though developers are now testing the code they write, you still need experienced testers on your team. With the pace of innovation and the high stakes in the marketplace, organizations need more—not fewer—people engaged in testing.

For testers who love their job, as many of them do, it’s an exciting time to be in the field. Companies increasingly are seeing the value of QA and testers continually are gaining more tools and tactics to excel at helping their organizations produce really great software.

About the Author / Kyle McMeekin

Kyle McMeekin is Senior Sales Engineer at QASymphony. Prior to QASymphony, he was a programmer analyst at Cognizant Technology Solutions. Connect with him on LinkedIn.
May 8, 2017
Getting the Most Out of QA in Complex Environments
We live in a world in which software is increasingly dominant. Not only do we rely ever more on software, but its complexity is becoming even greater. And with increased complexity, bugs are more likely to be intermittent, making failures difficult to reproduce. Sometimes this non-determinism results in failures never getting fixed. Any unfixed bug is potentially a security breach or catastrophic customer outage waiting to happen. With such risks on the line, ensuring your software is of the highest quality is essential to running a successful operation.

To cope with this complexity while still providing some assurance of delivered software quality, the DevOps industry has embraced a variety of automated testing regimes. A development team of a given size is able to run many more tests than it could before and must triage, and attempt to resolve, a correspondingly larger number of test failures. Diagnosing the cause of each test failure can still take a lot of time, particularly if the problem is intermittent or otherwise not easily reproducible. In fact, developers can spend on average as much as 50 percent of their programming time debugging, not coding. Think of all the innovation that could occur if that time were reallocated.

Traditional QA

When it comes to Quality Assurance (QA), developers and testers are all aware of the sequence of deliverables, beginning with development delivering an untested product to QA, which releases the tested and approved product to the end user. However, when a bug is found, QA and Development must communicate with one another to solve the issue quickly. The effectiveness of the delivery channel between QA and Development hinges on the quality of the information about the failure.

Some common challenges raised by inefficient QA processes include:
- Complex control flow – Difficult to make inferences about how a failure unfolded
- Non-deterministic failures – Hard to reliably reproduce a failure to investigate the root cause
- International R&D – Collaboration is made more difficult by geographic separation
Conventional test systems inform that there is a bug but offer limited information about the cause. Workflows such as continuous integration (CI) often offer clues, such as informing which commit to the codebase caused the failure. However, there are several cases where CI cannot directly help identify problematic code changes. For example, the triggering code change may be entirely correct, yet cause a pre-existing bug to manifest. The developer cannot simply stare at the modification in the expectation that the source of the bug will eventually appear. Conversely, the failure may not easily reproduce for the developers even if they run the same test again. In short, CI is often very good at making the easier bugs very easy to fix, but typically is of little help with the more difficult bugs, particularly intermittent ones.

The Next Evolution of QA

To bridge the gap between identifying a failure in the QA process and resolution by development, DevOps teams have adopted a variety of technologies to complement their automated testing setups. New to this scene is technology capable of recording all or part of a program’s execution for subsequent replay and analysis, capturing test failures as they happen. The recording contains all the information a developer needs to understand the exact set of conditions that led to the test failure. It can be analyzed on different machines to the one on which the failure occurred, facilitating collaboration between colleagues who can work on separate machines to resolve the same issue—a particularly useful benefit for large software companies with geographically distributed teams. By gaining the ability to debug an exact replica of the failure, developers immediately reduce the time and effort required to diagnose and fix failures, therefore speeding the development cycle.

Being able to generate recordings of failed tests, especially those intermittent issues which might not otherwise get fixed, coupled with the productivity gains of increased collaboration between development and QA allows companies to maintain a competitive advantage in a volatile marketplace.

About the Author / Greg Law

Greg is the co-founder and CEO of Undo. Greg has 20 years’ experience in the software industry and has held development and management roles at companies including the pioneering British computer firm Acorn, as well as fast-growing startups, NexWave and Solarflare. Greg left Solarflare in 2012 to lead Undo as CEO and has overseen the company as it transitioned from the shed in his back garden to a scalable award-winning business. Greg holds a PhD from City University, London, that was nominated for the 2001 British Computer Society Distinguished Dissertation Award. He lives in Cambridge, UK with his wife and two children.
March 16, 2017
Automation Killed QA – It’s Time to Deal With It

Quality assurance (QA) had a good run.

From the time it was considered an afterthought by organizations and software vendors alike to the era of massive outsourcing contracts, followed by significant vendor investments in related tooling, and, finally, its recognition as a critical success factor among high-performing organizations, quality has evolved to become even more important than functionality to end users.

Customer experience, today, is defined by speed and quality above all else. Namely, can I access the information and products that I want in the blink of an eye and get expected feedback based on input? As a result, customer experience has also shifted: It’s all about application experience, not the friendliness of a checkout cashier or sales/service rep.

With this massive shift toward customer experience platforms (think omni-channel) and increasingly lofty expectations, the operative question becomes, How are we enabling velocity in our testing practices to ensure the highest quality while keeping up?

We’ve tried everything from outsourcing to insourcing, investing in tooling, investing in people and even building testing architecture teams to oversee the automation, quality and consistency of testing results. Roughly seven years ago, my top priority for testing teams was to automate 30 percent of our tests. Today, if you are not 100 percent automated, or close to it, you are severely lagging in output.

But what’s next? What if I’ve automated all of my tests and my manual testers have been replaced with automation engineers, but I’m still having quality issues and being perceived as a bottleneck?

The answer is simple. Automate everything.

Makes sense, right? Isn’t everyone telling you that? But what are we defining as “everything” from a testing perspective? Therein lies the problem; we are thinking about testing as its own factory within the larger software development life cycle.

While we believe we are changing this perception by going agile and moving most QA functions into scrum teams, the reality is we still look at testing as a standalone function. Centralized or decentralized, this is the case, and we need to think differently.

Automating for quality begins with defining requirements. How am I ensuring that I’m covering all cases? How do I ensure I’m translating the business requirements properly? These questions, the time it takes to answer them and the risk of human miscalculation all impact quality and velocity.

We must start our automation for quality at the beginning, during establishment of business requirements.

Let me introduce you to next-generation QA. Intelligent “continuous testing” platforms are replacing humans for testing, automating tests, creating data and building test cases. These platforms are able to translate model-based business requirements into complete automation—test cases, test data and test script automation.

Orchestration across your software development life cycle creates your environments, lays down your test data, deploys your software, runs your test scripts and reports results. No more human error, no costly resources and guaranteed test coverage that ensures quality.

Integrating your existing automation around continuous integration, portfolio management reporting, and execution reporting to this framework now gives you the ability to report and automate your ideas all the way through to outcomes.

Early adopters are already doing this. QA as a functional organization is gone. If you aren’t already automating to this end, you are severely behind.

Questions? Comments? Perspective? Have at it in the comments below.

— Ron DeCanio

January 27, 2017
DevOps Chat: Steve Hazel, Sauce Labs

I recently had the chance to sit down and speak with Steve Hazel, CTO and co-founder of Sauce Labs. With so much being written and talked about the role of quality assurance (QA) and testing in a continuous integration/continuous delivery (CI/CD) world, it is good to hear from someone like Steve who is an actual thought leader in the space. Steve tells us what he thinks the limits of automated testing are, the future of QA and more. Have a listen for yourself to find out, or follow along with the printed transcript below!

Alan Shimel: Hi, everyone. Alan Shimel, devopsy.kinsta.cloud here, for another DevOps Chat. Today’s DevOps Chat’s guest—that was a tough one—is Steven Hazel, co-founder, CTO at Sauce Labs. Steven, welcome to DevOps chat.

Steve Hazel: Hi, Alan. Glad to be here.

Shimel: Thank you. Glad to have you here, actually. So Steven, there might be some people listening who maybe aren’t familiar with Sauce Labs. So, before we start, why don’t we, if you don’t mind, give our listeners a quick overview? Tell us about Sauce Labs.

Hazel: Yes. At Sauce Labs, we do automated testing for both web and mobile apps. We run a cloud service that gives our customers access to a wide range of platforms, whether that’s operating systems and browsers for web testing, or mobile devices and emulators and simulators for mobile testing. You can get access to all the different platforms you need to test on, and you can get access to a lot of platforms at once to run your tests in parallel to get through a large test suite rapidly, which is really what people need for today’s continuous integration and continuous delivery workflows.

Shimel: Got it. Of course, Steven, Sauce Labs was closely linked with around Selenium and some of the open-source testing tools that are out there today, right?

Hazel: That’s right. Selenium is the dominant web-testing technology. In mobile testing, there’s a variety of technologies, but we support Appium, which also uses the web driver protocol that underlies Selenium. We really think that fitting in with those open-source tools is the smart way to go. It fits with what people are doing.

It gives people a lot of choices about how they want to do their testing, what kind of languages and frameworks they want to use on their end. So, we just plug right into Selenium tests, if you’ve already got Selenium tests, or Appium tests. You can just change the config and point them at Sauce Labs, and they’ll run unmodified on our systems.

Shimel: Got it. Steven, we were talking before we started recording today, I was telling you about a recent article we had over on devopsy.kinsta.cloud. The gist of it was basically that DevOps was the death knell for QA. I told you I didn’t agree with that and you agreed with me. You said you didn’t agree with it, either. But for our audience—they’ll respect you a lot more than they respect me—why don’t you agree with that, Steven?

Hazel: Yeah. I think that to suggest that there’s no longer a role for manual QA when you take on a DevOps process is just not in line with the facts. If you look at what we see large organizations doing, they’re not dropping manual QA for automation. They’re changing the mixture of what they have manual testers doing by quite a lot, but they still have manual testing teams. You still need a manual testing team.

So, what I see happening, as you start to take on automation, your testing coverage improves dramatically because you can afford to do a lot more testing when you’re automating than was ever possible with a manual testing team. But automated testing, the classic example is exploratory testing. You’re just not gonna get the kind of creative, exploratory walk-through of your app with an automated test that a manual tester can do. I think there’s still a role for QA organizations, beyond just manual testing, in being the other side of the conversation, right?

A lot of people have the concern that if developers are writing tests, it’s the inmates running the asylum. The exact developers writing the software are the ones deciding whether it’s high quality, and you want another perspective in there. You want a perspective that’s maybe closer to your end user. So, I think that the other side, beyond just manual testing, that you see with QA teams continuing to have a role is in deciding what cases should be tested, whether you have good test coverage, encouraging good testing practices, even when you are automating your tests.

Shimel: Absolutely. Really, Steven, I guess the boogeyman in the room around the friction with DevOps and QA is automation. There’s some schools of thought that says, “Well, we can automate everything. The more we automate, the better we are.” There’s another school of thought that says, “Well, you can automate as much as you can and automation’s not a bad thing, but there are just gonna be some things that are not automatable, if that’s a word now, or may never be automated. Where do you go in on that continuum?

Hazel: Well, being someone who’s pushing forward automated testing, I want to see automation go as far as we possibly can. I think that that’s very far. That said, I think that the way the mixture of tests changes as you adopt automation is maybe a little bit counterintuitive. You might imagine, well, we’ll automate some percentage of our manual test cases, maybe 80 percent or 90 percent. Maybe we could get to 100 percent if you’re a zealot for automation.

But actually, what you see is, it’s not that you’re automating a percentage of what was there in manual testing. You’ll actually have more automated test cases than you were able to do with manual testing. So, if you just looked at the number of test cases, your automated cases are gonna way outnumber the original count of manual test cases just because you can, right? You can now cover a lot more edge cases, a lot more scenarios because of the power of test automation. So, that’s a real advantage to test automation.

Shimel: Sure.

Hazel: At the same time, certainly there are things, visual testing, there’s some interesting approaches out there, but I don’t think we’ve yet seen a way of making sure that the app looks right, especially as it changes. You really need a human being involved when there are deliberate visual changes to make sure that they came out right. I think it’s those things where automation really can’t give you an answer about, does it look right? Does it act in a way that’s gonna be acceptable to somebody?

I think you still need a human being in there at some point, checking those things, but it’s a different process. It’s no longer this phase gate before every change is released that you used to have. It’s now a process that can be made asynchronous from the bulk of your development and release process.

Shimel: Absolutely. If you don’t mind, Steven, a little bit of time on, you mentioned a little bit, but what do you think of the kinds of tests that lend themselves well to automation? Certainly, a visual test of an interface, different arts. What are the sweet spots for automated testing?

Hazel: I think certainly at the bottom of the testing pyramid, unit testing brings tremendous value. It’s something that’s very easy to automate in a way that’s completely deterministic, very fast. Gives you really quick feedback as a developer as you’re making changes to the code on whether you’ve broken something. But the coverage with unit testing, the coverage of functionality that may be important to your end users, it’s really hard to achieve. It’s really hard to achieve good, solid feedback from your unit tests that you haven’t broken anything anywhere.

Functional testing, the next layer up, which is what really we focus on, gives you that complete end-to-end check. You’re actually testing the features of your product and you’re making sure that end to end, with all the various components involved, they work properly. That’s more of a challenge for automation, but we’re seeing, I think, a majority of development organizations out there seeing success automating functional tests and real value from having that be something where you can run through hundreds of test cases in minutes and get feedback to your team. So, that works quite well for automation.

I think it’s that next level up, as you start to get into the types of tests where you’re looking at, what’s an acceptable experience for the end user? User acceptance testing as distinguished from functional testing, where you’re really starting to look for, does it look right? Does it act right? Is the response time throughout the entire process acceptable?

You can zoom in and measure a lot of these things very specifically, but getting the answer to, “Overall, though, was it a good experience?” That’s the kind of thing where it’s really tough to cover that with automation.

Shimel: Yeah, agreed. Steven, if you don’t mind, I’m an old security guy prior to DevOps. One of the things within the security community that we’ve really pushed around is the idea of automated security testing shifting left in a DevOps environment and allowing security testing to be done in an automated fashion at a CI/CD-type of level. Where do you see that in this whole discussion?

Hazel: Security testing isn’t what we focus on at Sauce Labs, so –

Shimel: Right, I know.

Hazel: I’m not an expert in this, but it is something that I follow and care a lot about. I think I see a lot going on with static analysis, with some of these tools that can help to constrain the types of mistakes that you could have in a program, type systems and things along those lines. I think really that’s very badly needed in making code secure. Some of the issues we’ve seen come up over the last few years, things like Heartbleed, the Glibc bug, really those kinds of things aren’t acceptable, aren’t something that we can continue to have happen, and I think the solution really is fully automated tools, like static analysis tools and type systems, that can lock down the code and just constrain the types of mistakes that could possibly happen.

Shimel: Sure. You know, Steven, I see it as very analogous to the QA situation, which is I think a lot of security people get very territorial around job security because they’re afraid somehow that doing automated security testing earlier in the process—shifting left, so to speak—is somehow gonna affect their livelihoods, or make them obsolete.

Hazel: I can see that. We see that among QA folks, for sure. To me, that’s a shortsighted view. If the world is getting more intense security problems, more intense quality problems, we’re shipping more software, you just can’t solve it 100 percent by throwing bodies at the problem. I think that what’s gonna happen is not that those people’s careers are in jeopardy; instead, as a society, we’re just gonna do more and better security and quality testing than we’ve been able to do in the past.

Shimel: Yeah, agreed. Steven, we’re running a little low on time, so let me get just one or two more questions. Sauce Labs has been, in my opinion anyway, for what it’s worth, somewhat of a DevOps darling in terms of really encompassing—you’re living the DevOps model. You’re working with open-source tools, testing and automation, enabling automation. Sauce Labs is a player in the DevOps ecosystem. Where do you see it going?

Hazel: Well, toward the enterprise is one big trend we’re seeing. Earlier on in the history of Sauce Labs, a lot of our customers were essentially software companies, companies that their primary focus is technology. Now, a lot of our customers are retailers, financial institutions—companies that you don’t think of as primarily a software company, but all of these companies have apps, they have complex web experiences. They’re really adopting DevOps in a big way. They’re moving to what is, for them, a very different process than what they’ve been doing and allows them to ship better software faster, with more complexity than they were able to handle before.

Shimel: Absolutely. Steven, we are low on time. I’m gonna ask you the final question, which I ask most all of my guests here on DevOps Chats. That is, if you had to recommend one book to our audience that they should read, a must-read for them, over the next six months or a year, what would that book be?

Hazel: Oh, wow. A book recommendation. I think the obvious choice in a conversation about DevOps is “The Phoenix Project.” If you haven’t read it, you’ve got to read it. It’s a quick, enjoyable read, but it really gets the concept across better than probably anything else is going to. It’s an obvious recommendation, but I still think it’s worth it.

Shimel: Sometimes the obvious is good. I will just add to that, if you already have read “The Phoenix Project,” Gene Kim’s next book, I think co-written with John Willis and Jez Humble and someone else that I’m blanking on – oh, darn it, the guy who started DevOps, who coined the phrase. Patrick. Patrick Debois. Their new book, I think it’s called “The DevOps Handbook,” will be out, I want to say, in the fall. Not a sequel to “The Phoenix Project,” but another good book to look at.

Anyway, Steven Hazel, CTO, co-founder of Sauce Labs, thanks for being this episode’s guest on DevOps Chat, and continued success with Sauce Labs.

Hazel: Thanks a bunch. Thanks for having me.

July 25, 2016
How to Fail at Test Automation
Let me start by admitting that I am not a test automation expert. I have done some work with test automation and have supervised teams who practiced it, but when it comes to the intricacies of it, I have to call a friend. It is from such friends that I have learned why so many test automation efforts fail. Talking to people about test automation validates my impression that this is the DevOps-related practice that people most often fail at.

Let me share the four reasons why test automation fails, in the hope that it will help you avoid these mistakes in your test automation efforts.

Before I go into the four reasons, allow me one more thought: Test automation is actually a bad choice of word. You are not automating testing, you automate quality assessments. What do I mean by that? It is a mistake to think of test automation as automating what you otherwise would do manually. You are finding ways to assess the quality of your product in automated ways and you will execute it way more often than you would do manual testing. This conceptual difference explains to a large degree the four reasons for test automation failure below.

Reason 1: Underestimating the Impact on Infrastructure and the Ecosystem

There is a physical limit of how much pressure a number of manual testers can put on your systems. Automation will put very different stress on your system. What you otherwise do once a week manually you might now do 100 times a day with automation. And into the mix an integrated environment, which means external systems need to respond that frequently, too. So you really have to consider two different aspects: Can your infrastructure in your environments support 100 times the volume it currently supports, and are your external systems set up to support this volume? Of course, you can always choose to reduce the stress on external systems by limiting the real-time interactions and stub out a certain percentage of transactions or use virtual services.

Reason 2: Underestimate the Data Hunger

Very often test automation is used in the same system where manual testing takes place. Test automation is data-hungry, as it needs data for each run of test execution and, remember, this is much more frequent than manual testing. This means you cannot easily refresh all test data whenever you want to run test automation and have to wait until manual testing reaches a logical refresh point. This obviously is not good enough; instead, you need to be able to run your test automation at any time. There are a few different strategies you can use (and you will likely use a combination):
- Finish the test in the same state of data that you started with;
- Create the data as part of the test execution;
- Identify a partial set of data across all involved applications that you can safely replace each time; or
- Leverage a large base of data sets to feed into your automation to last until the next logical refresh point.
Reason 3: Not Thinking About the System

Test automation often is an orchestration exercise as the overall business process in test flows across many different applications. If you require manual steps in multiple systems, then your automation will depend on orchestrating all those. By just building automation for one system you might get stuck if your test automation solution is not able to be orchestrated across different solutions. Also, some walled-garden test automation tools might not play well together, so think about your overall system of applications and the business processes first before heavily investing in one specific solution for one application.

Reason 4: Not Integrating it into the Software Development Life Cycle

Test automation is not a separate task; to be successful it needs to be part of your development efforts. From the people I have spoken to there is general agreement that a separate test automation team usually doesn’t work for several reasons:
- They are “too far away” from the application teams to influence “ability to automate testing,” which you want to build into your architecture to be able to test the services below the user interface;
- Tests often are not integrated in the continuous delivery pipeline, which means key performance constraints are not considered (tests should be really fast to run with each deployment);
- Tests often are not executed often enough, which means they become more brittle and less reliable. Tests need to be treated at the same level as code and should be treated with the same severity. This is much easier when the team has to run them to claim success for any new feature and is much harder to do when it is a separate team who does the automation. It also will take much longer to understand where the problem lies.
Of course, absence of failure does not mean success. But at least I was able to share the common mistakes I have seen and, as they say, “Learning from others’ mistakes is cheaper.” Perhaps these thoughts can help you avoid some mistakes in your test automation journey. I do have some positive guidance on test automation, too, but will leave this for another post.

And in the case you found your own ways of failing, please share it in the comments to help others avoid those in the future. Failures are part of life and even more so part of DevOps life (trust me, I have some scars to show). We should learn to share those and not just the rosy “conference-ready” side of our stories.

Test automation is for me the practice that requires more attention and more focus. Between some open-source solutions and very expensive proprietary solutions, I am not convinced we in the IT industry have mastered it.

One bonus thought: If you cannot automate testing, automate the recording of your testing.

If you cannot automate testing, find a way to record the screen with each test by default. Once you identify a defect you can use the recording to provide much richer context and make it a lot easier to find the problem and solve it. Verbal descriptions of error are very often lacking and don’t provide all the context of what was done. I keep being surprised how long triage takes because of the lack of context and detail in the defect description. There is really no excuse for not doing this. Record first, discard if successful, attach it to the defect record if you find a problem.
July 6, 2016
How DevOps is Killing QA
If you work in software quality assurance (QA), it’s time to find a new job.

The overall software development process has always consisted of the following core activities:
- Creating a requirements analysis;
- Producing a specification of the software artifact;
- Constructing the software;
- Investigating the quality of the product or service;
- Finding and fixing defects; then
- Deploying to production.
In most organizations, this process generally has occurred within one or more methodologies such as waterfall or agile. In waterfall, the testing process occurs relatively late. The software is produced, bugs are found by the QA team and then the software is fixed. These last two practices repeat in a cycle until management is comfortable that it can be released to the public.

In agile, individuals and teams—including QA—work closely together to release and then update software on an ongoing basis rather than deploy an entire platform all together and all at once. DevOps, as we have written in our “What is DevOps?” guide, is the next generation of agile development. Agile is a way of thinking in software development; DevOps goes one step further and is a concrete cultural change that implements the philosophy within an organization.

And that implementation eliminates the need for QA as a separate entity in the organization and disperses the responsibility for quality across different teams, even though many have argued that the discipline is still needed in one form or another.

Past Interpretations of QA in DevOps

One common argument is that DevOps sits at the center of a Venn diagram of dev, ops, and QA:

On one side, QA works together with development in trying to push more of their tests into the continuous integration system. Tests must have zero human intervention and generate their own test data.

On the other side, QA works with operations to collaborate on monitoring tools; perhaps to continuously run smoke tests in production. Chances are that operations is developing scripts for backup and restore of production systems, rollback of deploy, or scripts for disaster recovery.

Tim Hinds at NeoTYS looks at the issue differently and says that “DevOps QA” is meant to prevent defects rather than find them:

QA takes a critical role in this organizational structure because they have the visibility and the directive to push code out when it is working, and roll it back when it is not. This is a very different mindset from QA organizations of 10 years ago, whose primary responsibilities involved finding bugs. Today QA groups are charged with preventing defects from reaching the public site.

With all due respect, both are wrong.

Why DevOps Does Not Need QA

DevOps often uses continuous integration (CI) and continuous delivery (CD). In CI, developers use various continuous integration tools to integrate code into a shared repository multiple times each day, and DevOps relies on automation to determine the quality of the version. You cannot have any human interaction if you want to run CD, which ensures that any version of a batch of code in the repository can be released at any given time.

Essentially, the traditional QA cannot work in a full CI/CD environment. In older structures, the responsibility for software quality was in the hands of QA. Today, it’s part of the DevOps culture and methodology—the developers now own the responsibility rather than a separate entity within the organization.

Specifically, DevOps entails the use of vendors and tools such as BUGtrack, JIRA and GitHub to aggregate and report software bugs and defects. Test automation tools including Selenium, Cucumber, JUnit, TestNG and JMeter manage, execute and measure functional tests and cycles.

In the end, if there is a layer of people in the middle between development and operations, then you cannot perform CI and CD seamlessly and, therefore, cannot do DevOps. To run a proper DevOps operation, you cannot have QA at all.

The Future of QA

So, what will happen to all of the people who work in QA? One of the happiest jobs in the United States might not be happy for long as more and more organizations move to DevOps and they become more and more redundant.

According to the U.S. Bureau of Labor Statistics (BLS), software quality engineering is one of the high-tech professions whose growth rate is projected to be slower than average:

However, the BLS numbers are likely too generous because the bureau does not yet recognize “DevOps” as a separate profession at all:

As evidence, just look at how the relative number of Google searches in Google Trends for “sqa jobs” is slowly declining while the number for “devops jobs” is rapidly increasing:

For QA, the trend definitely does not look good.
June 7, 2016
Trust, You Must: 8 Tips for Adopting CI/CD

We have done continuous deployment (CD) for four years, and enable many of our customers to do continuous deployment with quality assurance (QA). We’ve helped tons of companies adopt and optimize their CD setups. We’ve found that continuous deployment is all about trust — augmented with processes and infrastructure.

To that point, here are some common tips for adopting continuous integration (CI)/CD that I find myself sharing frequently:

Use a CI Tool

This sounds obvious, but you’d be surprised at how many companies try and roll their own. Don’t. There’s no upside for your company—and infinite downside. We love Circle, but there are plenty of other great options. Most are at least somewhat prescriptive when it comes to workflow, which is important when you’re getting started.

Do Unit Testing

It’s always worth reminding everyone that continuous deployment is simply continuous integration applied to deployments. To do continuous integration, you need good unit test coverage. If you haven’t got a solid base of unit testing and continuous integration already, you’re not ready for continuous deployment.

Have Solid Production Monitoring

Bugs and regressions will get shipped; it’s inevitable. Put a system in place, by looking at the data in production, to tell you when a regression or bug was shipped. Tie this into automated rollbacks, so that if a significant performance or key performance indicator (KPI) regression (i.e., signups per minute) appears, your CD system rolls back to the most recent stable version while your team investigates.

Trust Your Team

We talked about the infrastructure requirements, but if you don’t have trust, then continuous deployment isn’t going to work for you. Trusting your team is about giving your developers the freedom to make mistakes, deploy as they see fit and check each others’ code. It is possible to combine continuous deployment with a culture of hierarchical permissions, but we have yet to see that truly work.

Remove Red Tape from the Code Review Process

Your team should review code changes, typically through a pull request, which goes hand in hand with trusting your team. Developers should be trusted to choose the most qualified and insightful person to review their code. One of the most common mistakes among teams trying to adopt continuous deployment is that they only let one or two “senior” developers review the rest of the team’s code, “bottlenecking” all releases. Don’t make this mistake.

Involve Developers Closely in Production Operations

The most common mistake we notice among companies that don’t make a successful transition to continuous deployment is that their development organizations are siloed. Dev and Ops should be joined at the hip. Have your developers deeply involved in the setup and definition of your CD infrastructure, and half the battle is already won.

Shift Left: Test Early and Often

Your team needs constant feedback. Think of your QA objective as getting the right feedback in the right hands at the right time. When it comes to deployments, this means knowing what’s working and what’s broken as the breaking change is made. The sooner you uncover the bug after it’s created, the easier it is to fix. The best companies doing continuous deployment have comprehensive unit test and integration test coverage that runs before any code hits the staging server.

Treat Staging as Your Holy Place

The final pre-production environment is usually known as ‘staging.’ A common mistake is to have developers ship new code direct to staging, while figuring out if the new code is ready to go live. This is a terrible idea. As soon as more than one developer is trying to ship their changes, your Slack channel will blow up with cussing and recriminations. Isolate the pre-production environments to each project, and enforce the “staging as ready to go live” philosophy. This way, integration testing can happen at a self-defined pace, without anyone else’s release being blocked.

About the Author / Fred Stevens-Smith

Fred Stevens-Smith is Co-founder and CEO of Rainforest QA, a quality assurance-as-a-service startup backed by Storm Ventures, Rincon, Andreessen Horowitz, Morado and Crosslink.

May 5, 2016