devopsy.kinsta.cloud

Tag: Automated testing

Mobile Testing Delivers Big for Pizza

Mobile applications have been disrupting industries worldwide. For some industries, the disruption focuses on business internals, as enterprise apps enable manufacturing processes to be restructured and streamlined. In other industries, the disruption focuses on the customer and how mobility can make life more convenient.

When it comes to pizza, however, mobile development changes everything.

I remember how, as a kid, we would hope the teacher wasn’t looking when we used the public payphone to place an order for a pizza to be delivered a few minutes after we raced back home for lunch.

Now you can do it with the click of a button. Using a mobile application, you can order exactly the type of pizza you like, and it will be delivered to your location whenever you want.

There are few businesses better-suited for a perfectly working mobile app.

When we were asked by a multibillion-dollar pizza giant to deliver on its mobile testing, we knew the pressure would be on. The company needed a new quality assurance strategy to handle a mobile app that would direct traffic to its more than 3,500 pizza parlors located within the United States alone.

The Big Step: From Manual to Test Automation

At the onset of mobile testing, manual testing was the primary method. As the mobile universe expanded to thousands of devices, a more efficient way had to be developed to support business-critical apps being used by people with different devices and mobile operating systems running over different local networks.

Enter the dedicated mobile testing lab.

Utilizing our cloud testing solution, a lab of 60 devices was connected to the two mobile testing teams located in the United States and India. Mobile testers had access to the devices as they needed them. With a test automation solution in place, more than 200 scripts were written and deployed to handle the regression testing.

Every device that wasn’t actively being used by a tester would be busy conducting regression tests. The 60 devices running at the same time enabled parallel execution, wherein an app could be tested over a matrix of devices and mobile operating systems. With our performance testing solution, different network conditions were simulated, expanding the overall mobile test coverage exponentially.

A dedicated mobile device lab meant that the devices were made available exclusively to the business. This prevented any unnecessary breaches in security. At the same time, all remote mobile testing was conducted within the customer’s virtual private network (VPN), bolstering security even more.

One of the biggest time-savers was the ease in which test cases could be re-created. As a DevOps solution, both the developers and the testers had access to the same devices. So when a specific bug was reported on a function that wasn’t executing properly over this device running that operating system, the developer didn’t have to worry about finding that device with the right operating system. He simply read the report, clicked the link and accessed the same device in the dedicated lab. The test cases did not have to be re-created because they were available in real time to the developer.

Mobile testing became a 24/7 operation. Right as the American team was checking out for the day, the Indian team was running the reports on their mobile testing activities, and continuing their work. At the end of the day in India, the Americans came back to work to read the Indian team’s testing results.

Mobile Testing Produces Sizzling ROI

By upgrading their regression testing from a manual process to an automated one with parallel execution through the mobile lab, the regression testing cycle went from 10 days to 17 hours. The testing coverage also increased to 90 percent for the native mobile app.

As a result, the company was able to get the latest version of its mobile application to market sooner. The impact was felt beyond the DevOps team as the entire IT department began to plan their strategy based on the dramatically shorter release cycle.

“What makes mobile testing a DevOps success is test re-creation and CI integration.” says Tal Barmeir, Experitest CEO.

The customer uses Jenkins for its CI platform, uniting developers and testers toward automating as much of the DevOps process as possible. Granting the developers and testers access to the same devices established a powerful tool for test re-creation.

Their entire DevOps process was retooled to generate returns that truly deliver!

July 14, 2016
Give Test Automation the Respect It Deserves
One of the key mantras of DevOps is “continuous.” Effective DevOps relies on automating routine tasks. Any step in the process that requires human intervention or manual input becomes a roadblock that can slow or halt productivity. To the extent that it’s possible, anything and everything should be automated—and that includes testing.

Test is like the Rodney Dangerfield of DevOps—it gets no respect. It should, though. Test—and test automation—is a crucial element of a successful DevOps program. A recent Gartner survey found that 50 percent of respondents listed test automation as one of the top enablers of DevOps success.

Joan Wrabetz, CTO of Quali, is focused on automating test capabilities and ensuring that testing gets the respect it deserves in DevOps. DevOps isn’t really a process with a beginning and end. It’s a continuum—a self-perpetuating cycle. When test teams take the lead, they need to find ways to push automation back into development and forward into staging and production. “In fact, the problem of test automation is particularly difficult, and this means that the team that implements test automation is in a great position to lead DevOps initiatives beyond test,” Wrabetz says.

That self-perpetuating cycle “starts” with planning and development and “ends” with deployment into operations, but that flows right into monitoring and management, which leads to discovering issues and identifying new feature requests—and that takes you right back to planning and development. If done properly, test falls somewhere in the middle of that cycle, so it’s imperative to streamline and automate that functionality as well.

“Sometimes test drives DevOps. Test organizations have been focused on automation already for 10 years, so often they are the driver for DevOps,” explains Wrabetz. “When test is the focus of DevOps, it is called continuous integration. However, it is also important to work to continuously expand automation outside of test.”

Key Drivers

According to Quali, there are five key components that are essential to successful automation in a DevOps environment:
1. The infrastructure. It starts with one very important characteristic that is critical to testing and often is not automated—the ability to automatically mimic the production infrastructure in which the software (or hardware) being tested is expected to run. Having the ability to give each tester (or set of automated tests) their own personal replica of the target production infrastructure—a.k.a. a “sandbox”—is critical to ensuring that the software coming out of test can be deployed automatically and continuously into production. Automating tests is not nearly as difficult as automating the underlying infrastructure in which those tests run.
2. The production environment workload. Often not considered in automating testing is the requirement to mimic the workloads that would be common in production. This includes the network traffic, other application workloads and network security profiles. These activities need to be automated within the sandbox to create a more realistic environment for testing.
3. The Tests themselves. Of course, the tests themselves also need to be automated and there are a variety of tools to assist in this process. As software gets more complex, this continues to be a challenge, with mobile testing replacing GUI testing as the most challenging type of testing to automate.
4. Reporting and result analysis. To provide continuous automation from development through testing to delivery, it is important to see and analyze test results automatically.
5. Tool Integration. Automation into and out of testing needs to be enabled. This means sandboxes and test automation suites need to be API-driven so that they can be started by a previous tool in the DevOps tool chain and can also initiate a tool that follows testing in the tool chain.
Show Some Respect

Testing is a crucial part of the DevOps continuum. It needs to be done properly and effectively, but it also needs to be a streamlined process that drives DevOps forward rather than a hurdle that slows everything down.
July 1, 2016
What Happens to Test in a DevOps World?
I am always surprised by the urban legends around DevOps. One of my favorites is, If you implement a DevOps methodology, you can stop doing test. People famously quote that Netflix only tests in production as evidence of this. This is perhaps more like of game of Telephone, where much of the story gets lost in translation. It is true that Netflix created a tool called “Chaos Monkey” which generates failures in production as a way to test the fault tolerance of its software in large-scale production use. However, this is not Netflix’s only test; it is an additional type of testing at scale that allows the company to test in new and different ways. Netflix still tests in software development.

Okay, now that we are past the notion that DevOps is about eliminating testing, let’s look at what really happens with test in the DevOps world. In fact, the goal is for testing to happen everywhere and all the time. Let’s look at the two directions test can shift: left or right.

Shift Left

DevOps methodology encourages automation of activities across the entire development process. This can be realized by automating the link between development and test. Often implemented with an automated build tool such as Jenkins, a set of automated tests are triggered whenever a developer completes a build of new code. Of course, this means that both development and testing processes must be automated. Once you reach this goal, then testing can happen all the time. Every time developers want to check in new code, they can trigger a small set of “smoke tests.” If those succeed, they can check in code and when the build happens, they can trigger a larger set of regression tests. The result is a left shift of testing into development.

Benefits of left-shifting test include:
- Move testers into scrum teams to work on stories together with developers.
- Design tests in parallel with development.
- Give developers and testers the flexibility to help each other to reduce bottlenecks.
- Early detection of bugs, shortening the critical metric for time between creating a bug and fixing it.
- Increased transparency of code dependency issues because they are raised during the nightly (or more often) regression testing.
- Virtual elimination of the test bottleneck that occurs after development is completed.
Shift Right

The urban legend about Netflix may be wrong, but that doesn’t mean that there isn’t merit to right-shifting test into production. Automated testing in production is effective for tests that would be impossible to perform outside of production, including scale testing and resiliency testing. Production is also a great place to introduce A/B testing by offering different interfaces to some customers and measuring the differences in business results.

Benefits of testing in production include:
- Better understanding of the real production environment and workloads.
- Early feedback from customers on new features and user experience.
- Better resiliency.
- Overall better user experience for customers.
No Shift

Wait, if we shift left and shift right, is there any testing left to be performed in test? There are still a range of tests that need to be performed by the test team. With full automation, these tests can be performed at a variety of times and don’t need to be a bottleneck to release. These tests include:
- Performance testing
- End-to-end use-case testing
- Security testing (don’t think about doing this in production!)
- Manual testing – often for user interface or mobile testing that is most difficult to automate
DevOps methodology is not promoting the end of testing, but in fact a widening of testing to encompass all of the steps in the DevOps cycle. You know what they say: Test early and often! And next time you are sitting around the campfire discussing urban legends, let’s leave Netflix out.

About the Author/Joan Wrabetz

Joan Wrabetz is the CTO for QualiSystems. Most recently, she was the VP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.
June 13, 2016
Why Financial Companies Should Embrace Automated Testing

The drive to deliver new functionality to customers faster is being felt across every industry, and especially so for financial markets that are under pressure from new market entrants. Adopting agile development and continuous delivery can help financial institutions gain an edge and beat competitors to market, but it’s vital that quality is not forgotten.

Financial institutions are accelerating to update software and release new products in days or weeks instead of months. That means software must be tested rigorously and quickly. Manual testing simply cannot compete with automated testing when it comes to speed, accuracy and the ability to perform functions repeatedly over time. However, it can be challenging to implement successfully.

The Customer is Always Right

Customer power is forcing banks to improve customer experience. Accustomed to slick, efficient website experiences from such leaders as Amazon and Google, customers now are demanding similar experiences from their banks and credit card companies. Millennials expect easier access to banking, precipitating a shift to self-service, which presents all kinds of new IT challenges for banks.

When customers don’t like their online banking experience, they will simply choose another provider. A full 18 percent of millennials switched their primary bank in the last 12 months, according to a recent Accenture survey. When failed or flawed software gets into the customer domain, the fallout can be very damaging and expensive. Think: lost sales, disgruntled customers, bad PR and a massive price tag to fix the problems.

Test Automation a Solid Foundation

Introducing test automation isn’t just about finding defects more quickly and efficiently, though it will do that. It’s also about shifting the burden of regression testing and freeing manual testers to focus on issues such as exploratory testing, usability and penetration testing, where they can add the most value.

To implement automated testing successfully, financial institutions must devise a long-term plan that dovetails with their business objectives. Having a plan—and sticking to it—outweighs the importance of getting C-level support and winning a huge budget for automated testing. Without a plan, the most well-intentioned objectives can get rejected by senior management, stalled by other forces and, ultimately, forgotten.

Automated Testing Misconceptions

The plan should include a robust dismissal of two popular misconceptions about automated testing. The first—and bigger—one is that automated testing will produce instant cost savings, as fewer resources (mainly of the personnel kind) will be needed to perform testing.

Over time, automation will cut costs. But cost should never be the main factor when considering automated testing. Applying test automation greatly outweighs all the negatives of manual testing (time-consuming), but it also opens new doors for more sophisticated testing and, hence, the competitive advantage.

The second misconception is that automated testing is a one-time only exercise. Reality proves that automation is always going to be an ongoing process. Automated testing should be treated as (yet) another development activity: You write the production software and you write the software that tests the production software.

Overcoming the Challenges

Financial institutions face a threat from newcomers to the market, unencumbered by legacy system baggage and physical branches. They are offering disruptive products and new business models. How can existing financial institutions compete?

It obviously makes sense to include the business and marketing teams in the development process to develop new solutions for customers. Extend the DevOps mindset to break down those silos. A united team that includes test and marketing alongside developers can ensure that automated tests are well-designed and focused on business aims.

Pulling together to design and test new ideas can enable financial institutions to reinvent themselves and innovate. Embedding test automation in everything mitigates risk and increases velocity and quality. It can deliver the data businesses need to make the right next move.

Long-Term Vision, Short-Term Plans

Fast feedback is paramount to make the right business decisions, but financial institutions must cater to existing legacy systems and integrate new software deliveries from a range of vendors. Without test automation in place, it’s simply not possible to integrate deliveries, test them and reject them as quickly as possible if they are wrong.

The reality is that organizations must be able to quickly introduce new tools and technologies, because agility and innovation go hand in hand, but the impact on legacy systems must be repeatedly gauged. Test management tooling should be fluid and extensible to cater to this diversity. To accurately measure quality and risk in real time, test automation must be dynamic and flexible.

Failure to embrace automated testing will leave traditional financial institutions exposed and in danger of being swept away by the new wave of lean, agile, IT-focused competitors.

About the Author/Viktor Clerc

Viktor Clerc is product manager for testing applications at XebiaLabs, a provider of automation software for DevOps and Continuous Delivery helping companies to accelerate the delivery of new software. You may reach him at vclerc@xebia.com.

March 1, 2016
Testing: Mind the DevOps Gap
If DevOps is a continuum, it starts with planning and development, and ends with deployment into operations. Much of the focus in DevOps and within the DevOps toolchain has been on application build automation (development) and deployment of apps into operations. But, for DevOps to pay off, there must be a continuous flow and automation from development all the way through to operations, and that means providing automation for the steps in between these two.

What is in the gap between development and production delivery? Testing. Not surprisingly, when Gartner surveyed its clients in 2015, more than 50 percent of them ranked the need for test automation as one of their top priorities for DevOps success. This really illustrates the recognition that DevOps cannot succeed if only one part of the DevOps continuum is automated. Companies that have started to focus on DevOps recognize that they must “Mind the Gap.”

Closing the Testing Gap

Why is automating the various testing steps (regression, performance, stress, GUI, QA, interoperability and security testing) so difficult? There are five key components that must be automated successfully:
1. The infrastructure – It starts with one very important characteristic critical to testing that is not automated – the ability to mimic the infrastructure in which the software (or hardware) being tested is expected to run in when it gets to production. Having the ability to give each tester (or set of automated tests) their own personal replica of the target production infrastructure—a “sandbox”—is critical to ensuring the software coming out of test can be deployed into production automatically and continuously. Automating tests is not nearly as difficult as automating the underlying infrastructure in which those tests run.
2. The production environment workload – Often overlooked in automating testing is mimicking the workloads that would be common in production. This includes the network traffic, other application workloads and network security profiles. These activities need to be automated within the sandbox to create a more realistic environment for testing.
3. Test automation – Of course, the tests themselves also must be automated and there are a variety of tools to assist in this process. As software gets more complex, automation continues to be a challenge, with mobile testing replacing GUI testing as the most challenging type of testing to automate.
4. Reporting and result analysis – To provide continuous automation from development through testing to delivery, it is important to see and analyze test results automatically.
5. Tool Integration – Finally, automation into and out of testing needs to be enabled. This means that sandboxes and test automation suites need to be API-driven so they can be started by a previous tool in the DevOps tool chain and can also initiate a tool that follows testing in the tool chain.
DevOps projects tend to start in one part of the continuum from development to operations. There is nothing wrong with taking this approach. However, it is also important to “Mind the Gap” and work to expand automation across the continuum continuously. One way to ensure that automation can expand easily is to deploy sandbox technology across the continuum, beginning with development and continuing through configuration and release management. The sandbox ensures that at each step, the production configuration and workload is being replicated.

Sandboxing technology has been used in development in the past, but is relatively new to the entire DevOps process. As more companies move to continuous integration and continuous deployment, however, the need for sandboxes is growing.

The sandbox should provide the following:
- The ability to create an accurate replica of the real production environment that deployment is intended for. This can include the physical, virtual and cloud infrastructure as well as applications, network traffic and load.
- The ability to initiate a sandbox via APIs so that the sandbox can be triggered by DevOps tools as an integral part of a complete end-to-end DevOps automation.
- The ability to automate DevOps processes inside of the sandbox (e.g. automated regression testing), as well as to automate the teardown of the sandbox and trigger the next step in the DevOps cycle when those processes complete.
- Business intelligence about everything that happens inside of sandboxes so that DevOps processes can focus on continuous improvement.
- The ability for many people to have their own sandboxes running simultaneously with isolation in shared labs.
DevOps depends on continuous automation from development, through test and QA, to configuration management and production deployment. Sandboxes provide a common automation environment that can be used at each step of the DevOps cycle to close the gap between development and deployment.

About the Author/Joan Wrabetz

Joan Wrabetz is the CTO for QualiSystems. Most recently, she was theVP and CTO for the Emerging Product Division of EMC. Joan received her MBA at the University of California, Berkley, and MSEE at Stanford, and a BSEE at Yale. She has held teaching positions at the University of St. Thomas, St. Mary’s University and the University of St. Thomas, St. Mary’s University. Joan holds patents in load balancing, distributed systems, and machine learning classification and analytics. Her experience inclues: founder and CEO of Aumni Data, CEO of Tricord Systems, Vice President and General Manager for SAN operations at StorageTek, Founder and CEO of Aggregate Computing, Management and senior technical positions at Control Data Corporation and SRI International, Partner with BlueStream Ventures.
January 28, 2016
BoomTown No Bust with VM Supported Parallel Testing Environment

An Introduction to BoomTown DevOps Processes

BoomTown uses a specific set of DevOps processes in order to deliver websites with the latest real estate listings and housing data as well as a CRM solution and analytics tools for real estate professionals. “Our development, operations, and QA teams automate everything in sight while offloading to cloud providers and other services the tasks that those providers are best suited to,” says Brian Baumgartner, QA Manager, BoomTown.

As only one example, BoomTown recently adopted Amazon Web Services (AWS) for hosting various aspects of its infrastructure. “It enables us to provide high availability and resilient service while allowing our development team to focus on revolutionizing the Real Estate software industry,” says Baumgartner.

Docker containers support regular deployment processes at BoomTown, containerizing the environment setup for a more reliable, orderly deployment experience, whether the firm is pushing code to QA or to production, explains Baumgartner. TeamCity, the increasingly popular and frequently appearing name in build management tools and CI servers courtesy of JetBrains is BoomTown’s go to build platform. New Relic is BoomTown’s performance management solution of choice, enabling live transparency into the production application and server performance that keeps BoomTown citizens (customers) content.

BoomTown Before & After Using Sauce Labs VMs to Scale Testing

Prior to applying Sauce Labs, BoomTown’s QA analyst team processed all software tests exclusively in a manual fashion. Every test run crossed their laps / machines / laptops. “Before a new feature release, an analyst would execute a full set of regression tests against the product or feature, often through several different browsers and versions to verify the new release and its function. This process alone took up to two weeks for large projects,” says Baumgartner.

BoomTown now uses a test automation framework constructed using several popular DevOps tools. The custom built testing automation leverages Ruby, Selenium WebDriver, Cucumber, MySQL, Sinatra, and Angular JS, says Joey Gryder, Senior Software Engineer in Test, BoomTown. The framework offers BoomTown views into historical and real-time test results as well as the capability to run manual tests through its dashboard front end software connection to its backend systems.

BoomTown applies a single web service to assert control and manipulation of test system automation features. A “test runner” library enables QA to run Cucumber features in a parallel execution approach. BoomTown uses MySQL to store test results. “When we combine all this with Sauce Labs’ VMs, we can schedule nightly test runs using our full testing suite on a variety of browser configurations,” says Baumgartner. This has enabled BoomTown to run up to 100 tests on software nightly.

Parallelization

Where time, resource, and person hours’ savings are concerned, the secret sauce in BoomTown’s software testing framework is its use of Sauce Labs VMs to support many automated tests in parallel. This has cut BoomTown’s longest test run time of an hour and a half down to less than 15 minutes.

“With Sauce Labs, it is trivial to spin up any number of parallel VMs, configured against your desired browsers to run through a set of Selenium WebDriver test steps. Without such a solution, BoomTown’s test team would have had to either execute its entire test suite locally through a single machine and browser, or build an in-house testing lab to achieve parallel browser testing,” explains Baumgartner. (Sauce Labs is obviously not the only player with competitors including BrowserStack and TestingBot; let your due diligence be your guide in selecting the best service for your needs.)

BoomTown saw offloading the work as the best option in order to accelerate otherwise time intensive testing tasks, preventing QA from becoming a sort of time-related bottleneck on CD. “We did this simply by increasing our VM count and running more tests in parallel,” says Baumgartner.

Adding VMs and increasing the number of parallel testing instances is an easily scaled approach that continues to speed tests and drive CD even as BoomTown adds tests and lengthens total test runtime.

Using automated parallel tests supported by Sauce Labs VMs frees up QA analysts to be more creative in their testing, finding more and more interesting ways to break the application under testing loads, says Baumgartner; meanwhile the automated testing ‘machine’ verifies the mundane and repetitive tests that the team has already established.

“We recently incorporated automation as a quality gate in our CI process. Before a pull request is accepted, it must pass a set of critical path UI tests that run up in Sauce Labs. This sets a certain level of quality expectations during our daily development process, which saves us time in testing,” explains Gryder.

November 19, 2015
Meeting Challenges in incorporating automated testing throughout the DevOps cycle

Enterprise development teams find it difficult to perform QA tests on software in short DevOps cycles where they cannot squeeze in lengthy regression tests. DevOps teams must insert automated testing approaches to resolve this predicament. “DevOps teams need to be more selective in prioritizing tests that occur at each stage of the integration to delivery pipeline,” says Marc Hornbeek, Senior Solutions Architect of DevOps, Spirent.

At each point between stages of the development lifecycle, there is an opportunity to insert the most applicable tests to determine whether the software has incurred errors. It is crucial to test early. The earlier in the development process that team members discover their errors, the sooner they can fix them, ensuring fewer error-related issues in releases. The earlier in the development lifecycle that errors occur and go unchecked, the more opportunity they have to grow and become increasingly debilitating to the software.

With best practices, DevOps shops can use the right tests in the right stages of development and achieve quality assurance without depending on large test suites that run at the end of the cycle, according to Hornbeek.

The best time to create automated tests is together with or prior to developing the code for the given software feature itself. This will enable development shops to keep pace with the speed of DevOps; the old approach of automating testing after the feature is developed has truly become ineffective, according to Hornbeek.

DevOps Targets Quality / Requires Testing

DevOps has the goal of creating better software faster. Apart from effective, continuous, automated testing at keenly selected intervals, DevOps is only faster. This would achieve only a form of dev similar to what we had before DevOps, i.e., the kind where coders believe they have done well even though operations staff sees a lack of fitness of the software for production use.

Enterprises must embed a culture of QA inside the culture of DevOps rather than separate out QA testing from DevOps using organizational silos; communicating only through inefficient bug reporting methods is no longer the state-of-the-art, let alone practical with the increasing industry expectations of DevOps.

Testing: Why Must I Be Misunderstood?

The impact of testing is often overlooked. DevOps shops that take testing for granted may not invest in or improve upon testing as testing needs change and grow. “This was true before DevOps and continues to be true with DevOps. Developers focus on feature creation, while the business leaders focus on delivery,” says Hornbeek. As a result, testing is often not a focus of DevOps.

Further, there is a common misconception that testing just the part that is done after the builds are created is all that is necessary; but enterprises must integrate continuous automated testing with DevOps throughout the development-to-delivery pipeline, according to Hornbeek.

“When testing is conducted a long time after the feature is developed, there are many inefficiencies. The developers lose context of the problem and have a harder time recreating, diagnosing, and fixing problems,” says Hornbeek. Automated continuous testing eliminates this delay, improves software, and shrinks time to quality software releases.

Slothful Testing Gone Awry

There are high-profile examples of slothful software testing where projects grew out of scope and way off target. A chief example is the original rollout of Healthcare.gov, says Hornbeek. “In another example, a network equipment manufacturer tried to implement DevOps twice without success. They finally realized that they had underestimated continuous testing,” says Hornbeek of a Spirent customer.

To prevent these kinds of outcomes, says Hornbeek, follow these best practices:

• Create automated tests as you develop the features.
• Do not accept features into the integration branch without automated tests.
• Ensure that the developer test environment is a good representation of the production environment.
• Select DevOps-ready tools that have Rest APIs.
• Test priority features so you can easily integrate them into the DevOps orchestration framework.

Follow these steps, according to Hornbeek, to incorporate automated testing for QA:

• Establish a clear leader with responsibility over the entire dev-to-deliver pipeline who is visible and will champion continuous testing.
• Assign an architect for continuous testing tools selection and to oversee the technical implementations.
• Determine best practices for Continuous Testing and decide what practices best fit the organization.
• Address the culture, roles, and responsibilities for continuous testing.
• Put some automated metrics in place, such as a goal to automate 85% of all tests.

Editors Note: I had a chance to catch up with Mark Hornbeek at the Enterprise Summit 2015. Here is my interview with Mark.

November 11, 2015
Ancestry.com Brings BlazeMeter into the Fold

In order to gather and organize your lineage and the branches in your family tree, Ancestry has spent more than a generation (the company was born in 1983) wading through the 16B+ images and records buried in historical society data, government archives, and information caches of like organizations, groups, and communities the world over.

Data in hand, Ancestry transcribed it and converted it into a searchable format for your investigation so you can connect the dots from great, great aunt Tildy to great, great, great grandpa Jebediah. “These historical records date back to the 11th century and form more than 30K collections that include birth, marriage, and death records as well as military service, immigration, and even incarceration records,” says Norman Warren of the Ancestry Performance Team, Ancestry.

Ancestry’s platforms enable it to unite descendants and the curious with true tales from 70M family trees that subscribers build up by adding their own branches. Interestingly, Ancestry data now relies on a million plus DNA samples and counting.

“Our servers respond to more than 75M searches per day,” says Warren, who coaches Ancestry’s teams in refining load testing practices as they release new testing scripts before and after sprints and based on their individual development timing.

Load Testing Before & After

When the Ancestry team went looking for deeper, broader load testing capabilities, they were hoping to add routine testing across the site 365-days a year. The folks at Ancestry wanted a malleable solution that would invite them to apply JMeter to vertical load testing and permit them to cut costs for load testing by leveraging the advantages of open source testing tools.

Ancestry had been using a load testing approach that did not allow Warren’s teams to use home-brewed scripts. But in order to benefit from a testing solution that did embrace such scripts, Ancestry had to ensure a flow of updated quality scripts from every applicable internal group. Because Ancestry ultimately opted for JMeter and BlazeMeter, that presented challenges such as JMeter and BlazeMeter learning curves, the need for users to make JMeter requests in parallel, and Ancestry apps that are harder to script.

Growing pains accompanied BlazeMeter adoption including the pains of stretching to embrace different devops teams in order to retrieve testing scripts. By pointing out the advantages of the new testing paradigm, making the argument for time saved by site wide load testing, and providing training in JMeter, Ancestry made the leap to the BlazeMeter supported approach. “BlazeMeter provided excellent JMeter training while we evangelized for teams to own their scripts,” says Warren.

Ancestry solved its technical and learning curve challenges with steps including installing the JMeter Webdriver Set plugin and learning to debug and test scripts before they would run optimally in the BlazeMeter cloud service.

“We had to work to get adoption, but it was well worth the effort to build accountability over different sections of the site. We now ask teams to take ownership of their own scripts to review, modify, or emulate user traffic flows,” says Warren.

Results

“BlazeMeter enabled more frequent load testing at a competitive cost and good site-wide load testing with the ability to add loads to stacks or to stop loads on one of the stacks during the test,” says Warren.

BlazeMeter helped the Ancestry teams to foster a strong confidence that they could support heavy traffic loads at their most hectic times. “The BlazeMeter team helped us to translate the metrics we use for measuring traffic into a number of concurrent users for loading the site. In this way, BlazeMeter helped us to measure load test results against a baseline, such as traffic on our most recently busiest day,” says Warren.

Ancestry leveraged BlazeMeter’s swift feedback about site throughput and the response times of each page to speed site responsiveness for each user. Ancestry was able to predict and automate the flows that are most commonly used by customers in order to streamline processes and speed the user experience even more.

“We created a new baseline from new site wide load tests of additional traffic flows by loading flows incrementally using a special BlazeMeter feature that enables us to dynamically add more concurrent users in the midst of a load test,” explains Warren.

November 6, 2015
Continuous Testing: What exactly is it?
***This post is based on themes and ideas from the great e-book, “Continuous Testing,” by our friends at Parasoft.

The aptly named DevOps practice of Continuous Testing synchronizes Testing/QA with Dev and Ops processes optimized to achieve business and development goals.

A few years ago when Agile was all the rage, lightning-fast Dev sprints left the Ops and QA personnel playing catch-up due to inadequate and sluggish testing practices with minimal coverage across frequent builds.

The widespread culture of Agile development has accelerated development while software testing still lags behind, forcing organizations to cut corners in QA or slow down Dev processes entirely.

Instead of speeding up the conveyor belt beyond control or slowing down development to kill efficiencies, Continuous Testing in DevOps encourages a systematic approach toward process improvement.

Continuous Testing goes beyond automation and encompasses all practices — including tooling and cultural change — that help mitigate risks before progressing to subsequent Software Development Lifecycle stages.

Continuous testing and Automation

The approach to Continuous Testing can vary and follow diverse pathways to ensure the best user experience is delivered, free of defects.

It’s barely viable to repeat all tests every time a new feature is added, so the Continuous Testing strategy fosters a company-wide cultural change to achieve four capabilities: Test early, test faster, test often and automate.

End-to-end test automation practices are intended to integrate QA into existing fast-paced Dev and Ops processes as a means to create continuity while maintaining faster development cycles.

It’s important to understand the role of automation in Continuous Testing. Automation alone does not enable continuity in testing, but it helps provide a qualitative assessment of risk and practice actionable tasks to mitigate these risks throughout the SDLC.

In essence, automated testing constitutes the detection process for software issues and defect prevention, whereas Continuous Testing addresses the wider challenge of improving the effectiveness of these detection sensors.

What does Continuous Testing Look Like

This example demonstrates how collaboration, automation and tooling enable end-to-end testing that’s performed faster, more often and better aligned with Agile Dev and Ops practices.

Continuous Testing starts within development processes as Devs use open source tools such as Selenium to test the functionality of their code.

Tools such as GitHub or other internal repositories can then be used to store tests and version together with the software code. DevOps teams can use the same tests to integrate frequent builds of the software with the Load Impact API.

Once the code reaches preproduction, dedicated QA teams can use these tests by modifying dynamic test parameters as appropriate. In the production environment, Ops can reuse these tests for acceptance testing and ongoing post production monitoring.

Once the application goes live, integration with existing APM tools help keep tabs on server-side app metrics.

Continuous Testing in this example is broken up into at least four environments: Development environment, Continuous Integration environment, QA Pre-production and Performance testing environment, and the Staging environment. Additional post-live monitoring further enable end-to-end continuous testing.

Key Elements of Continuous Testing

Parasoft, a global software solutions company, is one of the great thought leaders in DevOps and Continuous Testing. The company has laid out some key elements of the process in a fantastic e-book, and they’re definitely worth sharing here:

Risk Assessment: Covers risk mitigation tasks, technical debt, quality assessment and test coverage optimization to ensure the build is ready to progress toward next SDLC stage.

Policy Analysis: Ensure all processes align with the organization’s evolving business and compliance demands. Primary objectives include:
- Identifying trends associated with injection of dangerous patterns within the code
- Augmenting defect prevention practices for high risk areas
- Isolating risk in target areas
Requirements Traceability: Ensure true requirements are met and rework is not required. An object assessment to identify which requirements are at risk, working as expected or require further validation.

Advanced Analysis: Using automation in areas such as static code analysis, change impact analysis and scope assessment/prioritization to prevent defects in the first place and accomplishing more within each iteration.

Test Optimization: Ensure tests yield accurate outcomes and provide actionable findings. Aspects include Test Data Management, Test Optimization Management and Test Maintenance.

Service Virtualization: Early, frequent and everywhere testing requires access to real-world testing environments. Service visualization enables access to the virtual form of the required testing stages, cutting the waste time to test environment setup and availability.

An effective Continuous Testing strategy incorporates all of these elements and forces an organization-wide cultural change to synchronize Dev, Ops and QA/Testing as part of the true DevOps philosophy.
May 27, 2015