devopsy.kinsta.cloud

Category: Continuous Testing

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.

  • Gremlin Brings Chaos Monkey Testing to Spinnaker CD Platform

    Gremlin Brings Chaos Monkey Testing to Spinnaker CD Platform

    Gremlin this week announced it will integrate its hosted chaos engineering service with continuous delivery platforms starting with the open source Spinnaker project developed by Netflix.

    Chaos engineering traces its roots to a resiliency testing philosophy that posits applications should be able to keep functioning regardless of any service or infrastructure failure. Most IT organizations, however, don’t have the expertise required to build the tools required to testing resiliency based on those “chaos monkey” principles that randomly turn services off to see if an application fails.

    Matt Jacobs, principal software engineer for Gremlin, said the decision to integrate with Spinnaker is being driven by the need to infuse chaos monkey resiliency testing directly into the application development and deployment pipeline at a more granular level. That capability is now a bigger requirement than ever because as organizations embrace microservices to build the next generation of cloud-native applications, it’s simply not possible for DevOps teams to manually track every dependency, he said, adding resiliency testing based on chaos monkey principles is just another type of testing that should be automated as part of a larger continuous integration/continuous delivery (CI/CD) platform to ensure an application doesn’t fail completely.

    Integrating with a Spinnaker project that has drawn support from Google, Microsoft and Oracle provides the easiest path to achieving that goal. Spinnaker is also now a core element of a Continuous Delivery Foundation led by Alibaba, Anchore, Armory, Autodesk, Capital One, CircleCI, CloudBees, DeployHub, GitLab, Google, Huawei, JFrog, Netflix, Puppet, Red Hat, SAP and Snyk.

    The Gremlin service is accessed via a user interface and comes in two forms: a free edition and an enterprise edition, which comes with support from Gremlin. While the concept of chaos monkey testing has been around for some time among the most advanced DevOps practitioners, enterprise IT organizations, which tend to be more conservative when it comes to introducing anything that might deliberately break their applications, will need a considerable amount of hand-holding. Gremlin is designed to learn the application environment in a way that introduces chaos in a controlled fashion before an application is deployed in a production environment.

    On the positive side, resiliency testing based on chaos monkey principles tend to accelerate the rate at which IT staff learn how an application has been architected, said Jacobs. That can prove especially invaluable when training new members of a DevOps team, he noted.

    Ultimately, the decision to embrace resiliency testing comes down to how proactive an organization wants to be when it comes to preventing downtime. When mission-critical applications are offline, there is a direct impact on the amount of revenue an organization generates. There are plenty of functional and performance testing tools, but the real difference between having an issue and a catastrophic event that requires DevOps teams to get out of bed in the middle of the night often comes down to how truly resilient any given mission-critical application really is.

    — Mike Vizard

  • Sauce Labs Combines Functional and Performance Testing

    Sauce Labs Combines Functional and Performance Testing

    Sauce Labs announced it is now moving beyond functional testing, adding a front-end testing tool it has dubbed Sauce Performance to make it easier to accelerate testing within the context of a DevOps process.

    Lubos Parobek, vice president of products for Sauce Labs, said the goal is to enable developers or testers to begin testing both how well application functions work and perform much earlier in the DevOps process using the same testing tool.

    Parobek said that as organizations embrace modern application development architectures based on microservices, it’s become crucial to test the capabilities of any module within an application because the number of interdependencies between modules continues to increase. That approach accelerates the rate at which applications can be developed. However, any flaw that impacts how the application works or performs can have a debilitating impact on any number of microservices that make up the application, he noted.

    Sauce Performance makes use of debugging and performance trace logs to identify the root cause of an issue in addition to capturing metrics such as Time to First Meaningful Paint, Time to First Interactive, Page Weight and Speed Index. Sauce Performance also employs algorithms to automatically establish a baseline that can be used for regression testing.

    Parobek said Sauce Performance serves to streamline testing processes as well. Most existing front-end performance testing solutions require integration with a load/stress testing tool or a separate application performance monitoring (APM) tool. However, Sauce Performance allows developers to use their existing Selenium test scripts to capture both functional and performance test data.

    In addition, Sauce Performance has been designed to integrate with multiple continuous integration/continuous deployment (CI/CD) platforms to make it possible to feed test results directly back to developers, which can then decide to prioritize issues that need to be addressed.

    Sauce Performance is available as part of the Sauce Labs Continuous Testing Cloud.

    Parobek said organizations want to be able to make it easier for developers to test their own code as part of their DevOps best practices. That doesn’t eliminate the need for dedicated testers for evaluating functions, but potential performance issues today are being addressed often long before applications are deployed in a production environment.

    One of the reasons applications are not as thoroughly tested as they might be is that many organizations require developers to learn how to use tools that have different user interfaces and then switch between testing tools. The result is a complex testing process that makes it more tempting to short-shrift elements of the testing process as application delivery deadlines loom.

    More challenging still, many organizations still don’t continuously test applications after they are deployed to ensure new functionality does not adversely impact the user experience.

    For as long as applications have been built, tension has existed between the developers who create applications and the people who test them. That debate may never be resolved, but everyone agrees in this age of DevOps that it’s better to test early and often.

    — Mike Vizard

  • Compuware Advances DevOps on Mainframes

    Compuware Advances DevOps on Mainframes

    Compuware has updated its Topaz for Total Test suite to enable organizations applying best DevOps practices to the development of mainframe applications to combine unit, functional, integration and regression testing within a single test scenario.

    In addition, Compuware is making it easier to test multiple applications simultaneously. The latest update to Topaz for Total Test adds the ability to not only orchestrate a sequence of steps, but also design tests to measure how programs interact with each other and various data sets.

    Also added is the ability to automate several testing functions, including the ability to launch functional tests within a browser and execute unit, functional and integration tests without having to manually create their own Job Control Language (JCL). According to Compuware, testers now can design test cases upfront based on program specifications before developing a program. What’s more, Topaz for Total Test has been integrated with the DevOps pipeline management platform developed by XebiaLabs.

    Finally, Compuware announced that the Topaz 3270 Emulator now makes it possible complete one-off “green screen” tasks without leaving Topaz. Initiating a CICS transaction or accessing REXX utilities through the Emulator within Topaz, for example, makes it simple for developers to quickly resume Eclipse-based tasks without having to exit its testing environment or toggle between tools.

    Compuware CEO Chris O’Malley said the pace at which IT teams in mainframe environments have been adopting DevOps practices has accelerated considerably in the past year. Not only are applications being developed and deployed faster, the quality of those applications has increased because DevOps tools provided by Compuware make it easier to both visualize a complex application development process and then test those applications as they are being developed. Given the often mission-critical nature of mainframe applications, it’s crucial that any increase in development velocity doesn’t have a negative impact on application quality, he noted.

    DevOps has become a greater imperative in mainframe environments, O’Malley added, as organizations look to leverage investments in existing mainframe applications rather than try replicating functionality that already exists on a mainframe in a distributed computing environment. That approach helps to reduce sharply the amount of technical debt an organization needs to service when they duplicate functionality across two or more platforms. Mainframe teams that historically have relied on waterfall developments process have been challenged by keeping pace with the rate of change. As applications that are deployed on a distributed system look to invoke a mainframe application function, organizations that have deployed those applications need to keep pace with a much faster rate of development, especially when it comes to application programing interfaces (APIs).

    It may take a while longer for DevOps practices on mainframes to become adopted widely. But as the rate of innovation continues to increase, thanks to the rise of various digital business transformation initiatives, it’s apparent that mainframe application development will need to become more agile. The challenge at this point is not so much the availability of DevOps tools for the mainframe, but rather training the teams that build and deploy those applications on when and how to implement DevOps practices.

    — Mike Vizard

  • Perfecto Applies AI to Writing Test Scripts

    Perfecto Applies AI to Writing Test Scripts

    Artificial intelligence (AI) is starting to be applied to the writing and update test scripts. Perfecto has unfurled Perfecto Codeless, which employs machine learning algorithms to automate writing of test scripts in a way that allows them to not only run continuously, but also adjust to changes made to the application.

    Eran Kinsbruner, chief evangelist for Perfecto, a unit of Perforce, said writing test scripts is still largely a manual process, which means as the number of applications being developed continues to escalate, the testing process becomes highly constrained.

    To address that issue, Perfecto has created a codeless approach delivered via a cloud service as part of the company’s Smart Automation platform. If any code within an application gets deleted, moved or changed, the changes will automatically be reflected in the test script. That capability eliminates the need for DevOps teams to wait for a developer to update the test script.

    Exactly who in an organization is writing those test scripts varies: Kinsbruner noted that many developers now prefer to write their own test scripts during the development process before their code is reviewed by a peer. That approach eliminates many of the more routine errors, enabling more time to be spent on optimizing code. Other organizations employ dedicated testers who spend most of their time writing test scripts. But many of those organizations are finding that testers cannot keep pace with the current rate at which applications are being developed in the DevOps era.

    There’s no doubt machine learning algorithms are about to play a much bigger role in application testing and DevOps at large. AI is not going to replace DevOps teams anytime soon, but many mundane functions such as the writing of test scripts can be automated using machine learning algorithms. Arguably, testing is one of the areas that lends itself most to emerging AI-based platforms and services. After all, if testing becomes a bottleneck, the chances that applications with issues are deployed in a production environment increase substantially. Many of those issues can be fixed easily, but many times at the risk of annoying end users who increasingly are unforgiving when it comes to application experiences. It’s not uncommon for users to discontinue using a mobile app if the first experience isn’t immediately gratifying.

    Of course, many DevOps teams already view machine learning algorithms with a certain amount of skepticism. But the rise machine learning algorithms within DevOps is now all but inevitable. IT leaders, however, should realize it may take a while for those algorithms to learn the environment. The return on investment for any AI project is not likely to be immediate because a significant amount of time needs to be invested in training the algorithms. The good news is, those algorithms never take a day off and suddenly don’t resign one day to take a better-paying job elsewhere.

    — Mike Vizard

  • 5 Steps for Successful Continuous Testing Implementation

    Continuous integration (CI) calls for code to be always working. Continuous delivery (CD) and continuous deployment mean that we must be able to roll out a working configuration to production at any time. As DevOps becomes the norm, bugs in production are now a developer’s personal problem. Developers (and their managers) want to know immediately if code changes they are working on can break something.

    What is Continuous Testing?

    The only guaranteed way to prove code and configurations are working is to test them. That’s why we need continuous testing. In organizations making this transition, performance engineers and test automation experts shift from minding a backlog of test-building work, to facilitating a democratization of testing. Everybody builds their code and everybody tests their code, continuously.

    Continuous testing is becoming the “new normal.” In this modern and agile testing approach, developers are equipped with low-friction testing capabilities and tools and are expected to unit test any new or updated code. With testing shifted all the way left to developers in pre-production, and all the way right to post production monitoring, organizations are just seconds away from proving code is ready and working to acceptable service levels.

    Although continuous testing is becoming a more standard practice, it certainly presents unique challenges for execution.

    Here are five steps to help any company successfully implement a continuous testing policy:

    Define Tests Early

    Lack of clear requirements can result in wasted time and delays. Behavior-driven development (BDD), acceptance test-driven development (ATDD) and model-based testing using tools should be used so that all requirements are clearly documented and communicated.

    Test cases need to be clearly defined and test scripts created ahead of time, to enable continuous testing of code at all stages of production.

    Optimize Testing Processes and Test Coverage

    Save time, money and resources by testing only what you need to test. Visual models allow various paths to be explored and optimized so that only your tests provide maximum coverage, without unnecessary testing.

    It is also recommended to import test cases, remove duplicates and distribute and allocate optimized test cases in existing tools.

    Shift-Left Your Testing (and Shift It Right as Well)

    What do we mean by shift-left testing? Tests should be run earlier in the development cycle. Developers test as they go, with test automation including functional testing, performance testing, monitoring and security testing. This requires developer-friendly tools for easy adoption and use.

    By shift-right testing, we continue to test through production and post-production, including continuous monitoring. Gartner recently released a report on the shift-right testing, calling it a “must” for adopting DevOps practices.

    Provide Complete Test Environments

    The ability to provide virtualized test environments is critical to achieving continuous testing. Eliminate blocks and reduce wait times by providing complete test environments on demand, with dev-friendly tools (as code, CI/CD integrations, supported open source).  These environments should include test data on demand to ensure that teams can perform comprehensive tests with production-like data. These environments should be ephemeral; ready on demand, decommissioned after use.

    Get the Right Test Data

    The inability to get robust test data causes significant delays in many application release cycles. To accurately test new functionality, test data should be as close as possible to what the application will encounter in production. If the test data lacks certain real-world characteristics (i.e. actual fields, data specifications, negative scenarios, etc.), the tests are unlikely to find many potential issues or break the application where there are weak points. Understanding data models in order to extract the right data is a special skill in and of itself.

    Production data is clearly the best data to use for testing, but data privacy regulations and team silos often restrict its availability. The answer to this dilemma is a robust test data management tool that lets you copy production data and mask it to protect sensitive information, while maintaining the characteristics that make production data desirable for testing.

    Remember, continuous delivery and continuous testing are processes that can take time to implement successfully, but teamwork, continuous feedback based on data and the right tool set will help you on your journey. You can read more about implementing a continuous testing policy here.

    — Sushil Kumar

  • SauceCon Speaker Profile: Richard Bradshaw

    SauceCon Speaker Profile: Richard Bradshaw

    This is the sixth in a series of articles highlighting speakers at SauceCon 2019. SauceCon brings together the global community of Sauce Labs users and continuous testing experts, to learn from each other and level up their automated testing and continuous delivery skills. This year’s event will be held April 23-25 in Austin, Texas.

    Richard Bradshaw is known for running the Ministry of Testing, speaking at events all over the world and for being the “Friendly Tester”—a nickname given to him by a developer.

    However, one thing toward which Richard isn’t 100 percent friendly is the Test Automation Pyramid. He says it can and should bring about conversations, but thinks it lacks too much to function as a solid automation strategy, which is how many people use it. At SauceCon 2019, Richard will present some other options in his talk, “Pyramids Are Ancient – Let’s Talk Automation Strategy.”

    Richard hopes that people will take away the fact that to truly be successful with automation, you need to fit automation into your overall testing strategy. During his talk, he will review the steps that he takes when conducting an automation audit, steps attendees will then be able to immediately take back to work and begin applying. “I also hope that observing me critique existing models will motivate them to do the same to some of the models used in their current workplace, and identify ways to improve their approach,” he said.

    Richard is well-qualified to help testers improve in this area. As the BossBoss (aka CEO) of the Ministry of Testing, his primary role is to ensure they stick to their mission of “co-creating better testers.” Richard and his team offer numerous ways to learn and share testing knowledge. They run the TestBash conference, manage an online learning platform called The Dojo and host meetups all around the world.

    When he is not working or thinking about work (a rarity!), Richard loves cooking up a storm, hiking and traveling.

    You can hear from Richard and more than 30 other expert speakers at SauceCon 2019, the third-annual Sauce Labs user conference. To see a full list of presentations or to register, visit the SauceCon website.

    — Rebecca Cramer

  • Why 2019 Will Be the Year for Shift-Left Mainframe Testing

    Why 2019 Will Be the Year for Shift-Left Mainframe Testing

    While 2018 was the year for planning and implementing shift-left methodologies in testing, mainframe and server testers were, for the most part, left behind. These legacy infrastructure experts were tied down to old-school testing tools. Mainframes kept functioning, but tools and testing practices often became bottlenecks, preventing performance testing teams from testing more quickly mid-cycle and pre-release.

    Until now.

    The future of testing is in open source, and finally there are open source solutions being developed and tailored for mainframe testers. Now, mainframe testers can also shift left and become agile.

    Here are four reasons why mainframe developers and testers will benefit from shifting lift and embracing open source testing technologies:

    Frequent Releases

    Apache JMeter, BlazeMeter (open-source based), Taurus and Jenkins are ideal substitutes for legacy tools, for two reasons. These open source tools are very easy to use, so developers and testers can use them themselves. In addition, they can be automated easily—once scripts are set up, they just run, and developers can get the results and monitor the system and their code.

    These two reasons are the building blocks of what shifting left is all about: frequent releases. There is a much quicker option than developing, waiting for QA to run tests, arguing with QA about the results, fixing the code, waiting for test results again, realizing you forgot to commit an important part of the fix, sending the code again, waiting for tests and pulling an all-nighter to fix the code for the last time only to realize that your code clashes with a team member’s code and you can’t release.

    By integrating your code tests into the development pipeline, every piece of code is immediately and automatically tested, on its own and encapsulated with other parts of the code. Once the code is approved, a new version is immediately released, to the customer’s/user’s benefit. This also benefits the developers by giving them a clean slate to continue working on. If the code needs fixing, no problem—an alert is sent to the developer, who can easily identify the problem and fix it. The fixed code is immediately tested and released. Now that’s much easier and much more motivating for everyone involved.

    Fewer Bugs, Higher Code Quality

    Shifting left means more automated tests are run—and more frequently. By constantly checking the system for errors and by running tests every build, you and your team are constantly being alerted of errors, which you can then immediately fix. The result is higher quality code and much better system performance.

    Another positive outcome is less hassle for your team when developing, because you don’t have to keep dwelling on previous version errors. Those have already been fixed or dealt with, so you can focus on future plans and exciting developments, which will materialize in higher code quality.

    Cost Reduction

    Shifting left means faster and automated testing, resulting in a higher-quality product. By running more tests in parallel all the time, testers and developers can focus on developing new code, resulting in savings for the business. If you’re running tests in the cloud, you’re also saving infrastructure costs.

    A higher velocity of releases with quicker fixes and new features results in more satisfied customers, who stay with you instead of moving to your competitors. Customers stay with companies they feel is concentrating on the future, which is why Gartner’s Magic Quadrant “Visionary” recognition is considered to be a big deal for the testing industry.

    Better Team Collaboration

    The idea of automation and continuous testing might come across as being detrimental to teamwork, but the opposite is true. Even though developers become more self-sufficient with their own code, this complete ownership actually encourages teamwork. Team members will collaborate more on joint feature development through shared workspaces and dashboards, with each developer bringing the code they produced to integrate with others. They also will work together to discover the system’s shortcomings and learn how they can fix bugs and bottlenecks.

    Shift-left and automation for mainframe means more time for feature development, getting customer feedback and responding to it; quicker fixes; and faster releases. This is crucial for competing in the market, providing the best services, attracting the highest-quality workforce and maintaining product security.

    Start Your Mainframe Testing with Open Source Tools

    BlazeMeter recently contributed the JMeter RTE plugin to the open source community. This plugin enables load and functional testing on JMeter for IBM 3270- and 5250-based applications.

    Running mainframe tests enables mainframe developers and testers to finally introduce agility to their testing habits, bringing mainframe testers in line with their coworkers who already embraced shift left and open source testing toolkits.

    — Refael Botbol Weiss

  • CXOs Overestimating DevOps and Continuous Testing Maturity

    CXOs Overestimating DevOps and Continuous Testing Maturity

    More work is needed to achieve continuous testing, Forrester report shows

    Forrester recently completed an interesting new research project exploring what separates DevOps leaders from DevOps laggards. After analyzing data collected from 603 global enterprise respondents responsible for their firms’ DevOps strategies, Forrester learned that firms with the most mature DevOps practices do several key things differently:

    1. Allocate proper testing budgets and focus on upgrading their testing skills.
    2. Implement continuous testing to meet the demands of release frequency and support continuous delivery.
    3. Include testers as part of their integrated delivery teams.
    4. Automate end-to-end functional testing.
    5. Shift left testing to earlier in the development life cycle.

    Firms who follow these five best practices experience testing as a way to gain speed and efficiency, rather than a bottleneck to delivering faster.

    This was not terribly surprising. On the one hand, study after study shows that testing is the No. 1 bottleneck to accelerated application delivery. On the other, we’ve seen firsthand how organizations that commit to transforming testing successfully shift from waterfall-style manual testing processes to automated continuous testing integrated into CI/CD pipelines.

    What caught us off guard, however, was CXOs’ highly inflated estimation of their continuous testing, DevOps and Agile maturity.

    According to the report, only 26 percent of non-CXOs reported that their organization was following all five of these best practices—but 57 percent of CXOs believed that they were. According to the report:

    While most Agile and/or DevOps firms have implemented some of these five core testing practices, few have actually implemented all of them — only about a quarter of respondents say they somewhat or completely agree with all five best practice statements.

    This is an important distinction. While many CXOs and high-level decision makers believe they are ahead of the pack, our research shows that only the small fraction of firms that follow these best practices have more advanced Agile and DevOps practices compared to their peers — and their behaviors and attitudes support that assertion.

    Moreover, CXOs also reported much higher levels of Agile and DevOps maturity across the board than non-CXOs did.

    The good news for DevOps professionals is that CXOs are your champions: They believe you’re doing a great job! The bad news is that, at some point, you need to tell them about the gaps that currently exist in the process. You also need to collect and share some of your own data to make the case that actually adopting these five proven best practices will help you increase delivery speed while reducing business risk.

    Detailed Report Findings

    This was just one of the many interesting findings from research project, which was led by Forrester Vice President and Principal Analyst Diego Lo Giudice.

    As a high-level summary, the research uncovered the following:

    • Understanding of business risk is the most important determining factor of DevOps and Agile maturity.
    • Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).
    • Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

    Detailed results and analysis are available in the complete report: “Forrester Research: What Separates DevOps + Agile Leaders from Laggards?”

    Report Methodology

    In this study, Forrester conducted an online survey of 603 enterprise organizations in North America, Europe and Asia Pacific to evaluate current software testing practices and metrics tracked during survey development. Survey participants included decision-makers and individual contributors responsible for their organizations’ Agile and/or DevOps development strategies. Questions provided to the participants asked about their firms’ attitudes toward software development automation, risk management and testing practices, as well as the metrics they track and value in the software development life cycle.

     

    — Cynthia Dunlop

  • OverOps Employs Machine Learning to Measure Application Reliability

    OverOps Employs Machine Learning to Measure Application Reliability

    OverOps is adding a series of dashboards to its analytics platform that employ machine learning algorithms to identify which software application anomalies in a pre-production environment are likely to have the most impact on a production environment.

    Company CTO Tal Weiss said one of the biggest challenges DevOps teams face is prioritizing which issues to fix first in an application. The Reliability Scoring capability being added to the company’s namesake platform for analyzing code employ machine learning algorithms to keep track of introduced errors, regressions and slowdowns during every phase of pre-production. In addition to enabling DevOps teams to visually identify the root cause of an issue via a single click, that data is then used to identify which issues will impact reliability.

    Weiss noted that the Reliability Scoring can also be employed to set thresholds of acceptable application performance that would also be used to determine whether there is an issue. The data collected using that capability then can be fed back into a continuous integration/continuous deployment (CI/CD) platform to set up gates that would then prevent suspect code from being promoted, he said.

    The primary benefit, however, should be a major reduction in the number of troubleshooting instances any DevOps team should need to engage in after an application is deployed in production, said Weiss. In fact, the Reliability Dashboards will provide a central mechanism through which developers, testing professionals and IT operations teams can collaboratively track and measure the progress of an application development project, he added.

    The Reliability Scoring tool OverOps developed employs “micro-agents” that operate between application code and the hardware. Weiss said those micro-agents can not only capture data that was never previously available, they deduplicate, classify and gate critical issues as they move into staging and production. The Reliability Dashboards visualize that data in Grafana, a set of open source tools for visualizing data. DevOps teams can also opt to feed that data into any existing dashboards they may already have in place.

    In general, Weiss said the metrics surfaced by the Reliability Dashboard are critical to justifying investments in DevOps over the long term. Organizations expend a lot of time and effort making the transition to DevOps. But most of them don’t really know to what degree they are getting better at building and deploying applications. Reliability Scoring dashboards provide the historical context required to make those assessments, he said.

    Far too many DevOps teams are still starved for data needed to make critical decisions. In the absence of that data, teams address issues as they arise, rather than based on the impact they might have on the business. Worse yet, it’s difficult to identify new issues over those that might be another manifestation of a known issue that suddenly becomes a much higher priority.

    Regardless of the root cause of any problem, however, DevOps teams would do well to remember things that get measured are usually the first things to get fixed.

    — Mike Vizard

  • TCoE as a Way to Level Up Test Automation

    TCoE as a Way to Level Up Test Automation

    A testing center of excellence (TCoE) can help you leverage automated testing efforts.

    The “World Quality Report 2018-19” claims that test automation is almost a heal-all concept in QA and one of the most booming trends for the near future. Still, the general percentage of automated testing in overall test efforts is quite modest. Apart from technical and financial impediments, test automation can be hampered by a number of organizational challenges. Here, testing centers of excellence (TCoEs) may be of practical value.

    Let’s find out what TCoE is, why test automation is a must and how TCoE may come in handy on your way to higher test automation levels.

    TCoE: Capsule Review

    A TCoE provides QA optimization and control due to consolidated testing best practices, tools, standardized methodologies and testing metrics. It aims to centralize QA resources dispersed around the whole enterprise, thus being more relevant for large-scale organizations with a number of departments or even subsidiaries. Capgemini research results prove that TCoE can bring such tangible benefits as optimized QA resources (people, budgets and tools), reduced testing time (by 30 percent) and costs (by 35 percent in a three-year time frame) and increased test automation rate (up to 50 percent).

    Why Level Up Test Automation

    Test automation is not just hype, but a proven way to get more from your testing. The ever-growing popularity of Agile and DevOps/continuous delivery projects necessitates that testing be performed within a shorter time frame. Test automation tools are highly efficient in performing the most time-consuming types of testing and the ones comprising large amounts of data, such as regression, performance, data warehouse or load testing. With its ability to execute thousands of test cases during every run, test automation provides coverage unachievable with manual testing resources.

    Test Automation Challenges That TCoE Can Solve

    The promotion of test automation is acute as it can result in tangible benefits. For now, the average level of automated test activities is only around 16 percent, as reported by the leading CIOs from around the world. However, it is pointless to automate all testing activities. Test automation is not a panacea for every testing project or testing type. Each project’s automated testing efforts require a careful consideration and ROI assessment, as test automation has a number of limitations. For example, it can save the testing budget only in the long run. So, there’s no need to introduce automation for small-scale testing projects. Besides, even in projects with a large testing scope, test cases that are unique for each release or source code modification, rarely repeated test cases and those with difficult-to-measure or not purely objective output shouldn’t be automated.

    However, the low rate of test automation comes not only from its natural technical impediments but also because of the organizational problems bound. Luckily, TCoE can handle some of test automation organizational challenges.

    Test Strategy Lacking Test Automation

    If the test strategy doesn’t fully correspond to software requirements and its peculiarities, it may lack testing types appropriate for automation. TCoE may be of help here as TCoE managers can assist test leads in building a more effective test strategy with respect to high-level QA priorities and help BAs to come up with multilevel software requirements, if a software requirements specification turns out to be incomplete.

    Besides, the initial part of test automation requires a significant share of testing time (choosing the repeatable test cases and scenarios, scripting and maintaining them) and budget. Without TCoE, some test leads may try to economize and strike automated testing off the test strategy, sure that the moves will help them meet the rigid deadline and budget. If it’s truly impossible to implement relevant test automation and ensure acceptable testing quality stays within time and budget, TCoE managers can suggest revising the testing time and financial limits for the project.. The alternative, of course, often results in even more expenditures because of the release of heavily bugged software.

    The Lack of Skilled Test Automation Engineers

    QA teams consist mostly of manual test engineers without code writing skills needed to script and maintain automated test scenarios, while developers don’t have the knowledge of relevant automation testing tools or the experience in creating appropriate test cases. So, neither manual test engineers nor developers can fill the gap without sufficient training.

    TCoE can help, with its managers knowing each test engineer and having an established knowledge-transfer process. TCoE managers can find test team members (preferably, manual test engineers) who are enthusiastic about test automation and provide training to them on a broadly applicable programming language (for example, Java or Python) and the most adjustable test automation tools (Selenium, Ranorex, TestComplete, etc.).

    Silo Testing Processes

    Uncoordinated and varied testing processes can hinder test automation. In different QA teams, test leads may choose disparate test management tools and test automation engineers are likely to use diverse test automation frameworks. Besides, with QA resources being dispersed around the company, it’s complicated to create a common base of automated scripts that can be reused in different testing projects.

    Efficient test automation requires unified testing processes around the enterprise, which can be attained with TCoE. From the earliest stages of its maturity, TCoE focuses on aggregation, analysis and standardization of testing processes and test automation tools and frameworks. In addition, the TCoE team captures test patterns, creates modular automated test scripts and gathers them in a library to be fully or partially used through the same testing types or similar testing projects.

    Excessive Test Automation Spend

    Test automation may be pricey due to automation software license costs. Software license purchase for almost every testing project anew can turn test automation into a nightmare for the finance department and lead to excessive testing costs. TCoE handles this challenge by providing conventional licensed test automation software tools that can be used for a number of projects.

    Also, test automation requires ongoing engineering resources for maintenance. Application under test (AUT) changes from iteration to iteration and all the automated scenarios should be checked for their validity. Only a few of them can remain unchanged while the majority needs restructuring and correction. A number of TCoE’s qualified test engineers ready to cater to continuous automated testing can be relocated to the project where they are most needed, not being bound to a distinct one. Besides, common test cases and scripts that were automated once can be reused through different projects.

    Wrong Time for Test Automation

    Another test automation pitfall is the difficulty of choosing the right time to introduce it during a development life cycle. If a test strategy implies a too-early start with scripting test cases, a test automation engineer risks finding the prepared test scripts and suites outdated in the next iteration due to requirements changes. At the same time, starting with test automation too late may result in excessive testing time and costs.

    TCoE consolidates test automation best practices and QA professionals with the deep knowledge of software and its requirements. Thus, it’s much easier for TCoE managers than for each test lead separately to evaluate a certain project and decide when it’s the safest time to develop and implement automated tests.

    On a Final Note

    The relevance of test automation is here and is predicted to gradually grow in the future. Still, automated testing has a number of constraints and trying to reach higher levels of its implementation is bound to organizational challenges. TCoE can assist in solving such organizational test automation problems as an unbalanced test strategy, the lack of test automation engineers, uncoordinated testing processes, excessive test automation costs and inappropriate time for test automation.

    — Andrei Mikhailau