devopsy.kinsta.cloud

Tag: testing automation

  • This is Not Just a Test: DevOps and the Need to Automate

    This is Not Just a Test: DevOps and the Need to Automate

    Creating software is as much about the process as it is about the product. In recent years, we’ve seen the speed of innovation reach new heights due to more efficient development and quality control. DevOps has further accelerated development by combining software and operations in unified, self-improving teams. With automation taking care of rote tasks, developers can now tackle more difficult issues, resulting in even greater software quality and faster development. The need for speed to market is vital; a month or two can make or break a product launch. Then again, if a product isn’t tested thoroughly in the rush to get to market, flaws can slip through and an organization’s revenue and reputation can take a serious hit. In the U.S., software failures in businesses’ operations systems cost a total of almost $1.6 trillion in 2019. The problem is that it can be a bottleneck and create major delays in the release cycle. In fact, A recent survey of 4,300 developers revealed that it is the top reason that releases are delayed.

    In the past, the waterfall approach was common. Tests usually took place after code was written; carried out by testers who had no interaction with the original developers. As a result, bugs that could have taken moments to correct during development required days or even weeks to fix afterward. Even worse, the complexity of the finished software and its many dependencies meant that fixing a localized bug could introduce additional problems that needed to be fixed after another round of testing, further delaying the release.

    But even in a modern DevOps environment, testers and developers are still speaking entirely different languages. By and large, testers do not code and software developers are not expert testers. The two teams typically converse through multiple logs, event viewers and other operating system tools. It creates a great deal of friction and slows down the process.

    Shift Left 

    DevOps represented a big improvement over waterfall methods but, even so, it’s far from a set-in-stone process. It, too, has evolved and right now, shift left testing has moved into the spotlight.

    In this approach, the guiding philosophy is to test early and often, beginning very early in the development cycle and continuously throughout every stage whenever there’s an opportunity. As a result, bugs are caught early when the relevant code is still fresh in the developers’ minds, so it’s easier to fix. In the end, businesses are able to release higher quality software much faster and more frequently.

    Environmental Tests

    When it comes to the DevOps process, it’s not always clear where and how to automate the process. The following looks at how that can be incorporated into DevOps based on four different environments in a continuous delivery (CD) pipeline.

    These environments include local development (usually a coder’s personal device), a continuous integration (CI) or development server, test servers for user interface (UI) testing and the production environment itself. Each has stages with requirements that must be met for code to advance.

    Whenever any new software features or capabilities are entertained, though, developers and the QA team should begin by answering the question, “How can this be tested?”

    Moving Through the Stages

    Once code is created, it must be able to be built without error, passing all unit tests and local verification.

    This stage is particularly important because the development environment echoes the requirements of production; if it works here it’ll likely also work for the customer.

    At this stage, running early regression and developer verification tests ensures the build is ready while providing feedback about the reliability of the environment. It’s useful to have individual test cases as part of more than one schedule. This enables developers to test minor parts of the software without undergoing regression tests of all product features. These tests should be automated and triggered as part of the same process.

    Developers should have the ability to perform basic tests on their own code. After all, the sooner a flaw is identified, the easier it is to eradicate. Just as testers are not experts in coding, neither are developers expert testers. To shift left and effectively integrate tests at all stages of the development life cycle, it must be automated. This will allow developers to test larger quantities of code earlier and more frequently.

    Once through this stage, the software can advance to the test environment.

    This is where a variety of product tests take place, including those for function, load and performance. Passing this stage entails completing all automated functional UI tests, verification of visual appearance, an inspection of server logs and performance evaluation.

    This is where most regression tests are conducted and automation is crucial for scheduled tests to be triggered.

    Next, the product moves into the production stage.

    Tests still occur in this stage to pinpoint any remaining bugs. Automated tests are then run against the production environment to closely mirror interactions with real end users. At this point, it’s highly unlikely there will be major issues but, if there are, running tests at earlier stages could very well be the culprit.

    Throughout the process, ideally, testers will use a visual, no-code tool to create these automated tests. In this way, testers are able to more rapidly and intuitively implement test automation without having to tap into programming resources. Additionally, the integrated documentation of the actions taken during automation will be interactively connected to visual documentation of the test results. The upshot is that testers and developers gain a shared language for both building test automation and troubleshooting the issues it identifies.

    This is Not Just a Test

    Effective, scalable tests are crucial to delivering the reliable, high-quality software that today’s businesses rely on for day-to-day operations. This is why DevOps and development teams need to integrate tests at all stages of software development. Shift left speeds up the creation of better quality software, which provides a significant market advantage. After all, it’s not just about software quality; it can show which companies are best prepared to meet market demands.

  • Best Practices for Codeless Test Automation

    Best Practices for Codeless Test Automation

    Low-code and no-code development platforms have revolutionized how companies think about application development. The same can be said for software for quality assurance (QA), where codeless test automation solutions handle the burden of coding for organizations that can’t allocate extra programming resources or keep up with extensive writing and maintenance of automated tests.

    Codeless test automation can be a big win for companies that are resource-challenged and trying to get applications to market quickly. A recent Applause survey of more than 2,000 digital experts revealed that nearly 54% of organizations have strategic plans to purchase a codeless test automation tool in the future.

    Despite the heat in the market, though, it is not a fit for every type of application or use case. It is not, in most cases, a replacement for scripted automation, but rather should be used as a complement to traditional automated testing and manual functional testing.

    An example of this is with dynamic content in applications, which have unpredictable or even subjective outcomes. Consider a streaming media platform. An automated solution might validate that the “Play” button triggers the code to play a movie, but it cannot validate that the movie plays without audio or video glitches.

    In complex cases that aren’t a fit for either codeless or traditional automated methods, manual testing is a better alternative.

    When to Use Codeless Test Automation 

    Test automation is not going anywhere, even as codeless tools become more prominent. That’s because automation enables brands to move faster and meet the fast-moving target of building and testing software that is both high quality and able to be released to end users quickly. The downside of traditional test automation, however, is that building and maturing the process can be time-, cost- and resource-intensive.

    This is because traditional automation requires experts in coding, not only to write tests at the outset but to also maintain them over time. The complexity and need for different tools—like Appium, Selenium, Apple simulators, Android emulators, element locators, locator strategies and so on—all contribute to the complexity of traditional automation and the need for specific expertise. Plus, software development engineers in test (SDETs) are constantly being taken from QA teams and moved into development roles, which often means that these projects get stuck halfway through and never fully completed, which can then result in a tremendous waste of both time and money for companies.

    The overarching value of codeless test automation, on the other hand, is that anyone can do it. With a codeless solution, a user simply moves through a test case and the codeless tool can then transcribe that experience into an automated test. And, while codeless test automation tools originally addressed only web applications, more tools now offer the ability to run sessions and create automated tests on mobile apps—on both Android and iOS—as well as web applications.

    What it all comes down to, though, is that organizations shouldn’t be thinking of codeless tools and traditional automation as an either/or scenario. Codeless test automation tools are great for less-involved scenarios—like smoke tests or portions of a regression testing suite. Using codeless tools in this way allows SDETs and dedicated automation resources to focus on higher-priority and more complex automation. Ideally, manual testing should be used alongside both traditional automation and codeless test automation solutions as a way to maximize the speed, scale and quality at which software can be delivered to end users.

    As a rule of thumb, every organization should balance test automation and codeless test automation and consider automating when the following factors are in play: 

    • Repeatable task 
    • Time-consuming testing activity 
    • A stable testing environment with little change anticipated 
    • Repetitive test(s) 
  • Bringing Continuous Testing Into the Software Delivery Pipeline

    Bringing Continuous Testing Into the Software Delivery Pipeline

    For any highly competitive industry that relies on a smooth and stable user experience, it is vital that their development teams accelerate release cycles, lower operating costs and ultimately reduce risks. Continuous testing is one way companies can achieve these objectives.

    As a subset of software testing, continuous testing (CT) is valuable because it uncovers and addresses different problems at various stages of the development journey that other methods don’t. Most other testing methodologies require a large staff; CT doesn’t. CT leverages in-sprint automation, including end-to-end tests across the front and back end of the technology to cut costs and facilitate rapid delivery without adding more pressure on internal teams. Moreover, as part of a larger software delivery pipeline, CT is positioned closer to the beginning to execute tests early, often and comprehensively to realize cohesive release goals.

    CT introduces quality assurance with constant iterations of testing and actionable feedback at each stage of development to evaluate business risk coverage and ensure the best user experience and customer satisfaction. As the name implies, CT helps developers and product managers detect and fix critical bugs not only in the early stages of development, but continuously throughout the software development life cycle (SDLC), ensuring that errors in the code do not get unintentionally released into the wild.

    As software companies accelerate their release frequency, continuous testing sets a precedent for digital tools development and the quality assurance process.

    Benefits of Continuous Testing

    Continuous testing plays an integral role in the CI/CD pipeline, is part of the overall set of DevOps best practices and the information it provides is invaluable to development teams. The benefits of CT are also far-reaching, with positive impacts on the entire organization, including:

    Better Company-Wide Feedback

    Continuous testing facilitates feedback to all involved teams and stakeholders. For example, when a bug gets detected in code, ideally, the only team notified should be the team responsible for the part of the code with the bug. Without CT to automatically detect bugs and notify those responsible for fixing errors, triaging bugs could be a full team effort or, worse, a dedicated team’s main function. This approach is very inefficient and wastes the time of the engineers whose code is bug-free. The feedback provided from CT cuts down on these unproductive scenarios and allows stakeholders to make immediate and informed decisions.

    Higher Quality Code and User Experience

    A company that effectively uses CT should expect higher quality code and a more stable user experience. Stability measures application health and user experience; it is negatively affected by session-ending events like crashes. With CT, businesses will give their development teams better awareness of product quality and performance while also significantly improving overall business metrics by fixing broken apps faster.

    CT Breaks Down Siloed Teams

    Not only will CT enhance quality assurance, it similarly works to improve the development of digital tools through the breakdown of siloed teams. When teams become siloed, they inadvertently keep processes and knowledge to themselves, resist cross-functional collaboration and work in isolation toward their own goals. Predictably, separate groups fail to share insights or ideas, leading to stagnation and decreasing innovation. However, by leveraging CT, siloed teams can become more collaborative in development, testing and operations, increasing overall business efficiency.

    CT Enables Objective Evaluation

    Finally, CT enables an organization to pause and objectively assess if it is aligning efforts with its specific business goals. As mentioned before, CT uses automation to provide timely feedback on business risks, reducing the chances of a digital application or tool failing at launch. Likewise, dev teams and managers can more easily meet their business goals and optimize the value of a release. Additionally, objective assessments are helpful for critical decisions – as CT gives actionable data for decision-makers to support their choices.

    Today, consumers have come to expect error-free experiences within their software applications, with the most stable of these applications driving conversion, engagement and retention. Applying a continuous testing model to a software development pipeline will ultimately result in applications being delivered faster and at higher levels of quality than ever before, with an enhanced end-user experience. As companies combine agile development with continuous testing, they will optimize each aspect of the development life cycle and their larger business goals.

  • Lack of Continuous Testing Mastery Slows DevOps Progress

    Lack of Continuous Testing Mastery Slows DevOps Progress

    Progress in achieving the maximum benefits to be gained from investing DevOps processes appears to be held in check by a failure to master the art of continuous testing. A new survey of 923 IT and testing professionals published this week by CA Technologies suggests that only 20 percent of the respondents could be classified as “leaders” when it comes to implementing continuous testing.

    A full 93 percent of respondents identified testing automation as important, but only 1 in 5 respondents said they achieved a high level (80 percent or more) of test automation coverage. Seventy-five percent said continuous testing is important as well. Unfortunately, most of the respondents continue to rely on manual processes. In fact, respondents attributed continuous testing challenges to the lack of automation in almost every aspect of the testing process, including generation of test cases through test execution to critical activities such as fast, safe and efficient test data management.

    Scott Edwards, senior director of product marketing for CA Technologies, says failure to automate continuous testing will only lead to an increased DevOps divide. Organizations that have automated DevOps processes not only release more code faster, the report find they enjoy advantages such as:

    • 2.3 times more likely to have succeeded in left-shifting testing activity to developers;
    • 2.6 times more likely to reduce defects by more than 50 percent;
    • 2.4 times more confident in quality of output;
    • 1.9 times more confident in speed of delivery; and
    • 3.9 times more likely to be working in an organization exhibiting rapid revenue growth.

    Edwards says those results indicate that organizations that are further down the DevOps process will enjoy a substantial competitive advantage at a time when organizations of all sizes are realizing how dependent they are on software to differentiate themselves. Organizations may be operating in a variety of vertical industries, but increasingly they are becoming software companies that happen to specialize in a specific area, says Edwards.

    Making that shift requires IT organizations to move away from thinking about developing applications as an art to building what amounts to a software factory, he says. The factory approach not only eliminates silos in the DevOps process, it also should ideally allow developers to continuously test their own applications. Of course, the biggest issue with testing may very well be the simple fact that most developers build applications today using notebooks and workstations that don’t accurately reflect the complexity of the data center environments those applications will ultimately be deployed in. However, thanks to the rise of technologies such as service virtualization and cloud-based development environments, it has become easier to more closely mimic production environments.

    The degree to which an absence of continuous testing impedes both the rate at which applications are deployed in a production environment will naturally vary by organization. But the one thing that is almost certain is that organizations that have mastered continuous testing are much more likely to deliver a quality application experience than those that don’t.

    — Mike Vizard

  • xMatters Sees Value in Continuous Testing

    xMatters Sees Value in Continuous Testing

    xMatters builds an intelligent communications platform that alerts and connects company stakeholders for IT incident response and in manufacturing line workflow support and DevOps systems scenarios. Customers who count on xMatters also count on the effectiveness of the vendor’s continuous testing mechanisms: A failure of continuous testing early on at xMatters could lead to a magnified impact on the customers’ crises response, resulting in costly repercussions.

    XMatters uses continuous testing to promote continuous delivery of its intelligent communications platform, delivering changes to a limited number of features in an accelerated fashion. “This reduces our risks of failures and makes it easier to resolve defects,” said Deepa Guna, QA Architect at xMatters.

    For xMatters, the proper use and placement of continuous testing in the DevOps pipeline is critical to ensure the intelligent communications solution does not introduce errors during customer crises.

    xMatters Software, Languages and Tools

    The xMatters platform consists of a service stack architecture with front-end technologies accessible via the web and back-end technologies that support it. According to Guna, the platform takes advantage of Java, RESTful JSON, Node.js and Postgres, which run on data centers that are geographically dispersed in North America, Europe and the Asia Pacific regions.

    “Our data centers are built on infrastructure as a service, which includes technologies such as OpenStack, Puppet, Fabric and Consul,” Guna said. XMatters also has a mobile application for iOS and Android platforms as part of its communication service.

    xMatters Testing Approaches, Before and After DevOps

    The company’s testing before the advent of the continuous delivery pipeline included manual test execution of regression tests in the midst of a slow process of releasing one large set of features per month and deploying customer-reported defects, Guna said.

    Today, xMatters uses automated and manual regression tests and unit test code coverage monitoring, releasing small sets of features and resolved defects weekly while enabling customers to provide feedback early in the feature development process.

    XMatters uses several tests during the development, test and release phases of the DevOps pipeline and when delivering the platform to the customer. The development phase includes the following testing tools and their associated types of testing:

    • SonarQube analyzes the quality of the code.
    • Jacoco measures the code coverage of the developer unit/integrations tests.
    • Git + Jenkins run the unit and integration tests before xMatters commits the changes.

    The testing phase, Guna said, runs tests as follows:

    • Jenkins deploys artifacts in test environments and runs automated tests.
    • An automation framework that includes Jmeter, Gradle, Java, TestNG and Selenium runs its tests.

    The release phase tests software after this manner:

    • Jenkins deploys artifacts in test environments and runs automated tests.
    • An automation framework that includes Jmeter, Gradle, Java, TestNG and Selenium runs its tests.
    • Splunk and Takipi do exception monitoring.

    When xMatters delivers the platform to its customers, Guna said, it performs the following tests:

    • Runscope monitors the health of the system to ensure critical services are running.
    • Pingdom monitors the status of the customer websites.

    Drilling Down into Release Testing

    “During the release testing phase, we put the application through 1,000 fully automated and manually initiated tests that we execute with every release cycle. We also have tools that are monitoring for exception during the release testing,” Guna said. Passing this phase successfully, the new software platform release is ready to be pushed into the production environment.

    XMatters uses the continuous delivery pipeline to deploy the new platform release first to the customer staging environment and next to its production environment, Guna said. “All production environments have ongoing smoke tests that we execute every five, 10 or 15 minutes, based on customer tier, to monitor the health of the services.”

    Results Metrics

    xMatters’ method of continuous testing seems to be working for the vendor. “We have reduced the number of major incidents in production due to application defects by 40 percent this year compared to last year,” Guna noted.