devopsy.kinsta.cloud

Tag: functional testing

  • Applause Report Surfaces Functional Testing Issues

    Applause Report Surfaces Functional Testing Issues

    An analysis of more than 340,000 bugs collected by Applause, a provider of a testing platform designed to integrate with a range of DevOps platforms, found functional bugs accounted for 68% of issues. The research gathered data from 13,000 mobile devices and 1,000 unique desktops running 500 versions of operating systems and found that the majority of issues could be traced to functional bugs compared to visual (17%), content (9%), crashes (4%) and lags and latency (2%) issues that, collectively, only add up to 32%, the report found.

    The report also found screen readers comprised 66% of all accessibility bugs compared to keyboard navigation issues and insufficient color contrast which accounted for only 12%. In terms of localization, poor and missing translations account for more than two-thirds (67%) of bugs, the report found. Based on ongoing feedback data that Applause also continuously collects from customers, nearly half of organizations (47%) identified currency and number formatting as the most valuable bugs to identify when it came to localization.

    Overall, organizations ranked the discovery of crashes (75%), functional bugs (61%) and lag and latency issues (53%) as exceptionally valuable, according to Applause.

    Luke Damien, chief growth officer for Applause, said when it comes to testing, in general, there is still not enough focus on user and customer journeys. That lack of focus is becoming a larger issue as more organizations invest in digital business transformation initiatives that are driven largely by mobile applications, he noted. Payments are especially problematic because they often rely on application programming interfaces (APIs) exposed by a third party in a way that results in suboptimal application experiences that have a direct impact on revenues, he added.

    The challenge is that it’s not entirely clear how far left responsibility for application testing is shifting. In some cases, developers are assuming responsibility. In other cases, a dedicated testing team is still responsible. Developers, of course, will test applications as they build them but application experiences on a local machine may not always be replicated in a production environment. The issue is that most end users today are not especially forgiving when a mobile application fails to meet expectations. Months of development effort can be wasted simply because a function was not tested in a production environment.

    Less clear is to what degree testing will be automated in the future. Machine learning algorithms are making it possible to automate testing of both functions and user interfaces. It’s not likely the need for humans to be involved in testing will be eliminated any time soon, but many of the routine tedious tasks that conspire to limit the rate at which applications can be tested should be sharply reduced in the months and years ahead.

    In the meantime, an organization’s entire brand reputation is now tied to the quality of the application experience it enables on a mobile device. Revenue targets can now be easily missed if users of a mobile device choose one application over another simply because a function was broken—even for a short amount of time. As such, testing has never been more critical given organizations’ dependency on software.

  • Best of 2021 – Best Practices for Application Performance Testing

    As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the eighteenth in our series of the Best of 2021.

    When done properly, software application performance testing determines if a system meets certain acceptable criteria for both responsiveness and robustness. Before you jump in to testing, though, there are some best practices to remember.

    Start by defining test plans that include load testing, stress testing, endurance testing, availability testing, configuration testing and isolation testing. Align these plans with precise metrics in terms of goals, acceptable measurements, thresholds and a plan to overcome performance issues for the best results. Make sure you can triage performance issues in your testing environment; you should analyze issues impacting application performance by examining system functionality under load, not just the indicators of poor performance on the load testing tool side. Leveraging application performance management (APM) tools, which simulate production environments, provide much deeper insights into application functionality, as well as into overall performance under stress or load.

    10 Performance Testing Best Practices

    1. Test Early and Often

    Performance testing is often an afterthought, performed in haste late in the development cycle, or only in response to user complaints. Instead, you should be proactive. Take an agile approach that uses iterative testing throughout the entire development life cycle. Specifically, provide the ability to run performance “unit” testing as part of the development process – and then repeat the same tests on a larger scale in later stages of application readiness. Use performance testing tools as part of an automated pass/fail pipeline, where code that passes moves through the pipeline and code that fails is returned to a developer.

    2. Consider Users, Not Just Servers

    Performance tests often focus solely on the performance of servers and clusters running software. Don’t forget that people use software, and performance tests also should measure the human element. For instance, measuring the performance of clustered servers may return satisfactory results, but users on a single, troubled server may experience an unsatisfactory result. Tests should take user experience into account, and user interface timing should also be captured along with server metrics.

    3. Understand Performance Test Definitions

    It’s crucial to have a common definition for the types of performance tests that should be executed against your applications, such as:

    • Single User Tests. Testing with one active user yields the best possible performance, and response times can be used for baseline measurements.
    • Load Tests. Understand the behavior of the system under average load, including the expected number of concurrent users performing a specific number of transactions within an average hour.
    • Peak Load Tests. Understand system behavior under the heaviest anticipated usage for concurrent number of users and transaction rates.
    • Endurance (Soak) Tests. Determine the longevity of components, and whether the system can sustain average to peak load over a predefined duration. Monitor memory utilization to detect potential leaks.
    • Stress Tests. Understand the upper limits of capacity within the system by purposely pushing it to its breaking point.
    • High Availability Tests. Validate how the system behaves during a failure condition while under load. There are many operational use cases that should be included, such as seamless failover of network equipment or rolling server restarts.

    5. Build a Complete Performance Model

    Measuring your application’s performance includes understanding your system’s capacity. This includes planning what the steady state will be in terms of concurrent users, simultaneous requests, average user sessions and server utilization during peak periods of the day. Additionally, you should define performance goals, such as maximum response times, system scalability, user satisfaction scores, acceptable performance metrics and maximum capacity for all of these metrics.

    6. Define Baselines for Important System Functions

    In most cases, QA systems performance don’t match production systems performance. Having baseline performance measurements for each system can give you reasonable goals for each testing environment. These baselines provide an important starting point for response time goals, especially if there are no previous metrics, without having to guess or base them on the performance of other applications.

    7. Perform Modular and System Performance Tests

    Modern applications are incorporate many individual, complex systems, including databases, application servers, web services, legacy systems and so on. All of these systems need to be performance tested individually and together. This helps expose weaknesses, highlight interdependencies and understand which systems you should isolate for further performance tuning.

    8. Measure Averages, but Include Outliers

    When testing performance, you need to know average response time, but this measurement can be misleading by itself. Be sure to include other metrics, such as 90th percentile or standard deviation, to get a better view of system performance.

    KPIs can be measured by looking at average and standard deviations. For example, set a performance goal for the average response time plus one standard deviation beyond it (see Figure 1). In many systems, this improved measurement affects the pass/fail criteria of the test, matching the actual user experience more accurately. Transactions with a high standard deviation can be tuned to reduce system response time variability and improve overall user experience.

    application

    9. Consistently Report and Analyze the Results

    Performance test design and execution are important, but test reports are, too. Reports communicate the results of your application’s behavior to everyone in your organization, especially project owners and developers. Analyzing and reporting results consistently also helps to determine future updates and fixes. Remember to consider your audience and tailor reports to each audience. Reports for developers should differ from reports sent to project owners, managers, corporate executives and even customers, if applicable.

    10. Triage Performance Issues

    Providing the results of performance tests is fine, but those results, especially when they demonstrate failure, are not enough. The next step should be to triage the code/application and system performance, and involve all parties: developers, testers and operations personnel involved. Application Monitoring Tools can provide clarity regarding the effectiveness of triage.

    Additionally, remember to avoid throwing your software “over the wall” to a separate testing organization, and ensure your QA platforms match production as closely as possible. As with any profession, your efforts are only as good as the tools you use. Be sure to include a mix of manual and automated testing across all systems.

     

  • Make a Plan for Test Automation

    Make a Plan for Test Automation

    The amount of testing that we could be doing is massive. Most of us don’t look at testing across the spectrum and all-inclusively, but let’s do that for a second.

    We have functional testing at the code level, which is reasonably well automated already, if your shop is using such automation. Then we have integration testing, which is a superset collection of API testing, code integration testing, usability and regression testing. Integration testing is automated, and sometimes that’s done well, but this massive space is still growing. Next up would be security testing, which covers much of the same code that other testing is interested in from a strictly security focused perspective. Then, there is actual usability testing, which is more important for some areas than others—embedded needs far less usability testing than web apps, for example—and has different levels of automation at this point in time.

    Really, when you think about it, the complexity of this environment is massive. We not only need to test an array of different functionalities from an array of different perspectives, but we have to do it on an array of platforms. Testing on Windows is different than testing on Linux; testing in the cloud can use different tools than testing in-house; clients can be on any given platform, depending on the application and target market … we really do have a beast, here.

    And testers have always been behind, or have not covered things well. This is not a ding on testers, this is the result of the complexity at hand and the fact that, in many instances, one line of code can go wrong in a bunch of different ways.

    Enter automation, which is growing rapidly in the world of continuous integration (CI). Automated testing tools range from broad to very focused, and all help facilitate testing. The DevOps increase in development means more testing, and testing was already behind. This has been discussed pretty thoroughly in the past, but it finally feels like we’re getting to a point where we have viable solutions to the problem. If testers become test developers that review results of automated tests, and through CI those automated tests are increasingly worked into the process of development, we’ll see testers getting better coverage and quality seeing a continued increase.

    But that does mean you should keep your eyes on the market as it continues to grow, and pick tools that offer the most functionality that is most useful to your test team. And get them up and running. More testing means more work setting up tests. Even no-code test solutions for UI require someone to record the actions that are necessary to complete the test, and that takes someone’s time. But the return is long-term testing that doesn’t require a person until a red flag is raised by automation.

    I’m all for it. Both testing and security have the most to gain from automation, because they’ve been the most behind historically, and corner-cutting has cost organizations dearly. So, let’s get with it; check out some new test tools (there are plenty listware articles giving you a place to start), and let’s make great software better!

  • Using Incident Response for Continuous Testing

    Using Incident Response for Continuous Testing

    Incident response tools offer the ability for organizations to not only implement continuous testing but to also shorten the feedback loop from continuous testing back into planning and development

    At this point, you should be able to say the word “continuous” to any technical team, drop the mic and leave. Yes, the idea that we want to do more delivery, more testing and more interactions isn’t always novel. The reason we use phrases such as continuous testing and continuous improvement is to have the conversation about the unique set of tools and practices required to run these processes in production, and not just as part of continuous integration (CI). Part of supporting continuous testing is having a highly effective incident response process—arguably, it’s required.

    Continuous testing is used in the context of functional testing, performance testing and vulnerability scanning. All three come from different disciplines, but all suffer from the same challenge that they’re processes designed to be run during pre-release, not in production.

    Functional testing is the process of running automated tests of the application as if the test were a user, from the outside in. Performance testing is the process of putting artificial loads on the application to simulate high transaction volume to see how the application responds. Vulnerability scanning is the process of scanning application code to make sure components don’t have known vulnerabilities.

    All of these application quality processes are generally run in continuous integration (CI) environments and pre-production. If an organization were to move them to production, they get several benefits:

    • Full parity with production infrastructure. When you run these tests in CI, the environment it’s run on isn’t a one-for-one comparison with the infrastructure used for production. By running tests in production, you get a true picture of what is going to happen.
    • Better relation to production data. By running these tests in production you are running aside real user activity, which gives a more accurate picture from the outcome of the tests.
    • Reduced reliance on CI. Continuous testing allows you to streamline and even reduce pre-release processes, allowing more code to get to production faster. It also reduces the reliance on synthetics, mocks and service virtualization.
    • Specifically for vulnerability scanning, the ability to build DevSecOps workflows into production for vulnerabilities that show up at a later date.

    The mechanisms for running the tests in production are the same as CI, just with a different target environment. The problem is, when something breaks in production, it’s much more serious. So, to support continuous testing, you need to know what to do if there’s a test case failure or a failure in the application as a result of running the test.

    This is where incident response comes in. Today, most incident response tools and processes are used by SREs and developers to be alerted when something breaks. But, the same tools and approaches can be used for running tests as well. The only difference is, not only will the application be a data source of an alert, but so are the testing tools. This way, quality engineering and DevSecOps teams can confidently run their test suites and get alerted if something goes wrong.

    In particular for vulnerability scanning, continuous testing allows SecOps folks to get alerted when a new vulnerability is detected, which can result in patching holes long before the application is exploited.

    If an organization embraces continuous testing, they need to know:

    1. If a test fails. The effort to run tests, both in terms of human and compute time can be high. To wait for a result can also impact downstream processes. Therefore, it’s important for application quality and security to know that the tests themselves are running successfully. And, if they break, the QE team or SecOps team needs to be alerted so they can re-run the tests.
    2. If the tests break something in production. Not only do the teams who own the testing process and tools need to be alerted but if you’re on-call, you need to know as part of the context of an alert, that the incident was directly or indirectly impacted by tests being run. This can save a lot of time by killing tests versus chasing down issues with the application that may not be there.

    Incident response tools offer the ability for organizations to not only implement continuous testing but to also shorten the feedback loop from continuous testing back into planning and development. This can be done simply by reducing the amount of time it takes to get details from an incident back to the appropriate teams, but also for non-critical incidents that indicate potential issues in the application. Issues can be made more visible to everyone and communicated to product owners faster and more effectively. This is the other continuous, continuous improvement.

  • Test Value Distribution

    Test Value Distribution

  • Best Practices for DevOps in the Cloud

    Best Practices for DevOps in the Cloud

    DevOps is commonly defined as a methodology that combines software development with IT operations. The goal is to shorten a system’s development life cycle while delivering features, fixes and updates frequently in close alignment with business objectives. Putting DevOps into practice is an ongoing experiment for many organizations.

    Developers often struggle with continuous integration (CI) or continuous delivery (CD) performance, testing delays and other bottlenecks. In fact, one survey reported that only 34% of projects finish on time and only 42% come in on budget. Security and development teams must collaborate to keep up with the rapid development life cycle without sacrificing security with every update.

    Enterprise SaaS apps are not distributed like traditional software—they are continuously running services. So, CD is all about updating the service with a set of changes. In public clouds such as AWS and Heroku, this is done through pipeline stages (e.g. dev, test, staging and production). In platforms such as Salesforce, stages become sandbox environments, ephemeral scratch orgs and production instances.

    How can development teams shorten the development life cycle without sacrificing the integrity of the features and updates? In this article you will learn key DevOps practices to achieve predictable, reliable release management.

    Advanced Planning Ensures All Boxes Are Checked

    Agile planning is now de rigueur for enterprise development. It is best used if:

    • The product itself will most benefit from flexibility.
    • Regular face-to-face interactions and collaborations are needed between team members.
    • The end product requires regular updating, monitoring and continuous delivery to the client throughout the life cycle.

    For CD, ensure your user stories are sized correctly and that each story rolls up to an Epic that represents a standalone feature. This enables your team to release bite-sized pieces of functionality that are consumable by users and traceable back to your release plan. Also, define acceptance criteria for use in the verify stage so you know when something is done-done.

    Manage the Source Code, Especially When Working in a Shared Environment

    There are a few practices of app creation that impact CD. First, source code is not the only element of an enterprise solution. Schema, user interface, access control rights and static resources such as icons and images are all part of the creation process and should be managed just as diligently as source code. They should all be checked in to a version control system (VCS) as a single source of truth.

    Second, ensure that developers are making changes in a segregated environment. In some low code platforms, such as Salesforce.com, this becomes a challenge, but even in a database, multiple database administrations (DBAs) can step on each other’s changes when working in a shared environment.

    Functional Testing

    Automated quality processes are difficult to implement, but essential Selenium is a great tool for testing functionality and should be part of your DevOps practice. But there are several verifications that should be made before you get to functional testing. Static code analysis tools, such as PMD, are essential to ensure your development team is coding against a single style.

    Define a team style and enforce its use. Unit test coverage is also essential—incorporate them with a coverage of at least 75%. Finally, after these automated tests pass, implement a manual peer review. Old hands are able to spot opportunities for performance improvement where automated tools may not.

    Changes Should be Released in a Defined Pipeline with Strict Quality Gates

    Packaging for CD is about capturing changes that can be incrementally released into the next stage or environment. Development teams should dedicate a branch for each stage and bundle incremental changes into a promotional branch to leverage the merge conflict detection in your VCS.

    Changes should be released in a defined pipeline with strict quality gates that ensure the verification steps and enforce the order of release stages. Developers should not be able to bypass key stages throughout the development cycle—for example, moving from development to production.

    Focus on making smaller changes, which result in reliable, predictable releases. Finally, be sure to automate promotions. This enables your team to validate the promotion scripts in integration, test and staging before releasing to production, which decreases the likelihood of an unhappy surprise.

    Automate Configuration, Especially for End User Access Rights

    Make testing for proper configuration part of your verification process. Platforms such as Salesforce have powerful profiles and permission sets that enable user access to only the right information. Improper configuration can expose personally identifiable information (PII), credit card numbers or medical health data.

    Monitor for More Than Just Service Availability and Performance

    Most SaaS companies monitor service availability and performance. But excellent companies also monitor data access to detect improper configuration changes and monitor end user feature usage, which gives product managers visibility to their usage.

    — David Brooks

  • Sauce Labs Combines Functional and Performance Testing

    Sauce Labs Combines Functional and Performance Testing

    Sauce Labs announced it is now moving beyond functional testing, adding a front-end testing tool it has dubbed Sauce Performance to make it easier to accelerate testing within the context of a DevOps process.

    Lubos Parobek, vice president of products for Sauce Labs, said the goal is to enable developers or testers to begin testing both how well application functions work and perform much earlier in the DevOps process using the same testing tool.

    Parobek said that as organizations embrace modern application development architectures based on microservices, it’s become crucial to test the capabilities of any module within an application because the number of interdependencies between modules continues to increase. That approach accelerates the rate at which applications can be developed. However, any flaw that impacts how the application works or performs can have a debilitating impact on any number of microservices that make up the application, he noted.

    Sauce Performance makes use of debugging and performance trace logs to identify the root cause of an issue in addition to capturing metrics such as Time to First Meaningful Paint, Time to First Interactive, Page Weight and Speed Index. Sauce Performance also employs algorithms to automatically establish a baseline that can be used for regression testing.

    Parobek said Sauce Performance serves to streamline testing processes as well. Most existing front-end performance testing solutions require integration with a load/stress testing tool or a separate application performance monitoring (APM) tool. However, Sauce Performance allows developers to use their existing Selenium test scripts to capture both functional and performance test data.

    In addition, Sauce Performance has been designed to integrate with multiple continuous integration/continuous deployment (CI/CD) platforms to make it possible to feed test results directly back to developers, which can then decide to prioritize issues that need to be addressed.

    Sauce Performance is available as part of the Sauce Labs Continuous Testing Cloud.

    Parobek said organizations want to be able to make it easier for developers to test their own code as part of their DevOps best practices. That doesn’t eliminate the need for dedicated testers for evaluating functions, but potential performance issues today are being addressed often long before applications are deployed in a production environment.

    One of the reasons applications are not as thoroughly tested as they might be is that many organizations require developers to learn how to use tools that have different user interfaces and then switch between testing tools. The result is a complex testing process that makes it more tempting to short-shrift elements of the testing process as application delivery deadlines loom.

    More challenging still, many organizations still don’t continuously test applications after they are deployed to ensure new functionality does not adversely impact the user experience.

    For as long as applications have been built, tension has existed between the developers who create applications and the people who test them. That debate may never be resolved, but everyone agrees in this age of DevOps that it’s better to test early and often.

    — Mike Vizard

  • Dissecting the Role of QA Engineers and Developers in Functional Testing

    Dissecting the Role of QA Engineers and Developers in Functional Testing

    Until a few years ago, quality assurance was generally considered low-level R&D. As a result, quality assurance (QA) teams were characterized by a low budget, lack of attention and, more often than not, poor attitudes. Moreover, QA engineers were considered the enemies of developers and the product.

    However, today, this is no longer the case, and QA is part of the entire software life cycle. Today, organizations understand how a product’s quality affects the brand’s reputation and customer retention. In the Agile ecosystem, QA engineers are no longer solely focused on testing; they also fill an important role in the planning and delivery domains. The shift left approach—where testing is performed earlier in the life cycle—requires QA engineers to have higher technical abilities and involvement in the whole CI/CD process. Nevertheless, QA managers, who usually encourage assimilation in the scrum team, as well as rapid feedback for user stories and automation, must not forget the main objective of quality engineers: to ensure a high-quality product and a seamless user experience for customers.

    High quality demands testing and verifications. There are different types of testing that can be conducted to meet quality goals. In one of its blog posts, Testing Excellence describes almost 60 types of testing that can be performed on a product. We’re going to look at one that is quite clearly one of the most important types of testing: functional testing.

    Different than other forms of testing, functional testing is considered one of the most sophisticated test type, and the most comprehensive one.

    Same Team, Different Responsibilities

    One important fact every QA engineer must remember is that developers develop and QA engineers test. Even though QA and developers are part of the same scrum team, their work, methodologies and state of mind are completely different. In cases in which the significant contribution of QA isn’t visible to the whole team, testing might be considered an easy and clear task. Quite the contrary; testing an application is much more complicated and entangled than developing it. When planning tests and executing them, QA engineers must take into account several aspects developers likely never consider.

    Understanding How the User Thinks

    The primary distinction between dev and QA is state of mind. While developers write pieces of code that later become features in the application, QA is expected to understand how real users think, engage customers to learn business scenarios, and confirm that the application satisfies customer needs.

    Let’s look at eBay, for example. Every veteran R&D member will agree that eBay isn’t just a simple shopping application, but a comprehensive system that includes a whole set of capabilities on both web and mobile that requires attention and deeper understanding to determine whether they really function correctly. In such a rich and mature online platform, we can identify several types of users and many cases in which the application may be utilized. A seller performs different steps than a buyer and requires different functionality, while a support agent needs an entirely different version of the application. When planning tests, all these personas must be taken into consideration with a particular test plan for each. In other words, testing functionality is not just simulating user operations.

    In addition, eBay is not a standalone application—its web application lives inside the browser and subsequently depends on that browser’s rules and behaviors. This is true for eBay’s mobile apps, as well, which must meet the OS constraints to function correctly, whether it’s navigating to the seller’s store or making a purchase. This makes compatibility testing an integral part of functional testing. Most developers don’t think about the project availability matrix when they code, so QA is required to contemplate all the ways the customer will engage with the application and verify functionality for each.

    Viewing the Application as a Whole

    Dev and QA differ in their point of view—an obvious but important distinction. While developers focus on developing user stories, QA engineers test the user story, the feature it is part of and the application behavior with the new code. Integration testing is a crucial part of the application verification effort and requires a broader view of the application and how the new functionality is part of it. End-to-end testing is the fulfillment of the above two aspects, validating that real user scenarios in the context of multiple regions in the application behave as expected. End-to-end tests increase test coverage and reduce the risk of making the new code part of the application.

    Paying Attention to User Experience

    As noted, testing the application functionality is merely one step in functional testing. Testing the operations available in the application encompasses another type of testing—the user experience. If a new feature is developed, but users do not have easy access to it or cannot understand how to use it, it is as if the feature doesn’t exist. QA engineers must incorporate user experience as a main metric of features so users will not have room for error, which can lead to misuse and frustration.

    For instance, eBay’s “all categories” page previously displayed all the categories by default, even though some of the categories had no items. While this was the correct behavior from a functionality standpoint, it caused user confusion and mistrust. Consequently, the “all categories” view was changed to show only categories with items.

    Looking at the narrow current task, developers might implement features as they are told without questioning or contemplating the specifications. In the same way, testers who test features without realizing how customers perceive them are missing an important benefit of the test.

    The above three aspects—thinking like customers, testing the application as a whole and paying attention to user experience—revolve around the way developers and QA engineers relate to new features. While developers tend to follow the specification guidelines, QA engineers must “think outside the box” when it comes to user consumption of the feature.

    There are two other ways in which QA work is more complicated and yet invaluable—test maintenance and test automation.

    Maintaining High Quality Throughout an Efficient Release

    In general, developers split their time between developing new features and fixing defects. QA work is much more diverse. A substantial portion of the release is spent on testing functionality that already has been released and used. While the main effort of designing and writing test plans for this functionality was invested in past releases, every new feature developed requires a test plan and a maintenance effort on existing tests. One of the main goals for QA managers is reducing this effort to a minimum. Teams that succeed in creating clear and reusable tests simplify the maintenance struggle and optimize the test registration time.

    Eliminating Risk Through Automation

    However, for QA, the most cost-effective solution is test automation. At release end—the time when QA is at its most stressed and every commit might introduce a new risk—automation testing can eliminate risk and supplies early feedback on every change. Agile teams strive for the highest automation coverage percentage so the development team can have confidence in committing changes late in the release. That said, developers often aren’t interested in investing their time in automation. They support the QA automation effort, but they don’t want to “dirty their hands” by doing testing. Whether it’s UI automation—during which achieving element uniqueness is essential yet tricky and requires developers’ intervention—or API testing for APIs without proper documentation, QA faces intricate challenges that require sophisticated solutions, which developers are, in most teams, not part of. While the dev team focuses on the status of the new feature, QA attends to the application as a whole with extra attention to identify regressions created by a change.

    Join Forces for Higher Rate of Success

    Testing an application requires a completely different mindset than developing it. One of the main challenges in the Agile methodology is to create developer awareness of quality, involving other stakeholders in the quality assurance activities and making the QA process visible to everyone. QA managers must understand how day-to-day QA work is different than the work of other team members so they may truly represent the complexity, efforts and eventual success. R&D groups that join forces and understand that QA extends far beyond the actual tasks of the QA team have a much higher chance of improvement and set high standards for their product’s quality. Implementing the appropriate methodologies and utilizing the right tools lead to an increase in customer satisfaction and retention rate.

    About the Author / Ronit Eliav

    Ronit Eliav is the Director of Product Marketing at Panaya. With a deep interest in digital transformation and the latest IT trends, she enjoys creating thought leadership around such topics as agile and continuous delivery, DevOps, quality assurance and IT modernization projects. Connect with her on LinkedIn.

  • Micro Focus Unfurls Functional Testing Service

    Micro Focus Unfurls Functional Testing Service

    The merging of software assets from Micro Focus and Hewlett-Packard Enterprise (HPE) that occurred earlier this year included StormRunner, a series of load- and performance-testing tools. Micro Focus recently extended that portfolio with the introduction of Micro Focus StormRunner Functional, a suite of functional testing tools delivered as service that can be employed to test applications running on any browser, operating system or mobile device.

    Shane Evans, director of product management for application delivery management for Micro Focus, says StormRunner Functional spans a variety of automated tests addressing unified functional testing (UFT), UFT Pro and Selenium browser testing tools that include elements developed by both HPE and Micro Focus. The Micro Focus StormRunner Functional Testing service also includes pre-built integration with Google Analytics, Micro Focus Octane, GIT and Jenkins to streamline the testing process.

    Interest in automated testing has increased substantially in the last few years, says Evans, because developers are under more pressure than ever to roll out new applications and associated updates faster. Unfortunately, most developers don’t have the time required to test applications themselves, or there is simply not enough money in the budget to hire quality assurance teams to test every element of an application. Micro Focus StromRunnner Functional provides an alternative approach using a set of testing tools that have been widely employed across the enterprise for decades, says Evans.

    Evans notes Micro Focus is pursuing a comprehensive approach to application delivery management spanning everything from testing tools to application lifecycle management. Micro Focus is making the contention that most DevOps teams are a lot more interested in getting access to the results of a test than managing the actual testing technologies themselves, says Evans. In addition, advanced analytics and machine learning algorithms, which now are standard elements of testing process, require access to massive amounts of data to be effective. That data is a lot easier to aggregate in the cloud using SaaS applications than in an on-premises environment. In fact, Evans says, those requirements are among the factors driving a wave of recent acquisition across the application testing sector.

    Arguably, the biggest testing challenge organizations now face is moving away from processes based on IT Information Library (ITIL) guidelines to a more DevOps-centric process, under which continuous testing is more embedded within the application development process, Evans says. Functional testing is only one element of the testing process, but it has become more critical as end users become less forgiving of substandard digital experiences inside and out of enterprise IT environments. Regardless of who inside the organization is responsible for testing applications, the expectation now is that every application experience inside the enterprise will be on par with a consumer application experience.

    — Mike Vizard

  • The Future of Continuous Testing: Fail Faster

    The Future of Continuous Testing: Fail Faster

    In the early days of Agile and Lean development practices, a very noticeable gap appeared between dev teams and ops and QA. Development was accelerating at a lightning-fast pace, but testing processes could not keep up, slowing down the entire life cycle and preventing organizations from achieving continuous delivery. Traditional testing methods, which rely heavily on manual processes and frequently updated GUI tests, proved too cumbersome and inadequate for our new lifecycle timelines.

    The era of testing as an afterthought is dying out, and continuous testing has emerged as the new standard. The idea behind continuous testing is pretty straightforward: Implement a systematic approach towards improvement that incorporates not only automation, but also all of the other tools and processes that will help reduce risk before software progresses to release and production. Organizations embracing this are rethinking testing as a continuous process that starts from the sandbox, rather than a precursor to release.

    Continuous Testing: Achieving ‘Fail Faster’

    At xMatters, we strive to a testing strategy that supports our goal of committing to production multiple times per day—one that we’ve been diligently moving closer to, having progressed from releases every quarter to every month to every week to now, twice per week. Our approach is to conduct unit tests at the team level, code quality, sonar and static analysis, and then deploy to production at limited percentages that we bump up in stages. Our overarching mantra is “fail fast,” because the faster and sooner you fail, the faster and sooner you can fix what’s broken, and the less resources and time you will spend troubleshooting and resolving issues. This, of course, translates to real cost savings.

    Correlating failure to actual events is the biggest hindrance to the continuous testing dream, as pinpointing root causes is an incredibly time-consuming process. This is where automation will take us to the next level, if implemented in a smart way. The future of continuous testing will be adaptive so as to determine the causes of failures faster. We need to automate as much as possible the process of predicting where code quality issues are and identifying the precise data correlations that will enable us to fix issues rapidly.

    Part of this adaptive testing approach will include automating intelligent regression testing. We don’t need to run a black box test in every instance, for example, which can take hours every time. We’ll save incredible amounts of time, money and developer resources if we design tests that can detect new changes and only test those. Smarter regression testing will result in precious time saved that ultimately impacts—and can even make or break—the speed of innovation.

    Another critical consideration to smarter testing is making sure to account for a comprehensive understanding of risk. We need to incorporate real business considerations into our understanding of risk and bake those into our tests as well. We should hold our software to not only standards of code quality and integration success, but also to meet real business expectations for areas such as security, compliance and reliability. Testing from the bottom up must be done in tandem with top-down analysis to account for the full spectrum of potential risks that have the potential to impact the bottom line. Machine learning systems can also be extended to learn new business applications as new functionalities are added and assist in creating test cases that serve to enhance the quality of the application.

    Developing these intelligent automation tools remains our biggest barrier to achieving the continuous testing dream, and while many organizations are gunning for it, most are still mired by manual processes. A global study conducted earlier this year by CA Technologies, for instance, demonstrated that while 75 percent of organizations identify continuous testing as important, just 20 percent report a high quality of test automation coverage.

    It’s worth noting that not every aspect of testing can be successfully automated. The vast majority of functional tests will almost certainly not be manual processes—but what about user experience testing? We have yet to conceive of an approach to this that can truly be automated. Because nothing beats a real human’s assessment of experiential quality, this will be one of the areas of testing that will likely remain manual. Thus, while our goal is to automate 100 percent, realistically this won’t be the case.

    The end goal here is not only to achieve the dream of continuous delivery, but also to eliminate distractions as much as possible. The more repeatable processes you can automate, the fewer distractions your team will encounter, and the more they can focus on creativity, strategy and building new features and capabilities instead of fixing old ones. This is what will ultimately determine your ability to innovate, as you’ll no longer have to sacrifice either time, scope or quality.

    About the Author / Deepa Guna

    Deepa Guna is Senior QA Architect for xMatters. She has been working in technology since 2003. Connect with her on LinkedIn.