devopsy.kinsta.cloud

Tag: testing

  • 4 Ways to Tackle Common Test Automation Pitfalls

    4 Ways to Tackle Common Test Automation Pitfalls

    The demand for higher-quality products and new features delivered at breakneck speeds has the average engineer struggling just to stay afloat. With demands rising but resources not always following suit, engineering teams are embracing automation as a key testing option.

    According to a report by advisory firm KPMG, test automation increased by 85 percent from 2015 to 2017 across all industries. Automation is one way to address this accelerated development schedule, but it can also introduce other problems. To ensure that doesn’t happen, there are four common pitfalls that engineering teams need to look out for.

    Problem No. 1: Right Job, Wrong Tools

    It’s tough to build a shed with a spatula and a sewing needle. Likewise, you can’t build a successful test automation suite if the tools used don’t fit your company’s testing strategy.

    The solution: Pick the right framework for your suite.

    The first question you need to ask yourself is: Should you build or buy your test automation framework? The answer likely depends on how mature your testing structure is. Smaller teams new to the technology typically are best-served starting with an off-the-shelf enterprise solution. This approach makes it easier to get up and running quickly, while having the support to handle software bugs along the way.

    Larger, more mature testing teams may have more to gain from an open source automation framework. This cost-effective approach provides flexibility and customization, but often requires advanced technical expertise and the luxury of time to build such a solution.

    Problem No. 2: Living Fast, Dying Young

    Many automation projects never make it off the ground because teams move too fast and automate too much, too soon. Inserting automation into the delivery pipeline prematurely will cause workflow and build issues more often than not.

    The solution: Build test automation into your pipeline incrementally.

    It may be tempting to run full speed as soon as you are set up, but the truth is you need to take your time for developers to build confidence in the testing process. Once that confidence is established and you’re receiving valuable feedback, you will have the full team’s support as you continue to scale your automation strategy.

    Problem No. 3: Automation Without Documentation

    Automation can make testing easy. Sometimes too easy, to the point where engineers will simply kick back a failed test without looking into the reason it failed in the first place. Oftentimes, these failures are the result of an outdated testing script or an issue with the test environment, and not with the code at all.

    The solution: Triage automation test results just like manual bugs.

    Instituting an intermediary triage stage will save your engineering team time in the long run. There’s no need to treat your processes for automation results any different than you would bugs found manually. Having access to materials such as activity logs and screenshots will only help to strengthen and speed the triage process.

    Problem No. 4: Setting and Forgetting

    Automation makes testing quicker and easier. That’s why you invested in it in the first place, right? Unfortunately, there is more to such solutions than just plugging them in and letting them run.

    The solution: Take maintenance into account when building your framework.

    Many teams are surprised by the maintenance required to keep test automation suites running smoothly and effectively. They shouldn’t be. Solutions need to be updated continually to address user interface changes, new operating systems and browsers, and automation library changes. The maintenance of third-party dependencies also needs to be considered before automation is used for production test runs.

    Maintaining your automation suite is a continuous process. Make sure you have enough testing resources—either in-house, outsourced, crowdtesting or a combination of all three—to ensure your efforts don’t go to waste.

    Measure Twice, Implement Once

    Make no mistake, automation is more than a quick fix. If you are going to spend the time and money necessary to build a successful strategy, you should make sure to set yourself up with a solid foundation. A framework that is thrown together within days but falls apart in hours will only serve to make automation seem like an unpalatable solution within your testing department. Having the right partners in place and the right solution from the start will help automation become a reality in your company, and can bring the promises of speed and flexibility to life.

    — Drew Horn

  • How Can DevOps Enable Your Software Testing efforts?

    How Can DevOps Enable Your Software Testing efforts?

    Changing market dynamics are pushing the need to include DevOps in your software testing exercise to help meet business objectives in providing a consistent and flawless user experience.

    Digital transformation is not an easy task for business enterprises. There are myriad issues to conquer, such as department silos, legacy systems and a well-entrenched work culture that often refuses to align with the business goals. Needless to say, the imperatives of reaching out to the market quickly and with quality products often are not achieved because of these issues. To eliminate the bottlenecks that end up blighting business enterprises, a growing number of organizations are adopting DevOps, which involves an end-to-end automation of processes, be it for development, testing or deployment.

    DevOps takes the Agile methodology a step further by involving all the stakeholders of an enterprise. The DevOps testing strategy encompasses continuous improvement of software products to address the changing customer preferences.

    In DevOps software testing, the aim is to detect and eliminate glitches proactively to reduce the cost and time to fix issues that can lead to a poor user experience. In the highly competitive digital market, a poor user experience can stymie a business’s market readiness.

    However, businesses should realize that the DevOps software testing approach is a journey and should be leveraged to enhance the user experience on a continuous basis.

    What’s the Key Value Expected from DevOps?

    Considering that the development and operations teams of an enterprise each have different work cultures, systems, goals, tools and approaches, the quality and delivery schedule of applications can be thrown off. Hence, the significance of DevOps testing strategy comes into play to bring synergy between the teams and to enhance the quality and delivery of applications consistently.

    DevOps calls for the integration of development and operations teams, wherein QA is made everyone’s responsibility. The synergy helps in breaking the departmental silos and posits the role of QA as an enabler of value addition and to achieve customer delight. This not only ensures quality, but also enables faster turnaround on application development process. Ultimately, enterprises are better equipped for the changing market dynamics and challenges with their business applications.

    Key Drivers to Adopt DevOps

    George Spafford, research director at Gartner, said:

    “A DevOps initiative must focus on business requirements and not on ‘doing DevOps for the sake of DevOps,’ wherein the methods and tools become more important than what customers need. Organizations must avoid the all-too-common mistake of launching a DevOps initiative before establishing that a business reason exists to do so. For example, instead of focusing on release rates and doing things faster, start with the business value by asking what that will enable.”

    Hence, it’s important to work with a DevOps strategy that resonates with the business objectives. These are some key drivers for considering DevOps.

    • Quick deployment of quality applications across devices, platforms, operating systems and networks.
    • Increasing competition necessitating the faster release of apps.
    • Emerging technological platforms whetting the appetite of customers for better products.
    • The emergence of complex applications with footprints in the physical and virtual world.

    How Does DevOps Enable Effective Quality Assurance?

    • It makes QA a collective responsibility rather than being assigned to a single department or organization.
    • It enables better collaboration between teams through a shift of culture.
    • It ensures faster identification and elimination of glitches.
    • It is responsible for accelerating the time to market and keeps a tab on the software quality through customer feedback.
    • It provides continuous improvement of software through timely releases.
    • It reduces cost due to the absence of rework.
    • It optimizes the utilization of resources and prevents waste.
    • It helps to cut down the software development life cycle (SDLC) by adopting quick sprints.
    • It ensures continuous development, testing, integration, deployment, monitoring and innovation of applications.

    The Need to Measure DevOps

    Digital transformation has necessitated the need for rapid innovation, development, testing and deployment of applications. Post deployment, appropriate customer feedback should be considered and absorbed into the product to improve its performance further. The success of DevOps QA depends on the level of automation, not to speak of the degree of collaboration among various stakeholders.

    In fact, enterprises should use a robust automation software to execute quality assurance across platforms and devices. The automation software should be able to test the code as part of the development sprint on a variety of devices and test environments. The DevOps QA approach should be measurable based on certain metrics to ensure its effectiveness. The measurement platform can include the following:

    • The QA team dealing with continuous integration should be able to work with all the tools to find out glitches.
    • The entire setup should be simple to facilitate quicker learning, adoption and execution.
    • The test data including dependencies should be easy to access.
    • The entire build, deploy, release, verify and rework processes should be streamlined.

    Conclusion

    In the fast-changing digital landscape, the objective of reaching out to the customers quickly and with better quality of products is paramount. This is where the DevOps specialists turn things around with continuous improvement, innovation and deployment.

    As Gartner’s Spafford rightly stated, “DevOps challenges conventional IT thinking with its lack of a standard definition and approach, its constant evolution, and its requirement for acceptance and management of risk.”

    — Komal Lopez

  • Continuous Testing at More than 200 mph

    Continuous Testing at More than 200 mph

    “The old principle of tinkering with an instinctively designed car has long since been superseded by systematic testing of every major component and structure – both before and after the car is fully built and ready to race.”

    —“Understanding F1 Racing,” Formula1.com

    While speed is the most obvious shared goal between racing and software development teams, you might be surprised at the striking similarities between how these teams achieve that goal. Each relies on top talent, funding and innovation to reach the speeds at which their competition cannot, but there’s another oft-ignored component that’s quickly becoming one of the greatest differentiators between first place and everyone else: testing.

    The way F1 teams test their cars has gone through a transformation just as monumental as what we’ve seen in the software world over the last decade. The ability to continually test every major component and structure of a race car or piece of software is crucial, but the ability to quickly respond to the data that comes out of those tests is just as important. It’s continuous testing before, during and after the rubber meets the road—or the software hits the marketplace or app store—that determines the winners.

    The Evolution of Testing

    In the aforementioned Formula1.com article, an all-too-familiar scenario to those in the software world is described:

    “By midway through the first decade of the 21st century, a typical Formula 1 testing programme had become a major exercise in both manpower and logistics…spiralling costs, which invariably resulted in development work that was invisible and meaningless to fans …”

    Replace “Formula 1” with “software” and the above excerpt is just as factual for those who don’t know the struggles of yesterday’s approaches to software quality, and just as painful for those who personally experienced them. Manual processes, insufficient test automation technologies, shared unrealistic environments, lack of trust between teams and out-of-control costs are constraints of the past for today’s top-performing teams, but they remain for many still relying on legacy testing approaches and technologies.

    What Continuous Testing Looks Like Today

    Some of the racing enthusiasts from Tricentis’ leadership team were invited earlier this year to meet the Haas F1 racing team during their 2018 preseason testing at Circuit de Barcelona-Catalunya. We witnessed firsthand the remarkable transformation that Haas and F1 racing as a whole have embraced. Track testing is where, as Formula1.com says, “the steady evolution that happens to all Formula One cars during the course of their life begins, a constant improvement of tiny details and setup.” This constant evolution and iterative, rapid innovation is exactly the goal of high-performing software organizations today.

    After spending the day at the track, we noted the following approaches and methodologies that we see in innovative continuous testing and DevOps practices today:

    • F1 cars are continuously delivering data and feedback to the organization.
    • The cars have millions of data points and thousands of sensors, providing actionable data.
    • These data points and sensors are grouped into core processes, and isolated by key questions.
    • Test runs are done with the cars in as realistic environments as possible.
    • The speed at which that data can be received, interpreted—and most importantly—acted upon, is the difference between high-performing and low-performing teams.
    • The tiniest changes to the car make enormous differences. Nobody is showing up in an entirely different car, they’re making minute changes.
    • These cars cannot fail on race day or the race is lost.

    Today’s high-performing organizations are treating the delivery of software in the exact same manner as the world’s leading racing teams. Innovative technologies are being put into each stage of the delivery life cycle that provide invaluable feedback that helps enterprises better understand risks to their business. Being able to rapidly analyze—and act on—this amount of data has become imperative for avoiding disruption. Testing software in environments that are as realistic and as close to production as possible is mandatory, and those environments need to be available when they’re needed most. These needs have driven the adoption of what’s known as continuous testing—an evolution of traditional test automation that has proven unable to support software’s increased complexity, and expectations around the pace of development and delivery.

    What Questions Are F1 Racing and Software Enterprises Asking?

    While each have their own unique challenges, there are many similar questions that need answers—in real-time, and at any development, testing, or release stage—that the racing and software industries share.

    • How fast are we today and how fast will we be tomorrow?
    • How fast is our competition, and how fast will they become?
    • What are we measuring today that’s impacting the business?
    • How quickly can we introduce change?
    • What data are we using to validate that change?
    • What’s the risk of not changing?
    • How low can we get our mean-time-to-repair?

    The fastest time to market—or time to the finish line—is how you win the race. Your developers or engineers don’t just build the fastest car, or the most popular app to today’s standards and then call it a day. They’re constantly looking at what speed, UX, functionality or accessibility will be needed to be the best tomorrow, and then iteratively adding those improvements as quickly as possible. They don’t stop engineering, ever. And testing shouldn’t be performed any differently. Each new feature, condition, region, device, persona, currency, language and OS that your app needs to offer, integrate with, accept or run on is going to need continuous testing before, during and after that app’s release.

    Whether you’re responsible for the quality and performance of a billion-dollar race car or the applications for a billion-dollar enterprise, speed and quality are your path to the winner’s circle. Continuous testing ensures that you remain there.

    — Noel Wurst

  • Accenture Aims to Infuse AI into App Testing

    Accenture Aims to Infuse AI into App Testing

    Accenture plans to infuse greater amounts of artificial intelligence into application development testing following its acquisition of Real Time Analytics Platform Inc., which provides tools that make use of machine learning algorithms, neural networks and natural language processing (NLP) to analyze every stage of the software development life cycle.

    Kishore Durg, senior managing director and growth and strategy lead for Accenture Technology, the IT services arm of Accenture, will incorporate the AI technologies developed by Real Rime Analytics Platform into the Accenture Touchless Testing Platform, which is a suite of open source and proprietary testing tools that Accenture has infused with advanced analytics and cognitive computing technologies.

    Durg said Accenture is committed to automating application testing using AI technologies to help organizations increase the productivity of application development teams. At a time when organizations are investing in a broad range of digital transformation projects, the rate at which applications can be built, tested and deployed has become a critical issue.

    Most organizations on their own won’t be able to aggregate enough data to drive machine learning algorithms and neural networks to drive a meaningful AI model. Those models require access to massive amounts of data to be trained to identify potential issues within any application. Because of the size and scope of the projects Accenture works on, organizations that employ its service can benefit from all the data that Accenture can bring to bear to train AI models, Durg noted.

    The quality of the data used to train those AI models is also as important as the amount of data, added Durg. Accenture has the resources required to cleanse data before it gets fed into an AI model, as well the expertise needed to eliminate any biases that might get built into any algorithm being applied to that data, he said.

    As AI continues to evolve, it won’t be long before every application incorporates some form of an AI model. Each of those models not only needs to be trained, but they also need to be updated continually—and eventually replaced by more sophisticated models as more data becomes available. In effect, that requirement will create a need to extend DevOps processes to how AI models are incorporated within an application.

    At the same time, it’s clear that large swaths of DevOps processes involving, for example, application testing can be increasingly automated using a variety of machine learning and network networks, also known as deep learning algorithms.

    In theory, that should accelerate the rate at which applications are developed and deployed, while testing processes that are augmented by AI technologies should result in applications that are more secure, as many more vulnerabilities will be discovered long before an application gets deployed in a production environment.

    In fact, the inclusion of advanced algorithms across a DevOps toolchain might accelerate adoption of DevOps processes. After all, most IT organizations today have embraced DevOps unevenly largely because of the complexity involved. AI technologies one day soon may eliminate that issue altogether.

    — Mike Vizard

  • The Software Testing Metrics That Make Or Break DevOps Success

    The Software Testing Metrics That Make Or Break DevOps Success

    Organizations today are racing to develop software at a breakneck pace through efficient DevOps processes. But it’s important for them to avoid losing sight of the forest for the trees. If too much of that speed comes with quality sacrifices that degrade the user experience, the rapid software iterations hurt the business more than they help it.

    “Organizations adopt DevOps to accelerate the delivery of innovative software that creates positive user experiences,” said Clark Golestani, former CIO at pharmaceuticals giant Merck and current managing director of C Sensei Group. “If you start accelerating delivery without also rethinking your approach to quality, you will end up rapidly delivering updates that drive your customers to competitors.”

    This is why continuous and automated testing is so essential to DevOps success. But instituting effective and streamlined testing is easier said than done. Organizations must continually tweak what, how and when they test their software to achieve the optimal test coverage while introducing as little friction as possible into the software delivery pipeline. It’s a delicate balance, and the only way to achieve it is to keep meticulous track of metrics that evaluate how well the testing process is improving quality throughout the development life cycle.

    “As the saying goes, ‘You can’t manage what you don’t measure,’” said Andrew Magarie, senior market impact consultant at Forrester Consulting. “So firms need to be able to define and track the right metrics that can help enable quality at speed while minimizing risk.”

    Magarie and his team at Forrester recently conducted research into 94 different quality metrics frequently measured by DevOps organizations across four development stages: build, functional validation, integration and end-to-end regression testing. The study showed that the most effective test metrics that bubbled up across all of stages tended to clump into at least one of three categories.

    Risk-Focused Metrics

    According to Magarie’s report, the most mature and effective organizations fundamentally chose testing metrics that best helped them understand business risk posed by the software throughout the development process.

    “For this study, we use the following definition of business risk: Any application shortcoming that impairs the end user’s (or customer’s) expected experience and ultimately erodes confidence in the business,” he explained.

    For example, during the build stage mature organizations consider measuring the number of automated tests prioritized by risk as one of the most important assessments.

    “Tracking ‘unit’ tests prioritized by risk is the key. As the code base evolves, developers and testers need immediate feedback about change impact. This feedback is significantly more actionable if prioritized by level of risk,” wrote Magarie.

    As he explained, business risk will be different for every application, and the complexity of an organization’s architecture and transaction dependencies adds more variables to the equation. However, organizations need to seek ways to measure ‘relative risk’ across the software development life cycle (SDLC) if they truly want to take a disciplined approach to incrementally improving quality in their software.

    Contextual Metrics

    Forrester found that simple “counting” metrics don’t offer QA nearly as much value as contextual metrics. So simply tracking the number of tests will not provide teams with as much actionable data as the measurement of the ratio of requirements covered by tests. Approximately 69 percent of mature organizations say running functional tests and checking against covered requirements is a top measurement and 75 percent say the same about API tests run against functional requirements.

    Holistic UX Metrics

    The most mature organizations recognize that while application-specific or team-specific QA metrics are important, the ultimate goal is ensure the quality of the user experience (UX) across their entire interaction with an application. This means measuring the thoroughness of end-to-end regression testing is crucial. Approximately 70 percent of leading DevOps organizations say they believe the percent of automated end-to-end test cases as a top metric.

    — Ericka Chickowski

  • Testing Heroes to Recognize Individual Contributions to Continuous Quality

    Testing Heroes to Recognize Individual Contributions to Continuous Quality

    Testers have always been great at championing the superstars within the testing community, but who else outside of testing has been listening? Tricentis has launched a campaign aimed at making sure the rest of the world knows just how invaluable testers are, especially in today’s hyper-accelerated release cycles, and with user experience and customer satisfaction becoming the enterprise’s success metrics of choice.

    Testing Heroes will be an annual program in which software testers can be nominated for a variety of award categories, two of which—Tester of the Year and Test Manager of the Year—will be awarded a free trip to Vienna, Austria, to accept their award onstage at Tricentis’ Accelerate 2018 conference.

    Testing Heroes is split into six different categories:

    • Tester of the Year
    • Test Manager of the Year
    • The Explorer
    • The Automator
    • The Performer
    • The Risk-o-Meter

    A Unique Twist

    While most winners are decided on by the awarding organization, the 2018 Testing Heroes will be determined by the level of support from each nominee’s surrounding organization and network. The final round of Testing Heroes will be a voting round. Those whose nomination makes it to this round will be provided with a variety of ideas for encouraging others to rally behind their candidate to get them the recognition they deserve. There are myriad ways to spread the word in your office about the outstanding work being done by one of your own:

    • Encouraging others in your company to vote through Slack, email or an internal newsletter.
    • Recording a short, campaign-style video to share through social media.
    • An iPad loaded to the candidate’s voting page placed in a break room, kitchen or other communal area.
    • Desk-to-desk canvassing (to those who don’t mind you popping in unannounced!).

    One of our initial goals for this campaign was to learn more about the amazing contributions by those in the software testing community, but we quickly realized there is a bigger reward here: When a nominee’s peers bring to light the amazing work being done by one of their own during the campaign process, software testers’ efforts hopefully will be more understood and appreciated by a larger audience than they are currently. This is the real goal of Testing Heroes.

    We encourage you to nominate and then rally behind your own Testing Heroes today!

    — Anand Akela

  • Best Practices for Application Security Testing in the Era of DevOps and AI

    Best Practices for Application Security Testing in the Era of DevOps and AI

    Application security testing is no easy feat. And yet, it’s usually the first topic that most articles about application security address. The reasons are simple: As the pace of application development techniques (and their inevitable vulnerabilities) evolve, AppSec personnel have found themselves caught between the desire to keep pace with their management of security testing requirements and their ability to allow the developer teams to operate in the modern, fast-paced ecosystem of DevOps and artificial intelligence.

    To better understand the best practices for conducting AppSec testing in the era of DevOps and AI, it is important to first appreciate the technologies available to us, that allow us to do so.

    At the heart of the modern application security testing,  there are five main technologies to be aware of:

    1. Static application security testing (SAST)
    2. Dynamic application security testing (DAST)
    3. Software composition analysis (SCA), third-party code
    4. Fuzzing
    5. Penetration testing

    These five technologies account for the bulk of most of the activities around application security testing. However, many aspects of application security testing are rapidly changing, including efforts to incorporate automation and artificial intelligence into the equation.

    By including automation or AI, for example, code testing is allowed to be released in rapid fire sequence, with production use approvals baked inline through automated logic and decision-making. This idea of continuous integration/continuous delivery (CI/CD) can lead to a widening of fractures between the security and development teams. Therefore, DevOps users today need the ability to program their code and test it through automated code scanning techniques before pushing the code into secure staging environments.

    To successfully conduct application security testing in today’s era of advancing developer operations through automation and artificial intelligence, there are three critical best practices to keep in mind.

    1. The Earlier, the Better: Introduce appSec testing into the software development life cycle early on for maximum results: Regardless of the type of testing being conducted, waiting until the very end of the development process will only create more work. Users will either end up with more bugs than they could feasibly fix or find themselves a part of an overworked AppSec team as they attempt to document and deliver on discovered vulnerabilities and code flaws. Statistically speaking, code changes that happen later in the development life cycle are also more costly to the business.
    2. Conduct testing of business-critical applications as often as possible: When deciding which assets are next in line for application security testing, one should consider the applications and systems that are of critical business value to the organization’s revenue stream. Think of these systems in particular as the “crown jewels” for the majority of attacker motives, since they define the amount of money a given organization will procure for that day and, if attacked, could disrupt the business in myriad other ways.
    3. Test for third-party code security and interoperability flaws as you would your own software: The growing complexity of modern applications means there is a need to leverage open source or third-party components. As this is the case, these components must also receive the same level of scrutiny as the applications in one’s own enterprise. All third-party code referenced in the software delivery process must also be checked by performing software composition analysis for security and interoperability flaws.

    While the pace and techniques for AppSec testing will undoubtedly continue to advance, and while there are some AppSec testing techniques that may never become obsolete, it is imperative to ensure that one’s system for maintaining application security testing always adheres to the best practices to preserve a secure software environment for the future.

    — Anthony Lauro

  • How Automated Testing Can Help the Latest C-Level Issue: Quality Software

    How Automated Testing Can Help the Latest C-Level Issue: Quality Software

    Every year developers gather in San Jose for Apple’s annual WWDC, where software quality is typically a key point of discussion. This year, the company progressed from its 2017 buggy software debacle in full force—it’s a heads-on effort, marked by the industry leader naming 2018 its year of software quality.

    This raises some larger questions around the software industry itself. Has modern IT become so complex that the concept of “bug-free” software is no longer realistic, particularly for a company the size and scale of Apple? Or did Apple recognize this pain point and shift its focus to  software quality in an effort to be the harbinger of change?

    Of course Apple isn’t alone when it comes to the struggle with quality. Buggy software from organizations across the board persists today, but it simply cannot be ignored. Poor quality software creates long-term effects of waning customer confidence, brand erosion and revenue loss. These are very real concerns for the entire C-suite. Quality is no longer an issue limited to Dev or Ops—it’s critical to every organization’s success.

    For example, consider the fact that 88 percent of consumers say they’d abandon an app if they encountered bugs or glitches (source: QualiTest). Nearly half those people are likely to discontinue using an app entirely if they experienced a bug every day. And 32 percent say they would abandon an app the moment they encountered a glitch.

    The cost of acquiring a new customer is 5 to 25 times higher than retaining an existing one. Getting software right the first time through a culture of continuous testing is critical to every business, as it translates directly to retaining customers.


    Budgets for QA and testing are expected to rise to 40 percent in 2019 from 31 percent this year. However, one-third of developers still admit they don’t have enough time to properly test before software deployments. As everyone in this business knows, developers run fast. It is of the utmost importance to ensure software actually works well and will provide the expected customer experience. The garbage in, garbage out model is not an option. To fill the gap, more and more organizations are turning to automated testing.

     

    Any repetitive, labor-intensive task is typically a good candidate for automation, including those associated with testing. Automated testing involves execution of prescripted tests on a web or mobile app, with test data accessible in detailed reports to help development teams quickly identify issues in the code, as well as any issues in the automated scripts themselves.

    Performance of the automation framework and its scripts is crucial, and should be a high area of focus. Prescripted tests fall into different categories that comprise the test pyramid. They include:

    1. Unit Testing
    2. Code Coverage
    3. Code Syntax Checks
    4. Security Tests and Checks in Code
    5. Integration Tests
    6. UI Tests for Web, Hybrid and Mobile Apps – on multiple cross browsers on various platforms and OS versions

    Once created, automated tests can be run again and again at no additional cost—which is particularly helpful when source code is changed. Since it is much faster compared to manual testing, automated testing translates into considerable time and cost savings.

    Automated testing is especially valuable given the breadth of browsers, operating systems and devices in the marketplace. Consumers today run apps on a countless number of these, so developers must have access to every combination imaginable to deliver the highest level of performance. What good is your app if it doesn’t run (and, by extension, monetize) properly when it gains popularity in some part of the world with users on a different platform from the one you tested on?

    Ensuring that software works on as many options as possible allows a business to reach a maximum number of users. This is where automated testing can help, by executing thousands of different complex test cases during every test run. Testing efficiency can be achieved by executing tests in parallel or grid, and executing CI/CD jobs in parallel. For example, one code is deployed in staging, teams can simultaneously start jobs to test against browsers and cross-browsers, apps and localizations, etc. The result is attaining coverage that would be impossible through manual tests. Teams can scale apps fast, maximize CI/CD workflows, and ship higher quality software faster.

    Attaining CI/CD and automating all the tasks involved in creating a real-time test system is one of the arenas in which the next big software battle will be fought. More importantly, as software continues to eat the world, the C-Suite can no longer ignore the impact of poor quality software devouring your business. Apple’s bold step of identifying 2018 as the year of software quality is a call for other organizations to do the same.

    Automated testing is a key piece of the solution. How is your organization meeting the software quality challenge, and by extension, the evolving standards of customer experience?

    — Ritesh Arora

  • Continuous Testing Insights from 2018 DevTest Research

    Continuous Testing Insights from 2018 DevTest Research

    The year is far from over, but there already several interesting DevTest surveys worth your attention. These studies don’t just quantify the obvious; they actually report some unexpected findings regarding how far and how fast we’re advancing, and offer some very specific advice on what’s needed to improve.

    We strongly recommend that you spend some time reading all three of these surveys in their entirety. However, in case you’re short on time (or impatient … or both), we wanted to highlight the findings that are most pertinent for readers practicing or researching Continuous Testing.

    Sauce Labs – Testing Trends for 2018: A Survey of Development and Testing Professionals

    [Read the complete report]

    2018 marks the fourth annual “Testing Trends” report, which is based on a global survey of more than 1,000 technology professionals responsible for developing and testing web and mobile applications.

    Key findings in terms of testing include:

    • 87 percent report that management supports test automation initiatives.
    • 45 percent expect to increase spending on test automation in 2018 (55 percent at large companies).
    • The number of respondents with high levels of test automation dropped to 28 percent in 2018 from 32 percent in 2017 .
    • The release cadence is actually slowing, with hourly deployments dropping to 5 percent from 14 percent and daily deployments dropping to 27 percent from 34 percent.

    In other words, everyone recognizes the value of test automation and most companies are willing to invest in it. However, test automation rates are actually decreasing, while Agile and DevOps adoption are steadily increasing. In the 2017 report, test automation rates increased slightly, and delivery speed also increased slightly. The 2018 reported a similar correlation: Test automation rates decreased, and the release cadence slowed down.

    GitLab – 2018 Global Developer Report

    [Read the complete report]

    This expansive survey polled 5,296 software professionals from around the world. The majority of respondents were software developers or engineers who worked for small- to medium-sized businesses (SMB) in the hardware, services and SaaS industries.

    Testing wasn’t a common topic in this development-focused research, but it did earn a prominent spot in the report. Testing was the No. 1 response to the question, “Where in the development process do you encounter the most delays?” A dubious honor—but not a surprising one. Last year’s DevOps Review polled an entirely different audience and came up with the exact same finding.

    VersionOne – 12th Annual State of Agile Report

    [Read the complete report]

    The 12th edition of the world’s longest-running Agile study found that while 97 percent of the 1,492 respondents’ organizations are practicing Agile, 84 percent report that their Agile adoption is not yet mature.

    Respondents feel strongly that two testing-related items would help them increase process maturity across both Agile and DevOps:

    • 83 percent want end-to-end traceability from business initiative through development, test and deployment.
    • 82 percent want better identification and measurement of risk prior to deployment.

    Respondents also reported a relatively high level of adoption of development testing and “shift left” testing techniques. Adoption levels were reported at:

    • Unit testing – 75 percent.
    • Coding standards – 64 percent.
    • Pair programming – 36 percent.
    • TDD – 35 percent.
    • BDD – 17 percent.

    Testers might also be interested in the survey’s feedback on Agile management tools. Usage rates were reported at:

    • Atlassian Jira – 58 percent.
    • VersionOne – 20 percent.
    • Microsoft TFS – 21 percent.
    • HP (now Micro Focus) Quality Center / ALM – 14 percent.

    The most highly recommended tools were VersionOne, Jira and CA Agile Central. HP Agile Manager, Hansoft and HP Quality Center /ALM were the least likely to be recommended.

    — Cynthia Dunlop

  • Attaining Continuous Testing in DevOps through a Templated Approach

    Attaining Continuous Testing in DevOps through a Templated Approach

    The continuous demand market for flawless applications is being addressed through advanced software development techniques along with continuous delivery. As such, many organizations adopted the shift-left approach and agile best practices—which suggests the motto, “test early, test often,”—through which they get to imbibe QA practices and testing early in the software development pipeline. That also includes continuous testing for many organizations.

    Continuous Testing in DevOps

    Continuous testing today is the most preferred practice by leading software companies. Continuous testing implies testing through the whole software development pipeline, from product design to product release, by integrating the complete product cycle. Even after the product release, companies test to troubleshoot customer complaints to further improve the product.

    Continuous testing in DevOps involves encouraging the development engineers and operations to practice testing as well, and also makes them a part of the whole cycle instead of being a separate team. Continuous testing involves testing not just from an earlier stage but also through each and every single process. Its goal is to automate almost all the processes as possible.

    A Templated Approach to Continuous Testing in DevOps

    A perfect approach to continuous testing in DevOps should be a right balance between manual and automation testing, and to achieve it, organizations must practice virtualization to plan out the whole product development pipeline successfully. Many organizations are not yet ready to risk it all with continuous testing, as big errors are still a possibility if they don’t really know how to execute it. But, errors in the continuous testing method can be easily rectified as it is practiced through each process.

    Errors are common when it comes to product development; hence, the right set of tools to automate testing can be critical to finding and rectifying errors, since manual testing can miss certain errors. Manual testing also takes a considerable amount of time, which is a huge drawback. A well-planned but carefully structured templated approach can be practiced by companies to make a full transformation toward continuous testing.

    Continuous testing is achieved through imbibing multiple use cases over certain parameters to determine the strength of this methodology. Some of those parameters that would prove to be a sure-fire way of determining whether continuous testing is worth it are distribution, technology, scale and load, environments and security and release.

    Checking the Parameters

    The parameters to completely check a dimension such as continuous testing can be done to create readings based on elimination and weighted scoring to derive results from a use-case scenario. Let us see a brief application of running a hosted public-facing service through these parameters:

    Distribution – The distribution areas and the geographical location of this product are considered.

    Scale and Load – The number of test cases (3,500 automated cases in this scenario, by way of an example), the suites that are used based on the browsers, cloud test platform and the numbers of users are considered in this category.

    Environments and Security – Varied device uses, cross-browser testing, secured cloud environment and approval of releases are considered in this parameter.

    Release – The number and frequency of release and the technology stack are considered in this parameter.

    The tools that are used for this scenario are Selenium web driver, Selenium GRID, Jenkins, Sauce Labs, Sauce On-Dand Plugin.

    The process through which checking the parameters is done is by triggering automated test cases from Bitbucket automation repository using Jenkins during deployments, and the execution of automated scripts in Sauce Lab cloud, as parallel threads are done by interfacing Sauce Lab Plugin with Jenkins.

    The control mechanisms first include testing through smoke, regression and integrated automated tests execution. Then, the test evidence for all of the test cases is captured. Later, as the tests are completed and the screenshots which are captured by automated tests are validated, UAT sign-off is done.

    Some parameters for test automation framework are compared within Selenium and tests completed over various categories, their scores are compared and then they are passed onto the test environment. Some suite, regression suite and integration test suites are carried out during Dev and QA stages, later when the changes from these two environments are deployed. Smoke suite is carried out during pre-production, editorial and the production stages. Finally, the proven results of these stages are derived successfully.

    The proven results of this use-case scenario through these stages are:

    • Parallel testing visibly reduced regression hours from 150 hours to 15 hours.
    • Enhanced test coverage matrix addressed varied device and OS fragmentation, as well as browser combinations.
    • Quality gatekeeper for code promotion to production.
    • Reduced regression issues in production.

    Conclusion

    The example of going through such a use-case scenario helps us better understand how continuous testing can be integrated gradually into a product development pipeline through careful observation, planning and deployment through such parameters.

    Engrafting continuous testing in a DevOps eEnvironment needs careful execution to avoid errors and future defects. And with the right set of tools and a perfect execution, it can be achieved to successfully experience faster time to market and reduction in time and cost.

    For more information on how a templated approach can be applied in different scenarios, watch this webinar.

    — Niranjani Ramesh