devopsy.kinsta.cloud

Tag: QA

  • 5 Tips to Eliminate Knowledge Gaps in Dev and QA

    5 Tips to Eliminate Knowledge Gaps in Dev and QA

    To create and release high-quality software, the software development lifecycle (SDLC) was established using different roles and processes designed to work seamlessly together. This approach is of critical importance because it spans the initial planning phase, all the way through production release and monitoring.

    Roles vary within projects but typically include designers, business analysts (BA), developers, testers, scrum masters, project managers, product owners (PO) and technical architects (TA). Each one brings different expertise and skills to a project but ultimately, they are meant to collaborate and determine how to implement different aspects of the software.

    The amount of processes can also vary for projects based on team size, release schedules, availability of resources and the complexity of the software.

    What is the Knowledge Gap?

    Without a distinction of responsibilities or a roadmap that identifies who owns each task in the SDLC, everyone assumes someone else will complete the tasks. The result is that the majority of these important steps remain incomplete or undone.

    The reason this happens is because a knowledge gap exists among the team. This is especially true when teams are a mix of entry-, mid- and expert-level individuals. Sometimes, assumptions are made regarding the skill set, experience and domain knowledge that each individual brings to the project. Knowledge gaps can also develop at a more granular level, with teams using the same tools but in different ways. As an example, developers and testers might be using a single testing platform quite differently.

    As teams begin collaboration on features for the software, they may encounter the following situations:

    • Each team member has a different interpretation of the project requirements
    • Developers assume testers have their same level of technical expertise
    • A lack of ownership and assigned responsibilities among the team
    • A PO has a feature idea and each team member has different ideas for how that feature should work
    • The TA explains how a feature should be implemented using technical jargon that designers, POs, BAs and testers don’t understand
    • A tester is not allowed appropriate time to complete testing due to tight release schedules

    Identifying Knowledge Gaps

    Knowledge gaps not only negatively impact the release of high quality software, but also team morale, productivity, job satisfaction and empowerment. The first step to finding acceptable solutions is to identify exactly where a knowledge gap exists. Some questions to help teams do this include:

    • What are the common problems that occur in each phase of the SDLC?
    • What processes are in place during the requirements, design, development, testing, acceptance and release phases of the SDLC?
    • How effective is the communication between different roles in the team?
    • Are the responsibilities of each team member clearly identified?
    • How visible is the status of the project at any given time?
    • How often are release cycles pushed to accommodate for more development and testing?
    • Are the teams aware of what kind of customers are going to use the product?
    • Is the velocity of the team stable? How often does it fluctuate and by how much?

    Bridging Knowledge Gaps

    Finding gaps in a team’s knowledge is the first step. The next step is determining how to narrow the divide. The following key factors help bridge the knowledge gap in teams:

    Training

    Sufficient training and tools should be provided to help teams do their jobs better. Proper training has a domino effect on the success of the project. It helps designers understand how to design mockups so developers can effectively implement the feature; testers can attend code reviews without intimidation and use tools more effectively; developers will understand why the testability of a feature is important and understand which tools can aid the development testing effort; the scrum master can better manage project tasks; the project managers can ensure the team collaborates to meet release schedules and deadlines; and finally, stakeholders can have an overview of the project through reports that are generated using various tools.

    Visibility

    For teams to work seamlessly together, we need to ensure the results of everyone’s efforts are visible to the entire team. It is important to understand how each contribution maps to the overall goals of the project. To increase visibility, teams can consider using the following:

    1. Checklists: a list of items to be completed during each phase of the SDLC. If the items in the list are marked as done, then there is no ambiguity about which tasks need attention.
    2. Visual Dashboards: give a high-level overview of the project’s health and status. These can be created on whiteboards, easel boards or in software and during daily meetings, teams can address the dashboard to ensure everyone, including business stakeholders, are aware of the high-level project status.

    Clear Definition of Roles and Responsibilities

    Each person needs to understand his/her role on the team, as well as the tasks they must accomplish on a daily, weekly, monthly or quarterly basis. Goals, objectives and expectations for each team member should be defined and discussed with respective peers and/or managers. This eliminates confusion and increases productivity and task completion.

    Empowerment

    In an age where individuals are increasingly more technical and skilled, what they still lack in their job functions is empowerment and autonomy. Contrary to the conventional wisdom that a project needs a single leader, the leadership responsibility can be effectively divided within each role on the team. Having one point of contact each from the design, development and testing teams means each can help to lead their respective sub-teams and help to ensure everyone within the project is working toward the same goals and objectives. By operating within this structure, the whole team becomes empowered.

    Experimentation

    Once sprints are completed and knowledge gaps have been identified, the whole team should come together to discuss solutions to existing problems. Based on this discussion, the team can start experimenting with different solutions to determine what works well or doesn’t. This constant experimentation and feedback loop helps to make the team more creative and empowers them to come up with solutions that work for their organization.

    In summary, the knowledge gap has been a major obstacle for development teams in reaching their fullest potential. By identifying and reducing these gaps, teams can be more efficient and achieve faster release cycles with higher quality software.

    — Raj Subramanian

  • The Era of Intelligent Testing

    The Era of Intelligent Testing

    The state of the art in software testing has not kept pace with advances in software development over the past 10 years. Modern quality assurance (QA) groups struggle to play their role in a world of continuous delivery, leading to more defects in production, low QA morale and excessive testing overhead. Thankfully, innovations in machine intelligence and test automation have laid the groundwork for a better model—one that will allow QA to keep pace with development, increase productivity, improve product quality, and raise customer satisfaction across the software industry.

    Why We Need a New QA Toolset

    Existing QA solutions were built for a world where software changed infrequently and functionality was highly specified and documented.  This is no longer the case.

    Products Are Evolving Faster Than Ever

    Several forces are converging to accelerate development. Software as a service (SaaS) has removed many of the issues around operating system compatibility, packaging, deployment and versioning that consumed significant development time during the installed software era. Consider that in 2016 Amazon Web Services delivered more than 1,000 new features and services—a pace that would be impossible with packaged software.

    “Development cycles are getting shorter and new features are coming faster than ever…There simply isn’t enough time to test.”

                                                 – QA Leader at Fortune 250 company

    The proliferation of open source and cloud-based features as a service has also reduced the time that developers spend building generic functionality. At mabl, for example, the thought of building our own authentication engine never crossed our minds; we use Auth0. Instead of building our own data processing pipeline, we use Apache Beam and Google Cloud Dataflow. Rather than building our own ML framework, we use Tensorflow. And the list continues; reusable components and cloud-based services save us years of engineering effort, allowing our developers to deliver product features rapidly.

    Continued adoption of agile methodologies has further increased the efficiency of development teams. A recent McKinsey study of 1500 software projects demonstrated that agile teams are 27 percent more productive than those that follow methodologies such as waterfall.

    Finally, continuous integration and continuous delivery are accelerating the pace of change in software. A recent survey by Atlassian suggests that more than 50 percent of software teams are already using continuous integration and/or delivery, and more than 32 percent have plans to move to CI/CD. The same survey indicates that more than 50 percent of organizations are able to push changes to production daily.

    In the end, improved developer productivity means that software products are evolving faster than ever. Unfortunately, this places increased pressure on the QA function. As one QA leader from a Fortune 250 company said in a recent interview with mabl, “Development cycles are getting shorter and new features are coming faster than ever. And we have five productive developers for every QA engineer. There simply isn’t enough time to test.”

    Existing Automation Frameworks Impose Too Much Overhead

    QA rightfully relies on automation frameworks to improve productivity; tools such as Selenium, Appium and JUnit now enjoy widespread adoption. More than 80 percent of the respondents to a recent mabl survey use Selenium today, and QA managers are under constant pressure to increase automation. Unfortunately, while Selenium and other frameworks have helped many companies increase their QA velocity as compared to manual testing, these frameworks have significant flaws:

    • They all require specialized scripting expertise, and given the lack of available talent, teams have very limited capacity for QA automation.
    • They provide limited context in test results, which leaves teams with very little information to help them triage and address issues.
    • At scale, they require their own infrastructure, which takes time to provision, operate and scale.
    • They are tightly coupled to attributes of the product under test that change frequently (xPath, CSS, etc.), resulting in constant maintenance and false positives.

    These flaws only become more painful as the pace of development and change accelerates, compelling many teams to limit their use or discard them altogether, resulting in lower product quality.

    mabl survey – satisfaction with existing testing tools and processes (n = 104)

    “Please indicate the extent to which you agree with the following statements”

    Time Pressures Lead to Unhealthy Team Dynamics

    People in testing roles are increasingly at odds with their peers in development. Unable to keep up with the continuous pace of development, QA becomes a bottleneck to product velocity, leading to tension with their peers in development. This constant pressure also starves QA of its ability to make the very investments in automation and process improvements that are needed to improve throughput. The feeling of always being behind with little hope for improvement often leads to low morale within the team.

    Gaining consensus on the standard of quality has also become difficult for teams, since agile teams often steer clear of detailed functional specifications for each release, preferring instead to focus on user goals and requirements (see: Agile Manifesto). Because user goals and requirements are open to more interpretation, QA and development can disagree on not only the severity of defects but also the validity of the tests themselves.

    Modern software teams are also more open to evolving requirements during a given release cycle (see: “Responding to change” in the Agile Manifesto). Evolving requirements should drive updated test cases, but this requires deep engagement and coordination for already stretched personnel. Rather than testing for quality versus specified behavior, we rely much more on ad hoc and manually intensive exploratory testing to look for defects.

    There is Hope for QA

    While faster feature development, shorter release cycles, evolving requirements and unclear accountability for end-to-end quality all present real challenges for QA, a new generation of QA tools are emerging to confront these challenges head on.

    It Will Be Easy to Create and Maintain Tests

    Unlike existing test automation frameworks, next-generation QA tools will not require specialized scripting expertise. Instead, they will offer intuitive interfaces that allow anyone to quickly create and manage tests. This will help teams expand their test coverage and evolve their tests quickly as their products evolve.

    Tests Will Adapt Seamlessly to Change

    Because they use much more sophisticated, durable methods of simulating user behavior in automated tests, next-generation QA tools will not suffer from the brittleness associated with existing automation frameworks. Tests will not be tightly bound to individual elements within the front-end codebase that change frequently. Rather, they will use machine intelligence to create and maintain sophisticated models of tests, enabling tests to adapt the tests rather than fail when xPaths and other locators change.

    QA Will Run in the Cloud

    Today, teams struggle with the performance, cost and operational overhead of on-premises testing systems. Next-generation QA tools will take advantage of on-demand cloud computing resources to execute tests faster and more efficiently. They will tap into powerful data processing and analytics services to analyze test results and deliver insights. They will be delivered completely as a service, offloading the management and operational burden to tool vendors.

    QA Will Be Part of the Delivery Pipeline

    Next-generation QA tools will be tightly integrated into automated CI/CD pipelines orchestrated by Jenkins, Spinnaker, CircleCI and CodeShip. Tests will trigger automatically whenever teams make changes to the product under test and on-demand. The tools will notify team members when there are potential issues so that they can address them before they affect users. It will be simple to run tests and compare results across builds and environments. Decisions around releases, deployments, promotions and rollbacks will be largely automated.

    The Definition of Quality Will Evolve

    Beyond validating specific assertions in tests, the next generation of QA tools will use machine intelligence to automatically detect and highlight potential regressions in applications. As a result, we’ll think less about tests “Passing” and “Failing” and more about the extent to which changes improve or degrade the user experience. We will focus less on code coverage and more on product coverage. We’ll look beyond basic counts of passing and failing tests in favor of a more on holistic approach to assess the risk associated with a given build or deployment.

    Tools Will Be Deeply Embedded Within Our Teams

    Next-generation QA tools will effectively serve as extensions to development teams. They will share test results and insights through Slack, Jira email and other common communications channels. They will take feedback and training from the entire team—including product, development, QA and customer success—and incorporate what they learn into their tests. Most importantly, next-generation tools will describe tests and results in plain language, and they will provide useful context to help developers reproduce and fix issues, just as an effective QA teammate would.

    The Era of Intelligent Testing

    Software quality assurance has never been easy, and the accelerating pace of software development in recent years has made it even more challenging. Thankfully, a new generation of tools are emerging to address this challenge. Incorporating cutting-edge machine intelligence, delivered from the cloud and embedded into the modern development workflow, these tools will dramatically improve the effectiveness of QA, ushering in a new era of efficiency and innovation across the software industry.

     

    mabl survey – user sentiment (n = 104.  For simplicity, combined “Agree” and “Strongly Agree” into “Agree” category,  combined “Disagree”, “Strongly Disagree” into “Disagree” category.)

    “How satisfied are you with the following aspects of the testing process at your company?”

    — Dan Belcher

  • Dissecting the Role of QA Engineers and Developers in Functional Testing

    Dissecting the Role of QA Engineers and Developers in Functional Testing

    Until a few years ago, quality assurance was generally considered low-level R&D. As a result, quality assurance (QA) teams were characterized by a low budget, lack of attention and, more often than not, poor attitudes. Moreover, QA engineers were considered the enemies of developers and the product.

    However, today, this is no longer the case, and QA is part of the entire software life cycle. Today, organizations understand how a product’s quality affects the brand’s reputation and customer retention. In the Agile ecosystem, QA engineers are no longer solely focused on testing; they also fill an important role in the planning and delivery domains. The shift left approach—where testing is performed earlier in the life cycle—requires QA engineers to have higher technical abilities and involvement in the whole CI/CD process. Nevertheless, QA managers, who usually encourage assimilation in the scrum team, as well as rapid feedback for user stories and automation, must not forget the main objective of quality engineers: to ensure a high-quality product and a seamless user experience for customers.

    High quality demands testing and verifications. There are different types of testing that can be conducted to meet quality goals. In one of its blog posts, Testing Excellence describes almost 60 types of testing that can be performed on a product. We’re going to look at one that is quite clearly one of the most important types of testing: functional testing.

    Different than other forms of testing, functional testing is considered one of the most sophisticated test type, and the most comprehensive one.

    Same Team, Different Responsibilities

    One important fact every QA engineer must remember is that developers develop and QA engineers test. Even though QA and developers are part of the same scrum team, their work, methodologies and state of mind are completely different. In cases in which the significant contribution of QA isn’t visible to the whole team, testing might be considered an easy and clear task. Quite the contrary; testing an application is much more complicated and entangled than developing it. When planning tests and executing them, QA engineers must take into account several aspects developers likely never consider.

    Understanding How the User Thinks

    The primary distinction between dev and QA is state of mind. While developers write pieces of code that later become features in the application, QA is expected to understand how real users think, engage customers to learn business scenarios, and confirm that the application satisfies customer needs.

    Let’s look at eBay, for example. Every veteran R&D member will agree that eBay isn’t just a simple shopping application, but a comprehensive system that includes a whole set of capabilities on both web and mobile that requires attention and deeper understanding to determine whether they really function correctly. In such a rich and mature online platform, we can identify several types of users and many cases in which the application may be utilized. A seller performs different steps than a buyer and requires different functionality, while a support agent needs an entirely different version of the application. When planning tests, all these personas must be taken into consideration with a particular test plan for each. In other words, testing functionality is not just simulating user operations.

    In addition, eBay is not a standalone application—its web application lives inside the browser and subsequently depends on that browser’s rules and behaviors. This is true for eBay’s mobile apps, as well, which must meet the OS constraints to function correctly, whether it’s navigating to the seller’s store or making a purchase. This makes compatibility testing an integral part of functional testing. Most developers don’t think about the project availability matrix when they code, so QA is required to contemplate all the ways the customer will engage with the application and verify functionality for each.

    Viewing the Application as a Whole

    Dev and QA differ in their point of view—an obvious but important distinction. While developers focus on developing user stories, QA engineers test the user story, the feature it is part of and the application behavior with the new code. Integration testing is a crucial part of the application verification effort and requires a broader view of the application and how the new functionality is part of it. End-to-end testing is the fulfillment of the above two aspects, validating that real user scenarios in the context of multiple regions in the application behave as expected. End-to-end tests increase test coverage and reduce the risk of making the new code part of the application.

    Paying Attention to User Experience

    As noted, testing the application functionality is merely one step in functional testing. Testing the operations available in the application encompasses another type of testing—the user experience. If a new feature is developed, but users do not have easy access to it or cannot understand how to use it, it is as if the feature doesn’t exist. QA engineers must incorporate user experience as a main metric of features so users will not have room for error, which can lead to misuse and frustration.

    For instance, eBay’s “all categories” page previously displayed all the categories by default, even though some of the categories had no items. While this was the correct behavior from a functionality standpoint, it caused user confusion and mistrust. Consequently, the “all categories” view was changed to show only categories with items.

    Looking at the narrow current task, developers might implement features as they are told without questioning or contemplating the specifications. In the same way, testers who test features without realizing how customers perceive them are missing an important benefit of the test.

    The above three aspects—thinking like customers, testing the application as a whole and paying attention to user experience—revolve around the way developers and QA engineers relate to new features. While developers tend to follow the specification guidelines, QA engineers must “think outside the box” when it comes to user consumption of the feature.

    There are two other ways in which QA work is more complicated and yet invaluable—test maintenance and test automation.

    Maintaining High Quality Throughout an Efficient Release

    In general, developers split their time between developing new features and fixing defects. QA work is much more diverse. A substantial portion of the release is spent on testing functionality that already has been released and used. While the main effort of designing and writing test plans for this functionality was invested in past releases, every new feature developed requires a test plan and a maintenance effort on existing tests. One of the main goals for QA managers is reducing this effort to a minimum. Teams that succeed in creating clear and reusable tests simplify the maintenance struggle and optimize the test registration time.

    Eliminating Risk Through Automation

    However, for QA, the most cost-effective solution is test automation. At release end—the time when QA is at its most stressed and every commit might introduce a new risk—automation testing can eliminate risk and supplies early feedback on every change. Agile teams strive for the highest automation coverage percentage so the development team can have confidence in committing changes late in the release. That said, developers often aren’t interested in investing their time in automation. They support the QA automation effort, but they don’t want to “dirty their hands” by doing testing. Whether it’s UI automation—during which achieving element uniqueness is essential yet tricky and requires developers’ intervention—or API testing for APIs without proper documentation, QA faces intricate challenges that require sophisticated solutions, which developers are, in most teams, not part of. While the dev team focuses on the status of the new feature, QA attends to the application as a whole with extra attention to identify regressions created by a change.

    Join Forces for Higher Rate of Success

    Testing an application requires a completely different mindset than developing it. One of the main challenges in the Agile methodology is to create developer awareness of quality, involving other stakeholders in the quality assurance activities and making the QA process visible to everyone. QA managers must understand how day-to-day QA work is different than the work of other team members so they may truly represent the complexity, efforts and eventual success. R&D groups that join forces and understand that QA extends far beyond the actual tasks of the QA team have a much higher chance of improvement and set high standards for their product’s quality. Implementing the appropriate methodologies and utilizing the right tools lead to an increase in customer satisfaction and retention rate.

    About the Author / Ronit Eliav

    Ronit Eliav is the Director of Product Marketing at Panaya. With a deep interest in digital transformation and the latest IT trends, she enjoys creating thought leadership around such topics as agile and continuous delivery, DevOps, quality assurance and IT modernization projects. Connect with her on LinkedIn.

  • QA/Testing and DevOps: Partners, Not Adversaries

    QA/Testing and DevOps: Partners, Not Adversaries

    Software testing and QA is sometimes given low priority during development. The idea of treating software testers as adversaries by the developers seems old-fashioned and is short-sighted. It’s hard to believe that such an attitude can still exist, and yet it still persists, even in a modern DevOps team. But a DevOps-focused software team benefits when it considers QA/testing as an integral part of developing high-quality applications.

    In a business setting, the hidden costs of buggy or unsecure software are lost productivity, revenue and extra work by the development team (and even extra work by users, if the software isn’t running right). QA/testing helps test for system flaws, vulnerabilities and security issues, so why wouldn’t a DevOps team be supportive?

    QA/testing can help an enterprise avoid big mistakes. In fact, the DevOps Institute notes, “With the advent of DevOps and the movement to break down silos between developers and operations, it becomes critically important that all members of an IT team—regardless of what tools they use, or role they play—understand the essentials of testing.” And, according to Puppet, a standard for automating the delivery and operation of software, high-performing enterprises build quality testing into each stage of their software development process.

    Make QA/Testing a Priority

    Whether your DevOps-focused developers are handling the QA/testing themselves or passing it along to QA/testing specialists, it’s something that must be done. Ideally, developers themselves would handle QA/testing in an integrated DevOps environment, and that’s why some developers see it as an unwelcome chore. It’s not something they’re usually trained in, and they might struggle with it or see it as a burden. That could be the reason why some DevOps teams hit a “software testing bottleneck.” As data from a Vanson Bourne survey commissioned by Appvance shows, 73 percent of 200 enterprise IT decision makers in the United States have adopted DevOps processes, but 57 percent of those adoptees experienced a bottleneck during software testing that impacts how quickly new and updated software can be released.

    On the flip side, the developer’s company may have QA/testing professionals in-house, so the developers are hands-off in that area. Needless to say, the relationship between developer and the QA/testing pro could get a little rocky. Just think about it: The role of the QA/testing specialist is to find problems and point out issues with the developer’s code. Or, perhaps the developer or engineering staff is lagging behind schedule, so the team inevitably needs QA/testing to rush its processes, further creating a tense or adversarial relationship between the two groups.

    At one of my startups a while back, we worked for months on our first software release of a web application for an online marketplace. Late on a Wednesday night, we let the CEO of our company try it out—and the app immediately crashed. “Gee,” I said, “It was working on my computer.” Of course, it really wasn’t. We had synchronization problems, which are common in multi-threaded applications that access the same information for multiple users. And we made the mistake of letting the CEO be the first simultaneous user to try it out (creating a bad impression), instead of making organized QA/testing a priority at our company.

    QA/testing in a DevOps environment can help your team avoid mistakes, so passion for app functionality doesn’t supersede quality and robust bug-free software. In the end, what you need from your team is passion for both functionality and quality/testing. Otherwise, you’ll deliver a faulty product and lose customers or users—unnecessarily—because of a bad first impression by your software.

    Two Approaches for Integrating QA/Testing with DevOps

    When you’re integrating QA/testing into your DevOps environment, there are two different approaches or procedures to ponder. First is the more traditional test of the full system, sometimes called integration testing. The second is test driven development (TDD), which is popular in Agile and Scrum methodologies. The main difference between the two is in how software code is tested. During integration testing, developers review and test the code. Then, for critical facets of the software, a peer review of code is completed. Afterward, testers create scripts and deploy an array of integration and system tests. These could include scalability testing, load testing, stress testing and very importantly, security testing. Code vulnerability, SQL injections and malicious code would be things to think about.

    Under TDD, the order in which testing appears in the cycle is reversed. Each developer first writes “unit tests” and then writes the code. This is done iteratively in such a way that every discrete section of code is automatically tested. When new code is sent to the integration server, the tests go with it. The server itself runs unit tests for all the code that developers deposit in the server. On top of this, testers design system and integration tests to run automatically by the integration server. Finally, testers perform any manual tests that are impossible to automate. In this way, the integrity and quality of the code is constantly guaranteed.

    Academic studies and field experience suggest that TDD increases development time by approximately 15 percent … but hold on. Experience also shows that total cost of ownership of TDD-developed software is lower than software using integration testing only. It’s worth noting that by using TDD and integration testing, a company can deploy software of world-class quality that exhibits less than 0.3 errors per 1,000 lines of code during 18 months of use.

    QA/Testing: A Partner in Your Software’s Success

    If you’re hesitant about integrating QA/testing into your DevOps team and instead tempted to outsource it, you may want to push pause. It’s actually quite challenging if you do development in-house and expect to simply outsource what you think is independent QA with cheap testers. Outsourcing QA/testing and assuming that piece is done is not really outsourcing—it’s staff augmentation at a distance and it usually doesn’t work. You don’t want a “siloed team” to handle a critical component such as QA/testing. Instead, outsource QA/testing and bring your in-house and outsourced teams together around a common DevOps culture and mindset.

    DevOps experts Bob Aiello and Leslie Sachs, in an IBM report, say quality software “… requires not only a shift in behavior, but a dramatic shift in the organizational culture as well.” So whether you outsource or have an in-house squad, your developers and the entire team should take ownership of QA/testing, and remain open to new ways to test code efficiently.

    Ultimately, QA/testing should not be viewed as a time-consuming, disruptive chore by anyone on your team or in your enterprise. Think of it as a partner in your software’s success—not an adversary.

    About the Author / Steve Mezak

    Steve Mezak is founder and CEO of Accelerance, a leader in global software development outsourcing, as well as the author of “Software Without Borders,” co-author of, “Outsource or Else: How a VP of Software Saved His Company,” and a CIO columnist.

     

  • Moving from ‘Tester’ to ‘Quality Enabler’

    Moving from ‘Tester’ to ‘Quality Enabler’

    Speed versus quality. Can you really have one while the other exists?

    When Agile came on to the scene, it pushed to get development teams to create code as quickly as possible. Then DevOps came on as a practice that helped these Dev teams and the Ops side of the house to interact and collaborate more on best practices, tools to use and pinpointing issues in operations.

    But the testing practices and QA teams have had challenges keeping up with all this code that is getting thrown their way at an avalanche pace. That is probably why nearly two-thirds of self-proclaimed DevOps practitioners report that their biggest bottleneck occurred in the QA/Test phase. In fact, this cited bottleneck was more than 2 times greater than the No. 2 proclaimed bottleneck, planning (29 percent).

    It’s been said that “the consumer appetite to engage via digital channels is matched only by intolerance for any defects that might have an impact on their user experience and business outcomes.” If organizations are going to speed their delivery pipeline, they need to remember that the success of continuous delivery hinges on the effective employment of continuous testing.

    Part of the problem remains with manual testing practices. You can’t manually test millions of lines of code, then try to figure out the customer experience during integration testing with other internal or third-party systems and API, with hundreds of microservices or varying mobile devices.  The complexities of today’s composite apps make it impossible.

    And traditional, legacy testing tools don’t help, either. Gone are the days of waiting for the UI to be complete before QA teams begin their testing. And gone are the days of developers throwing code over the wall to the testing team and washing their hands of it.

    A few other facts:

    1. Ninety-one percent of QA professionals agree that the role of testing must change to meet the demands of today’s agile development needs.
    2. Only 1 in 3 organizations involve QA in planning requirements.
    3. Fewer than 1 in 5 organizations write new tests each time new code is checked in.

    The idea of continuous testing helps to solve that. Continuous testing is the practice of testing across every activity in the software development life cycle (SDLC) to uncover and fix unexpected behaviors as soon as they are injected. That can’t happen in the above environments.

    A new ebook from CA Technologies and devopsy.kinsta.cloud notes that to continuously deliver software, organizations need to make sure tests, procedures and processes around quality are designed to fit in line with the flow of development, integration and deployment. While continuous testing is dependent upon strategic automation, it is not synonymous with automated testing—a continuous testing process encompasses other important QA activities that impact overall product quality.

    As such, continuous testing depends on four major pillars:

    1. Code Quality – Did developers create the application code correctly?
    2. Pipeline Automation – Can the application code work across environments without manual intervention?
    3. Application Quality – Did developers create the correct features?
    4. Customer Experience – Are users perceiving value in the application delivered?

    These four pillars are discussed in more detail in this continuous testing ebook. You’ll learn the evolutionary journey you can take to shift testing practices left, and really make testing and quality everyone’s responsibility. You’ll even read how GM Financial was able to transform its load practices by implementing some of these practices. See how you can start using these pillars in your QA efforts.

    Now is the time to elevate QA’s new role from that of tester to quality enabler.

    — Scott Edwards

  • Q&A: Michael Beckley, CTO/CCO, Appian

    Q&A: Michael Beckley, CTO/CCO, Appian

    Recently I had the chance to speak with Michael Beckley, chief technical officer and chief customer officer at Appian, about his company’s move to go public and the DevOps space in general. For those of you who don’t know Michael, he is a founder of Appian and leads the company’s technology vision and oversees customer initiatives worldwide. He also serves on the Industrial Advisory Board for the University of Virginia’s Computer Science Department and is a member of the University of Maryland Advisory Board in Computing. He has an interesting take on the IT space and DevOps in particular. Read on for our discussion.

    Alan Shimel: Michael, what do you think has been the “special sauce” that has led to Appian’s success to date?

    Michael Beckley: With Appian, companies build unique business applications up to 20x faster and mobile-enable and modernize existing systems. For example, a company like Ryder was able to build a web and native Android tablet application that cut the time it took to rent a truck in half.

    But the “special sauce”? That’s Appian’s culture of innovation. Our engineers think long-term and invest in breakthrough tech like our patented SAIL UI technology for zero code native mobile apps. Thanks to SAIL, the web apps you configure in Appian automatically deploy to iOS and Android as native, full-featured apps or specialized micro apps we call “Sites.” As long as we get the culture right, we expect to continue to deliver enduring value for our clients.

    Shimel: With becoming a publicly traded entity, some companies have found “managing to the street” does not match their own long-term business plans and their customers needs, how does Appian avoid this?

    Beckley: Companies, like people, are an aggregation of their habits. Appian’s habit has always been to hold to our long-term vision of enabling every company to be a software company. We are not your typical Silicon Valley tech company. Our roots are in DC, we largely bootstrapped this business, and learned to be good stewards of capital while helping to support the core missions of federal agencies and some of the world’s largest brands. The proceeds following this IPO will be invested back into our business to benefit our customers.

    Shimel: What do you see post-IPO being the biggest areas for growth as it relates to new products and services?

    Beckley: We  plan to invest in innovating and enhancing our platform. While our platform is industry-agnostic, we have recently made, and plan to continue to make, investments to enhance the expertise of our sales and marketing organization within our key industry verticals of financial services, health care and government. We have made, and will continue to make, investments in research and development to strengthen our platform and expand the number of features available to our customers. As we continue to increase the functionality of our platform and further reduce the amount of developer skill that is required to build robust applications on our platform, we believe that we have the potential to expand the use of our platform.

    Shimel: Digital transformation is a term that is being used a lot these days. How far along the adoption curve are we on digital transformation?

    Beckley: While many speculate that digital transformation is just a tech industry buzz word, in reality, digital transformation is a living, breathing effort. According to a recent survey, digital transformation is a C-level priority and is considered a top focus for almost 50 percent of all companies this year.  From Appian’s perspective, we believe that the utilization of low-code software to drive digital transformation is still relatively new and our customers will be able to succeed in their digital transformation efforts by incorporating low-code development.

    Shimel: Where does the DevOps market intersect with the Appian business model?

    Beckley: Organizations are turning to DevOps to accelerate development and bring an application’s capabilities to a user quicker. The Appian low-code development platform fits into the DevOps model and supports the seven major DevOps categories including code, build, test, package, release, configure and monitor.

    Overall, our business model supports the DevOps market; we’re living in a world where citizen developers are playing such an important role in the enterprise and our platform empowers the citizen developer while alleviating developer resources to focus on the more challenging, business-critical developer work.

     Shimel: What about containerization and microservices? How do these technologies affect Appian?

    Beckley: Appian already uses containers and microservices. Appian’s enterprise customers often require a mix of cloud, hybrid cloud, and on-premise installations. We view containers and microservices as enabling technologies to support the diverse workloads and deployment models required by our enterprise clients.

    Shimel: What about security? What challenges does cybersec pose to Appian’s development of their platform?

    Beckley: Appian works with some of the nation’s most important and highly regulated industries, therefore we have built a comprehensive security and compliance structure that meets several industry standards. We frequently undergo third-party audits to ensure that we are always operating effectively to protect our customer data.

    — Alan Shimel

  • ShiftLeft & DevOps: A Narrative Within a Narrative?

    ShiftLeft & DevOps: A Narrative Within a Narrative?

    I procrastinated watching the season finale of “Westworld” just because I ended up reading a day later how Dolores ended the episode! I don’t know who to blame: my inquisitiveness or that guy who couldn’t hold himself. It kind of feels stale when you know the ending.

    Finally, I finished weeding through the multiple narratives across the episodes, and the flashback narrative merged seamlessly with the current one. That got me thinking about using that as an analogy and maybe understand whether the concept of ShiftLeft is really actually a “narrative within a narrative” of #DevOps.

    Ok, so what is ShiftLeft?

    I came across the term at my fav go-to source, @devopsy.kinsta.cloud, where Alan wrote on how much to Shift Left and its diminishing returns, and another very old article on the continuous testing approach, which primarily seeks to test early and often, to catch defects, yeah, early!

    Then it was thrown at me in the context of identifying repeatable activities, often manual with consistent results, performed by the Support & Operations team, and moving that work left from Level-3 all the way to L-0. Empowering the help desk, I would say!

    In both the scenarios, the underlying objective was to catch defects and or reduce life support activities early in the cycle (below) which costs more and more as we move up the food chain, which in turn makes a release and subsequent maintenance cheaper.

     

    As Alan also referred in the same article to the IBM Study, it’s almost 100 times more expensive to fix an eventual production defect which cost about six times more if caught at Implementation stage or about 15 times more if found during testing. Thus, “Early identification and prevention can improve reliable software delivery” would be the mantra.

    Now, to the crux of this article, Is ShiftLeft not what we aim to achieve via DevOps?

    DevOps basically creates a collaborative, non-siloed cultural change to bring both Development and Operations together early in the game.

    Throw in a bunch of tools to facilitate CI/CD, pull in the security guys early, expand automated testing coverage, include change and release—who are the catalysts, as I call these functions in my earlier article here—squeeze in a few monitoring tools and dashboards and voila! You have a complete and mostly successful DevOps team to dish out the most appetizing feast of quality software repeatedly.

    A table for two (Dev & Ops) has become a party of six! Yaaay!!

    So, from DevOps standpoint, ain’t we doing the same as ShiftLefting (If I may coin the word) with the resulting cost effectiveness, more efficiently?

    We do when we as a team make shorter and frequent sprints; gain insight, not just on how to code better or leaner but also by learning from the masters in Operations who run that software on the infrastructure they monitor and support daily. Or perform the VA scan early, or push a scenario as unit test instead of regression, and/or raise flags on potential capacity challenges.

    So again, isn’t DevOps team already doing that ShiftLefting as part of its charter?

    There is a limit to moving things left, but then, if your organization is still doing wireframes under a waterfall or is in the process of setting up a DevOps team, then this approach could be a stop-gap arrangement, an eventual merging of a narrative within a new narrative.

    But then my argument would be, Why stop-gap and why not a full frontal? Why not embrace the DevOps model, which comes preinstalled with all the ShiftLeft possibilities?

    Even if we are aware how an episode ends, procrastinating raises the risk of being left behind in the race to deliver quality, cost-effective product quickly.

    Technology is redefining businesses: Transportation is an app and soon probably will be driverless; banks are rebranding as tech companies in FinServ business; a fridge can reorder milk; trash cans can be Wi-Fi-enabled! Ah, the internet of things. Is Alexa listening?

    My other relevant posts are here and here.

    Thank you for the time.

    — Abdul Salaam Khurram

  • How Much QA is Enough in Software Development?

    How Much QA is Enough in Software Development?

    People working in quality assurance as testers, managers or engineers know what it’s like having to justify their existence; it’s been part of the job for many years. QA experts have had to fight the perception that what they do is a bottleneck, overkill and it slows everyone down.

    Finally, however, QA is getting the attention that it deserves. Companies are gradually investing more in QA skills and tools. This began with the onset of Agile and DevOps—methodologies which have now infiltrated mainstream software development practices everywhere. Since those practices focus on speed, efficiency and quality, engineering leaders have grown to understand that great products are not achievable without adequate QA checks and balances. Another risk is the unflinching nature of high user expectations: Thanks to social media, one user complaint can quickly escalate into others across the web, creating an uncomfortable perception problem for the software provider.

    Robust testing practices can prevent egregious flaws in the software that result in poor user experience and performance—or, worse, a breach. True, testing can sometimes get in the way of speed. Yet no matter how fast you release a product, if it doesn’t work well, no one’s going to buy it, much less use it.

    Nevertheless, we can continually improve the QA practice to fit into modern development teams’ workflow. Testers need to work with developers and master automated testing, while developers need to sometimes engage in testing; everyone should think about the appropriate level and amount of testing. You can have too much of a good thing.

    The Right QA Balance

    The below ideas can help your organization reduce conflict between QA aficionados and those who would rather code and release as fast as possible.

    1: Assess, analyze and report. Developers and testers alike are highly technical; they like numbers. Therefore, the more that the QA team can quantify the risk of an application, provide evidence as to the effectiveness of current QA practices and use metrics to show the need for more QA resources when warranted, the faster that everyone will get on board.

    There are plenty of tools that can deliver common quality metrics such as total number of defects, defect leakage, test coverage, test pass/fail rates, test completion rates, code coverage and application uptime. It’s important to share and discuss these metrics throughout the development team and with product and business managers. What constitutes high quality for the development team may not be the case with executives who are closer to the end user/customer.

    2: The gut check. After metrics, take into consideration the comfort level of your team and company concerning product flaws. In regulated spaces, or for applications that collect sensitive private data, the tolerance for bad code and security gaps is usually low. Gather feedback from different perspectives and roles in the organization for a balanced perspective on the QA practice and areas for improvement.

    3: Get people involved. Quality assurance is no longer a separate activity from development; especially when you are practicing Agile and DevOps, the responsibility for quality is shared. Therefore, developers, business analysts and operations managers all should be doing their work with an eye to quality, delivering feedback regularly to the QA team. Provide opportunities and incentives for any team member to attend events and training related to software quality.

    4: Streamline testing. Test automation tools are helping product teams be more comprehensive in testing, by automating low-risk test cases so that testers can focus their exploratory and manual methods on high-risk or new features. This is a positive development, although testers must take time to learn how to use the tools. Some other ways that can help teams be more efficient in testing include:

    • Conducting more testing early in the sprint. This avoids the 11th hour disaster of finding a major flaw right before the release deadline.
    • Per Agile methods, dividing development tasks into smaller chunks with team members from different disciplines to ensure testing is being done all the time and with visibility by all.
    • Using test driven development (TDD) or behavioral driven development (BDD), which requires that QA, developers and business stakeholders meet at the beginning of a project to agree upon requirements and test cases. Developers then write code against the test, until it passes. New tools are helping bridge the gap by writing requirements and user stories in plain English.
    • Outsourcing part of the testing load during a major project, which can help increase coverage and reduce stress on your team. However, understand that too much outsourcing will undermine any efforts to develop QA as an internal best practice and “mindset” for software teams.

    If your team is committed to QA as a best practice, not an afterthought, those at the top will need to invest properly in the right tools, training and people to get results. Misconceptions about QA still exist; contrary to the view of some DevOps experts, we can’t (and shouldn’t) automate every test. As well, even though developers are now testing the code they write, you still need experienced testers on your team. With the pace of innovation and the high stakes in the marketplace, organizations need more—not fewer—people engaged in testing.

    For testers who love their job, as many of them do, it’s an exciting time to be in the field. Companies increasingly are seeing the value of QA and testers continually are gaining more tools and tactics to excel at helping their organizations produce really great software.

    About the Author / Kyle McMeekin

    Kyle McMeekin is Senior Sales Engineer at QASymphony. Prior to QASymphony, he was a programmer analyst at Cognizant Technology Solutions. Connect with him on LinkedIn.

  • Getting the Most Out of QA in Complex Environments

    Getting the Most Out of QA in Complex Environments

    We live in a world in which software is increasingly dominant. Not only do we rely ever more on software, but its complexity is becoming even greater. And with increased complexity, bugs are more likely to be intermittent, making failures difficult to reproduce. Sometimes this non-determinism results in failures never getting fixed. Any unfixed bug is potentially a security breach or catastrophic customer outage waiting to happen. With such risks on the line, ensuring your software is of the highest quality is essential to running a successful operation.

    To cope with this complexity while still providing some assurance of delivered software quality, the DevOps industry has embraced a variety of automated testing regimes. A development team of a given size is able to run many more tests than it could before and must triage, and attempt to resolve, a correspondingly larger number of test failures. Diagnosing the cause of each test failure can still take a lot of time, particularly if the problem is intermittent or otherwise not easily reproducible. In fact, developers can spend on average as much as 50 percent of their programming time debugging, not coding. Think of all the innovation that could occur if that time were reallocated.

    Traditional QA

    When it comes to Quality Assurance (QA), developers and testers are all aware of the sequence of deliverables, beginning with development delivering an untested product to QA, which releases the tested and approved product to the end user. However, when a bug is found, QA and Development must communicate with one another to solve the issue quickly. The effectiveness of the delivery channel between QA and Development hinges on the quality of the information about the failure.

    Some common challenges raised by inefficient QA processes include:

    • Complex control flow – Difficult to make inferences about how a failure unfolded
    • Non-deterministic failures – Hard to reliably reproduce a failure to investigate the root cause
    • International R&D – Collaboration is made more difficult by geographic separation

    Conventional test systems inform that there is a bug but offer limited information about the cause. Workflows such as continuous integration (CI) often offer clues, such as informing which commit to the codebase caused the failure. However, there are several cases where CI cannot directly help identify problematic code changes. For example, the triggering code change may be entirely correct, yet cause a pre-existing bug to manifest. The developer cannot simply stare at the modification in the expectation that the source of the bug will eventually appear. Conversely, the failure may not easily reproduce for the developers even if they run the same test again. In short, CI is often very good at making the easier bugs very easy to fix, but typically is of little help with the more difficult bugs, particularly intermittent ones.

    The Next Evolution of QA

    To bridge the gap between identifying a failure in the QA process and resolution by development, DevOps teams have adopted a variety of technologies to complement their automated testing setups. New to this scene is technology capable of recording all or part of a program’s execution for subsequent replay and analysis, capturing test failures as they happen. The recording contains all the information a developer needs to understand the exact set of conditions that led to the test failure. It can be analyzed on different machines to the one on which the failure occurred, facilitating collaboration between colleagues who can work on separate machines to resolve the same issue—a particularly useful benefit for large software companies with geographically distributed teams. By gaining the ability to debug an exact replica of the failure, developers immediately reduce the time and effort required to diagnose and fix failures, therefore speeding the development cycle.

    Being able to generate recordings of failed tests, especially those intermittent issues which might not otherwise get fixed, coupled with the productivity gains of increased collaboration between development and QA allows companies to maintain a competitive advantage in a volatile marketplace.

    About the Author / Greg Law

    Greg is the co-founder and CEO of Undo. Greg has 20 years’ experience in the software industry and has held development and management roles at companies including the pioneering British computer firm Acorn, as well as fast-growing startups, NexWave and Solarflare. Greg left Solarflare in 2012 to lead Undo as CEO and has overseen the company as it transitioned from the shed in his back garden to a scalable award-winning business. Greg holds a PhD from City University, London, that was nominated for the 2001 British Computer Society Distinguished Dissertation Award. He lives in Cambridge, UK with his wife and two children.

  • The Tester: Saving Your Soul in a DevOps World

    The Tester: Saving Your Soul in a DevOps World

    Software testers have seen their careers go through some ups and downs in recent years. In the days of Waterfall, the tester’s job was clearly defined. Like other functions of software development and delivery, testers lived in a neat and tidy universe, testing features and design, finding bugs, sending back feedback to developers for correction and eventually, passing off code to the production people to work their magic. Testers had discrete roles with strong value propositions in the software organization.

    Then, Agile came along and new teams were created, lumping together developers, QA and Ops people for the sake of better collaboration and efficiencies. This began to create some confusion regarding who was responsible for what, when.

    DevOps brought order to the process, adding automation to every step, speeding cycles and reducing bottlenecks in a highly standardized, streamlined approach. In the continuous delivery pipeline, each step of the process creates its own contract for software quality—and that’s good.

    Development teams still value high quality, which customers demand. Yet in some DevOps circles, there’s been talk that testers aren’t needed much anymore. They bring bottlenecks to the process, forcing developers to work outside of the DevOps toolsets for code management and testing to respond to a defect ticket posted by the tester in an issue tracking tool.

    Eliminating that layer of process helps developers be as efficient as possible using continuous integration (CI) tools, proponents say. As well, some developers are more than comfortable handling testing, using test-driven development (TDD) methodologies.

    On the other hand, there’s faulty thinking here. Testers, whether deploying manual or automated testing methods, provide a valuable second set of technical eyes. Testers have a solid take on the user experience perspective, not just the fact that the code executes correctly.

    This human behavior and design insight is critical to the ultimate success of a software application. Finally, not all developers want to spend time testing nor have the skills to do so outside of unit testing.

    Moving forward into this new age of DevOps, testers play a vital role. The distribution of the tester into each phase of development (rather than at the end) is necessary to maintain both high standards of delivery and to keep communications fluid between different parties.

    Here’s how testers can contribute to DevOps teams:

    Testers need to be included in the continuous integration process

    Developers might be automating their unit test coverage, but they don’t replace all the test activities a tester can bring to the table. There are still other types of functional and non-functional test activities that must occur to validate the health of the product. Instead of transferring more testing responsibilities to the already overworked developer, testers can integrate their processes and systems with those of developers.

    Although there are version control tools (GitHub) and feature tracking systems (JIRA) to help developers, testers are often disconnected from these central tools. To connect with developer tools, testers can start creating and versioning their test cases within those development platforms. By doing so, testers can radiate feedback to the development team on issues to fix quickly through source code check-ins and in a way that won’t disrupt the normal flow of work.

    If testers can learn to use the DevOps tools developers love, much of the conflict goes away.

    Testers should increase their adoption of test automation, no matter their skill level

    Testers can learn newer test automation driven frameworks, especially those that help with transitioning manual test cases into automated scripts that can be reused, such as Cucumber or Gauge.

    For example, Gauge allows testers to write test specifications in Markdown, which provides a simple transition to automate your favorite programming language. These newer tools help testers transition to test automation despite not having experience writing tests this way—a common issue for testers working in DevOps.

    A growing number of test case management providers are starting to integrate their test design and execution around these automation frameworks to make the manual to automation transition smoother.

    Although developers might have automated unit test coverage, they struggle with developing functional and user acceptance automated tests. Testers can provide value by suggesting to the development team which are the best areas for test automation based on common issues and their unique knowledge of the application.

    They also can recommend areas for exploratory or manual test execution, depending on application complexity. This is especially helpful when teams starting down the path of DevOps do not have a regression test suite built that can be triggered through CI.

    Testers should focus on quality over finding defects

    Developers often complain that when testers find defects, it slows down their development processes. On a fast-moving Agile or DevOps team, filing bugs in a defect or issue tracking system is a total waste of time and creates unnecessary clutter.  Developers want instant feedback on code that’s broken and the related feature, rather than seeing another defect ticket they must open and close upon fixing.

    Therefore, testers should be giving feedback directly to the features that are being developed within a branch of code. They can do this through adopting tools that facilitate “ChatOps,” such as HipChat and Slack, which can integrate with version control systems. Testers can also provide direct feedback through a commenting system on the feature itself.

    With the advent of Agile and DevOps, which emphasize speed and frequent releases, testing is more important than ever in software development. By making some slight modifications to how they work and the tools they use, experienced testers can add considerable value to these teams by providing another layer of quality checks and a holistic view of an application’s performance and usability.

    About the Author / Ryan Yackel

    Ryan Yackel is Director of Product Marketing at QASymphony. He is responsible for all marketing activities around product releases, including development of strategic messaging. Ryan also helps to train the Sales teams on key value propositions for new products and features, and he provides customer feedback to the product team for roadmap initiatives. He can be found online at Twitter and LinkedIn.