Tag: continuous quality

  • Continuous Testing: The Quest for Quality at Speed

    Continuous Testing: The Quest for Quality at Speed

    The DevOps goal to reduce lead times by accelerating software deliveries presents a serious challenge for assuring the quality of the software that is delivered. As I indicated In my SlideShare post, “The Quest for Quality at Speed,” testing activities span end-to-end software value streams and often account for more than 50% of bottlenecks.

    The answer to these DevOps testing challenges is referred to as continuous testing (CT), which is more than simply automating tests. My blueprint for continuous testing identifies eight pillars of practices, each with strategies to reduce bottlenecks. In this article, I enumerate the strategies for each of the eight pillars of continuous testing and explain how to approach implementation for greater adoption of CT across an organization.

    Continuous Testing Pillars

    Continuous Testing Tenets

    There are five tenets which are guiding principles that enable continuous testing:
    • Shift Left: Conduct each test as early in the pipeline as possible.
    • Fail Early: Arrange the tests so the most likely problems are found during the earliest possible stage in the DevOps pipeline.
    • Fail Often: Run tests frequently and with many different conditions.
    • Test Fast: Arrange tests to run in quick cycles.
    • Be Relevant: Focus on the most important tests and results.

    Leadership and Culture

    Leadership and culture practices have a major impact on influencing behaviors and habits of everyone involved in testing across the DevOps value stream. Bottlenecks to CT include inadequate funding of testing due to lack of quality vision and goals at the executive level, and bottlenecks caused by distrust and lack of cooperation between Dev, QA and Ops. Executive sponsors need to clarify the definition of quality for the organization, and emphasize quality’s importance. Executives can communicate the vision and goals for quality and establish incentives that foster a culture of collaborative behavior.

    Test Strategies and Plans

    Test strategies and plans are needed to guide testing approaches for each product/service pipeline. Bottlenecks to CT include confusion caused by lack of product/service test strategies, and unplanned testing activities and resources are not included sufficiently in planning process. It is important to establish test coverage standards, test case standards, gate criterion and planning process that include testing requirements. CT strategies include making sure test coverage requirements are defined, all types of tests required are identified, allocate tests that are required at input gates, indicate test results are required at output gates and enforce that test tasks are included in planning backlogs and sprints.

    Test Management

    Test management is needed to coordinate testing resources and test schedules. Bottlenecks to CT include poor utilization of test resources due to lack of visibility and schedule bottlenecks due to lack of coordination of test schedules. It is important to have a database of tests and test resources, and test scheduling tools. CT strategies include version management of tests and resources, an inventory of test resources and dynamic test scheduling.

    Test Automation

    Designing, creating, selecting and executing tests in an automated fashion to get to a verdict as fast as possible is the primary strategy to accelerate testing. Bottlenecks to CT include slow test creation, slow test execution and slow time to realize verdicts. It is important to have effective automation strategies for test design, test creation, test selection, and test execution. CT strategies include test design templates, advanced test creation tools (E.g., TDD, keyword, behavior, model-based), dynamic test selection, tests designed for fast verdicts and horizontal and vertical scaling tests.

    Test Tools

    Test tools are designed to automate specific types of tests and integrate with DevOps platforms through APIs for orchestration, execution and reporting. Bottlenecks to CT include test tools that are not designed to scale, limited test configurations and lack of complete APIs, which make integration with DevOps platforms inefficient and error-prone. It is important to have scalable test tools with good APIs. CT strategies include limiting the selection of test tools to ones that are DevOps-ready or DevOps-native and that have well-defined, stable APIs.

    Test Environment Management

    Scheduling and setting up resources into the needed topologies for running different types of tests for developers and testers can be a major bottleneck if not managed efficiently. Bottlenecks to CT include lack of test resource scheduling, which causes conflicts for resources, manual test setups, which are time consuming, causing long wait times for testing and delayed retesting. It is important to have test resources scheduling, test environment orchestration and resource utilization management. CT strategies include test environment inventory tools, test environment sharing tools, test environment orchestration tools and monitoring test environment utilization.

    Test Results Analysis

    Reporting and analyzing test results can be a major bottleneck after test execution is completed. Bottlenecks to CT include poorly written tests which can report confusing results and that take a lot of time to analyze. It is important to have design standards for test results and test results reporting standards. CT strategies include test verdict design in test cases, common test reporting tools and test analysis tools.

    Steps To Implement Continuous Testing

    Start with one or two services and a small set of critical CT practices, then expand to more services and more advanced CT practices.

    A gap assessment is a great way to determine the current state of CT practices, and will help focus the implementation plan. Given a list of practices from each of the pillars in the continuous test blueprint, a cross-functional team of developers, testers and ops people can assign scores to measure importance and current level of practice to generate a gap score. The combined set of gap scores will give guide and direct current priorities for the CT plan.

    For example, beginning CT priorities could be to establish a strategy sponsor, define quality goals, obtain an inventory of test resources and requirements, put in place version management for tests and test standards, tool selection and test refactoring to match the standards.

    An intermediate CT goal could be to complete tool integrations with DevOps platforms, automate test scheduling, orchestrate test environments and optimize test results reporting.

    With maturity, advanced CT goals could be to put in place advanced test creation strategies, accelerated test execution, automated test environment management and optimize test results analysis.

    What This Means

    Testing activities are typically a major source of bottlenecks that slow the quest for quality at speed. Continuous test strategies that accelerate all aspects of testing can have a major impact on speeding up lead time for continuous delivery of quality software. Implementation strategies that start by applying specific continuous test strategies to a targeted small number of applications, and then expand to more advanced strategies applied to more applications is an effective approach to CT adoption across the organization.

  • Testing Heroes to Recognize Individual Contributions to Continuous Quality

    Testing Heroes to Recognize Individual Contributions to Continuous Quality

    Testers have always been great at championing the superstars within the testing community, but who else outside of testing has been listening? Tricentis has launched a campaign aimed at making sure the rest of the world knows just how invaluable testers are, especially in today’s hyper-accelerated release cycles, and with user experience and customer satisfaction becoming the enterprise’s success metrics of choice.

    Testing Heroes will be an annual program in which software testers can be nominated for a variety of award categories, two of which—Tester of the Year and Test Manager of the Year—will be awarded a free trip to Vienna, Austria, to accept their award onstage at Tricentis’ Accelerate 2018 conference.

    Testing Heroes is split into six different categories:

    • Tester of the Year
    • Test Manager of the Year
    • The Explorer
    • The Automator
    • The Performer
    • The Risk-o-Meter

    A Unique Twist

    While most winners are decided on by the awarding organization, the 2018 Testing Heroes will be determined by the level of support from each nominee’s surrounding organization and network. The final round of Testing Heroes will be a voting round. Those whose nomination makes it to this round will be provided with a variety of ideas for encouraging others to rally behind their candidate to get them the recognition they deserve. There are myriad ways to spread the word in your office about the outstanding work being done by one of your own:

    • Encouraging others in your company to vote through Slack, email or an internal newsletter.
    • Recording a short, campaign-style video to share through social media.
    • An iPad loaded to the candidate’s voting page placed in a break room, kitchen or other communal area.
    • Desk-to-desk canvassing (to those who don’t mind you popping in unannounced!).

    One of our initial goals for this campaign was to learn more about the amazing contributions by those in the software testing community, but we quickly realized there is a bigger reward here: When a nominee’s peers bring to light the amazing work being done by one of their own during the campaign process, software testers’ efforts hopefully will be more understood and appreciated by a larger audience than they are currently. This is the real goal of Testing Heroes.

    We encourage you to nominate and then rally behind your own Testing Heroes today!

    — Anand Akela

  • Quality Culture, DevOps Forge Better Connections Between IT, Lines of Business

    Quality Culture, DevOps Forge Better Connections Between IT, Lines of Business

    The automation of software development does not obviate the need for a collaboration between IT and the lines of business, or the development of a culture of quality on the part of the development team. Achieving that quality culture requires more than a change in mindset as developers seek to align quality with the speed and automation goals of DevOps.

    There are five items required for a quality-driven DevOps environment:

    1. Understanding the proper role of “minimum viable product”
    2. Collaboration between IT and lines of business
    3. Having metrics in place
    4. Across-the-board buy-in

    “Quality culture is characterized by employees who are motivated to consistently strive for the highest level of quality,” said Al Riles, principal consultant at Tunnell Consulting, a firm providing solutions to the complex problems faced by life sciences and biopharma companies. “That’s one key component of it, as well as the ongoing process of continuous improvement.”

    Minimum Viable Product and Continuous Quality

    Aligning quality with the speed and automation goals of DevOps is critical to success, and part of this is gaining a deeper understanding of the role of “minimum viable product” and understanding when that approach contributes to the development process.

    “The concept of ‘minimum viable product’ does not, as its name would suggest, contribute to a substandard product,” said C.R. Venkatesh, CEO of Dot Com Infoway, a CMMI Level 3 IT company. “Quite the contrary, the concept is the basis of continuous and iterative improvement, and encourages the use of user feedback, which often results in insights that might be missed by the in-house team.”

    The concept of quality culture and MVP are not mutually exclusive in a DevOps environment, so long as continuous improvement is part of the equation.

    Collaboration Between IT and Lines of Business

    The collaboration that is a key part of DevOps is also a key part of quality culture. “You have a culture in place where everyone embraces that fabric that includes the right type of behaviors and the right values, including communication, reward and recognition, as well as clarity on roles and responsibilities,” said Riles. “That working fabric gets well-defined, because everyone is functioning around the same key elements that constitute that strong corporate culture.”

    “Collaboration has always been important in a DevOps environment,” said Venkatesh. “But we need to define precisely where that collaboration exists. In an internal environment, that collaboration is between IT and lines of business. Increasingly though, IT and development is an outsourced function, and in such cases, that collaboration must be preserved. What this means is that it exists not just within the confines of the outsourcing provider, but also between the provider and the client. The developer must seek out alliances not only with the client’s sourcing manager, but also with other internal stakeholders in the project.”

    If You Don’t Measure, it Doesn’t Exist

    Quality initiatives require an active role in changing the mindset of all stakeholders, but mindset alone won’t get the job done. Metrics must be in place and tied to those quality initiatives. “Even in the smallest of development projects, defined quality metrics must be set ahead of time, with clearly defined outcomes,” said Venkatesh. “By asking what we hope to accomplish with a development project, clearly defining the goals and adhering to a set of predefined quality standards, developers—whether in-house or third-party—are able to accurately predict what the outcome will be.”

    Riles’ process involves a quality management maturity grid, which identifies multiple dashboards. “It allows the organization to become familiar with what a quality organization consists of, and then you can use this maturity grid to measure where the organization is once you embrace this initiative. Then, you can implement the initiative, and come back and re-evaluate yourself to see the progress you’ve made,” said Riles.

    Everybody Has a Seat at the Table

    Incentivizing all stakeholders, throughout every level of the company, is paramount and, according to Riles, “It has to do with communicating what the vision is, and having a clear quality message that’s coming from leadership. Then, there has to be a shared quality mindset among all staff.” Riles notes that the quality mindset is shared by everyone in terms of understanding the organization’s objectives, policies and procedures and their roles in hoping to achieve them.

    This is reinforced by leadership at all levels, who are actively engaged in supporting the development of a quality culture, and they engage and motivate others to do the same. For it to be sustainable, we realize that success is based on the sum of all parts of the organization. They must be continuously monitored and fine-tuned as the organization grows, and this includes communication, reward and recognition, and engagement.

    About the Author / Dan Blacharski

    Dan Blacharski is a thought leader, advisor, industry observer and PR counsel to several Internet startups, and author of the book “Born in the Cloud Marketing: Transformative Strategies for the Next Generation of Cloud-Based Businesses.” He has been widely published on diverse subjects relating to cloud computing and technology, and he is editor of BoozeHub.com. He lives in South Bend, Indiana with his wife Charoenkwan and their Boston Terrier, Ling Ba.

  • Redefining the state of quality with mobile apps

    Redefining the state of quality with mobile apps

    While many organizations are looking to agile methods to deliver mobile apps faster, they are also facing challenges in embracing the methodology across the organization. The keystone of agile for mobile apps is the focus on “the state of quality,” or the need to know if your code works, so will your app. The major problem that so many organizations are facing is this: DevOps teams are doing agile for mobile halfway. They perform nightly builds but no one is aware of the state of quality.

    What’s limiting visibility into the state of quality? There are three factors:

    1. First, unstable or limited test automation capabilities impacts the number of non-unit tests that can be run. Test automation capabilities are often similar to a slice of Swiss cheese – it looks good but there are definitely holes.
    2. Secondly, unstable development environments cause for workarounds, such as non-scalable scripting or skipping over major factors that impact a test environment, such as weak connections to wifi. Stabilizing a development environment is so crucial to the rest of the agile mobile app development lifecycle, while workarounds are not a sustainable solution.
    3. Finally, out of cycle tests are impacting the quality Many times, organizations keep test sections, like load testing, out of the lifecycle because of complexity. That means there’s never true insight into the state of quality during the mobile app lifecycle.

    To figure out how to know where the quality stands on any given day, organizations need to take a “no code left behind” type process. Continually embedding quality into the mobile app lifecycle will assure working code and working apps.

    Here are steps to achieve this:

    • Cut out the Swiss cheese: Combat limited test automation by implementing a wide and stable automation that meets requirements. Set those requirements before selecting an automation engine:
      1. Do you need to interact with the device or just the app?
      2. Do you need to analyze what is on the screen?
      3. Do you need self-recovery in case the system crashes?
    • Staff Accordingly: Devices are unstable by definition. Humans are more stable. Organizations can make an environment stable by staffing it accordingly, making someone available to handle testing.
    • Leverage the hybrid cloud: Create an always-on test lab with a person to take care of it. This reduces traps like USB connectivity issues, breach of security guidelines, Mac support, etc. Staffing an outlier to manage a test lab WILL break your cycle.
    • Bring it In-Cycle: Out-of-cycle test sections will also break your cycle. Build an environment that supports a variety of test cases and brings the outliers in such as different network conditions, multiple devices connected at the same time, capacity issues and the ability to always ready for load or security testing.
    • Educate the organization: In embracing Continuous Quality, educate your staff on the benefits and power of using automated testing in the cloud and the results that they will see experience with their own work.

    An environment that’s open and integrated and offers the least amount of limitations possible will allow DevOps to generate creativity from a comfortable place and give visibility into the state of quality. Continuous Quality is the only way to ensure your apps are working well and are released faster.

    About the Author/Yoram Mizrachi

    Yoram-Mizrachi-ws-newYoram Mizrachi is the CTO of Perfecto Mobile. Yoram brings to Perfecto Mobile a wealth of experience in networking, security and mobile telecommunications. Yoram founded Perfecto Mobile after serving as the CTO of Comverse Mobile Data Division. In this capacity, Yoram handled a variety of technological aspects in mobile applications, WAP and location-based services.  In 1999, Yoram was the CTO (and founder) of Exalink, which was later acquired by Comverse for $550 million. Prior to founding Exalink, Yoram held several technology-related positions in the fields of communication and cryptography.