devopsy.kinsta.cloud

Tag: continuous testing

  • SaaS: Building Tools for the DevOps-Minded

    SaaS: Building Tools for the DevOps-Minded

    DevOps, Continuous Delivery, Continuous Integration and Continuous Testing are no longer just methodologies and development strategies — they’re all a part of a growing economy within the tech industry.

    Great Software-as-a-Service (SaaS) companies such as Chef, Docker and New Relic have done an amazing job of spotting the trends early and building their tools and services around a new wave of development processes. Chef and Docker alone have reportedly raised nearly $250 million, and New Relic went public in 2014.

    In fact, this very website is another example of how impactful these new development practices are — as several specialized news outlets have been born out of these revolutionary development strategies. So, not only has the DevOps methodology helped developers create better products, faster; but the process has opened opportunities for technical experts to share their experiences on sites like this, e-books, webinars, conferences and any other way people communicate.

    On devopsy.kinsta.cloud, you’ll see the best articles on the web discussing how DevOps is defined in different organizations, why Fortune-500 companies are shifting old practices toward CD and the best practices of CI from industry experts.

    In the race to be the most DevOps-friendly solution — no matter the niche — hundreds of startups have been founded and funded, which has only enhanced the toolboxes of thousands of developers. You’d be hard-pressed to find a good SaaS company that doesn’t offer integration with Jenkins CI, TeamCity or Chef, and that’s because not only should a SaaS product solve an important problem, but it should have minimum disruption of your current development process.

    With all that in mind, Continuous Testing is one of the most exciting spaces in technology today. Continuous Testing is a process that aims to synchronize testing and QA with dev and ops, which ultimately yields better products more quickly.

    There are several factors that make Continuous Testing successful in a team’s software development lifecycle — such as open-source collaboration solutions, test automation APIs and tools to track the historic performance of applications .

    With the release of Load Impact Version 3.0, the global performance and load testing SaaS company has announced its latest feature that fits right in with the DevOps mindset: Performance trending.

    Performance trending allows users to plot runs of the same test over time. That gives companies a clear view into data that locates patterns of performance degradation or improvement in order to easily validate code and infrastructure changes.

    In the spirit of Continuous Testing, DevOps, CD and CI — performance trending improves users’ ability to gain both a low-level and high-level view into the performance of their systems. And with the increased number of tests run by DevOps professionals, performance trending yields valuable information for companies concerned with their application performance — which should be everyone.

    Between devopsy.kinsta.cloud and plenty of great conferences and meetups around the world, it’s safe to say this economy of DevOps-minded tools will only continue to grow in the coming years.

  • Continuous Delivery and The Proof of Quality Assurance

    Continuous Delivery and The Proof of Quality Assurance

    Quality Assurance is critical in Continuous Delivery. Every change we commit to the common source repository is a release candidate. That’s why we test our code before we release it. We find out whether objectives has been met or not. Unfortunately, testing is a discipline that many try to avoid and it’s one of the main reasons for having specialized testers in the teams who know how to break the code best. But shouldn’t be the developers of the code know best how to break it?

    (more…)

  • Continuous Testing: What exactly is it?

    Continuous Testing: What exactly is it?

    ***This post is based on themes and ideas from the great e-book, “Continuous Testing,” by our friends at Parasoft.

    The aptly named DevOps practice of Continuous Testing synchronizes Testing/QA with Dev and Ops processes optimized to achieve business and development goals.

    A few years ago when Agile was all the rage, lightning-fast Dev sprints left the Ops and QA personnel playing catch-up due to inadequate and sluggish testing practices with minimal coverage across frequent builds.

    The widespread culture of Agile development has accelerated development while software testing still lags behind, forcing organizations to cut corners in QA or slow down Dev processes entirely.

    Instead of speeding up the conveyor belt beyond control or slowing down development to kill efficiencies, Continuous Testing in DevOps encourages a systematic approach toward process improvement.

    Continuous Testing goes beyond automation and encompasses all practices — including tooling and cultural change — that help mitigate risks before progressing to subsequent Software Development Lifecycle stages.

    Continuous testing and Automation

    The approach to Continuous Testing can vary and follow diverse pathways to ensure the best user experience is delivered, free of defects.

    It’s barely viable to repeat all tests every time a new feature is added, so the Continuous Testing strategy fosters a company-wide cultural change to achieve four capabilities: Test early, test faster, test often and automate.

    End-to-end test automation practices are intended to integrate QA into existing fast-paced Dev and Ops processes as a means to create continuity while maintaining faster development cycles.

    It’s important to understand the role of automation in Continuous Testing. Automation alone does not enable continuity in testing, but it helps provide a qualitative assessment of risk and practice actionable tasks to mitigate these risks throughout the SDLC.

    In essence, automated testing constitutes the detection process for software issues and defect prevention, whereas Continuous Testing addresses the wider challenge of improving the effectiveness of these detection sensors.

    What does Continuous Testing Look Like

    This example demonstrates how collaboration, automation and tooling enable end-to-end testing that’s performed faster, more often and better aligned with Agile Dev and Ops practices.

    Continuous Testing starts within development processes as Devs use open source tools such as Selenium to test the functionality of their code.

    Tools such as GitHub or other internal repositories can then be used to store tests and version together with the software code. DevOps teams can use the same tests to integrate frequent builds of the software with the Load Impact API.

    Once the code reaches preproduction, dedicated QA teams can use these tests by modifying dynamic test parameters as appropriate. In the production environment, Ops can reuse these tests for acceptance testing and ongoing post production monitoring.

    Once the application goes live, integration with existing APM tools help keep tabs on server-side app metrics.

    Continuous Testing in this example is broken up into at least four environments: Development environment, Continuous Integration environment, QA Pre-production and Performance testing environment, and the Staging environment. Additional post-live monitoring further enable end-to-end continuous testing.

    Key Elements of Continuous Testing

    Parasoft, a global software solutions company, is one of the great thought leaders in DevOps and Continuous Testing. The company has laid out some key elements of the process in a fantastic e-book, and they’re definitely worth sharing here:

    Risk Assessment: Covers risk mitigation tasks, technical debt, quality assessment and test coverage optimization to ensure the build is ready to progress toward next SDLC stage.

    Policy Analysis: Ensure all processes align with the organization’s evolving business and compliance demands. Primary objectives include:

    • Identifying trends associated with injection of dangerous patterns within the code
    • Augmenting defect prevention practices for high risk areas
    • Isolating risk in target areas

    Requirements Traceability: Ensure true requirements are met and rework is not required. An object assessment to identify which requirements are at risk, working as expected or require further validation.

    Advanced Analysis: Using automation in areas such as static code analysis, change impact analysis and scope assessment/prioritization to prevent defects in the first place and accomplishing more within each iteration.

    Test Optimization: Ensure tests yield accurate outcomes and provide actionable findings. Aspects include Test Data Management, Test Optimization Management and Test Maintenance.

    Service Virtualization: Early, frequent and everywhere testing requires access to real-world testing environments. Service visualization enables access to the virtual form of the required testing stages, cutting the waste time to test environment setup and availability.

    An effective Continuous Testing strategy incorporates all of these elements and forces an organization-wide cultural change to synchronize Dev, Ops and QA/Testing as part of the true DevOps philosophy.

  • The Business Value of Continuous Testing

    The Business Value of Continuous Testing

    ***This post is based on themes and ideas from the great e-book, “Continuous Testing,” by our friends at Parasoft.

    Enterprise software development models have presented many innovative solutions for IT to align development efforts with business goals.

    Adopting new software development lifecycle methodologies is as much a business decision as an IT decision. From an IT perspective, this means the ability to produce profitable products at lower cost without compromising quality.

    Maintaining quality through the code changes to add features and push new releases has been a struggle until the inception of the DevOps-driven Continuous Testing model, which transforms costly, time-consuming IT practices into profit centers.

    The DevOps Patch to Agile Pitfall

    Having already adopted lean Dev and Ops strategies, Agile enterprise IT seems to have unlocked unprecedented efficiency and productivity by delivering better business value at a rapid pace.

    But in all the mayhem and chaos in jumping on the bandwagon of the Agile development, IT teams ended up stockpiling a mass of false assumptions.

    Simply implementing Agile processes and iteratively adding functionalities doesn’t necessarily streamline development.

    With this approach, teams ultimately ignored the true Agile philosophy of adapting, collaborating and planning, and ended up following an accelerated and reinvented version of Waterfall in hopes to develop fast, deliver early and manage issues effectively.

    A change was needed somewhere between design and development, and release and production phases where IT teams were failing to identify software defects and perform fixes fast enough.

    The lack of early, adequate, parallel and continuous testing was transforming IT into a cost center by drawing unnecessary expenses and time investments in reworking software to meet desired quality standards.

    The DevOps approach aims to address this challenge by facilitating continuous testing as an ongoing effort to identify and fix software defects early and deliver the right quality at the right time at the right cost.

    Continuous testing presents the following value propositions to raise business bottom line and open doors to unprecedented revenue opportunities:

    Informed Decision Making

    Business expectations about the reliability and speed of application delivery vary throughout the SDLC.

    Continuous Testing creates a centralized system identifying failures and defects that present business risk to the organization.

    Continuous Testing as a consistent effort provides adequate visibility and useful, real-time data. This can be a huge advantage to businesses, especially when it comes to application performance monitoring and testing.

    The assessment of stability and validity of software builds enables Ops and business decision-makers to trade off decisions between risk and time of release, cost investments, functionality, scope and quality.

    Business Agility and Competitive Differentiation

    Enterprises across all industries have development shops delivering software applications as core components of their service offerings.

    These departments, often driven by DevOps, must focus on churning out new, stable and high-quality software and race to deliver a stream of features that complement primary service offerings.

    For instance, mobile check deposit was a leading driver for bank selection a few years ago and financial institutions offering this online service (while doing it well) early gained valuable competitive differentiation.

    In the DevOps world, Continuous Testing creates a safety net for DevOps teams to develop, test and release new features to the market faster without compromising quality. This safety net works as a trusted testing environment, suite and tool to assess the impact of code changes accurately and quickly.

    With these capabilities, organizations gain the agility to bring new products to market and compete on innovation and not on the scale of business operations and investments.

    All in all, great DevOps levels the playing field between the eight-person startup and the massive enterprise competing in the same space. Who releases faster with a better product is more dependant on the process, not the funding.

    Stronger Business Performance

    IT performance is business performance. It’s measured in terms of the quality, productivity and functionality of IT solutions delivered for internal business operations or as marketable products.

    The primary objective of Continuous Testing is to ensure high quality software products are delivered on time, cost and scope. To achieve these goals, IT teams need capabilities such as automation-driven, self-testing builds in a clone of production environments.

    QA is expected to leverage frameworks and automation tools that allow testing early during the lifecycle. That shrinks time spent on test setup and maintenance, using service virtualization to replicate unavailable testing environments and establishing a comprehensive quality validation process with continuous integration systems.

    These practices work as core elements of Continuous Testing to test faster, sooner, more often and continuously, thereby boosting IT and business performance.

    These business values not only generate unprecedented new revenue streams but also unlock cash trapped within the business IT.

    The resulting increase in revenue encourages organizations to change their game entirely and pursue new DevOps-driven business models in improving service delivery, scaling operations immediately and competing on fronts of business innovation.

  • The continuous continuousness of DevOps

    The continuous continuousness of DevOps

    Defining DevOps is getting easier, but it’s still an elusive goal. Part of the challenge is that DevOps is more of a culture than a specific “thing” and how that culture exists is subjective and varies from one organization to the next. DevOps is what you want or need DevOps to be to some extent, but one thing seems to be consistent no matter how you define DevOps—it’s continuous.

    If DevOps is the buzzword du jour for tech, then continuous is the buzzword du jour of DevOps. Everything is continuous. Continuous development and continuous testing lead to continuous deployment and continuous delivery, which requires continuous support. Continuous monitoring produces continuous integration and continuous change. Continuous security results in continuous incident response…or vice versa. To top it off, all of the continuous activities continuously feed each other to drive more continuousness in some sort of DevOps Mobius strip of continuity.

    What does it all mean? I mean, aside from being a catchy buzzword and making that new DevOps tool or service sound sexy and impressive?

    In a nutshell it means there’s no end. You don’t really reach a point where everyone can shut down their PCs and grab a beer to celebrate the completion of the project. The project—whatever it is—is more of a living, evolving thing than a deliverable.

    There’s also no beginning—or at least no new beginning. All of the continuous activities cycle back over each other like a Spirograph drawing. Have you ever tried to find the beginning or end of a Spirograph drawing? Good luck.

    DevOps is the Alpha and the Omega, the beginning and the end, the first and the last.

    When you strip out the buzzwords what it really means is working in a more effective and efficient way. I’ve worked on software development projects the old-fashioned way. No, not Agile—Waterfall. The cascading tasks feed each other in what appears to be a logical progression. It makes sense to define requirements, design the software, implement it, test or verify it, and then kick back and go into maintenance mode…on paper at least.

    In the real world new requirements are discovered when you’re ready to implement, and testing results in new design elements. Those new requirements need to be designed and the new design element needs to be implemented and it all needs to be tested and maintained which inevitably creates new requirements, design elements, and so on.

    Waterfall was already DevOps—it was just dysfunctional because each “continuous” loop was treated as a unique and separate project. Items that were identified after the design phase were just accumulated on a list to be incorporated into the next phase or iteration. The net result was producing a somewhat stable product that you already knew was flawed, but that you’d get to fixing eventually.

    Agile development accelerated some of the feedback loops of Waterfall and resulted in faster refinements to projects, but it still had issues. The primary issue is that developers can’t go faster by themselves. Developers are just one cog in the machine and producing software faster just creates a bottleneck if operations isn’t prepared and the infrastructure isn’t ready to handle it.

    Chris Riley explained the challenge of Agile. “However, the Ops department was inherently left behind with deployments piling up faster than they could be released and customers never actually received the value they demanded. This trend ultimately gave rise to DevOps, a new-age version of Agile methodology encompassing the Dev and Ops segment to enable service delivery agility and quality.”

    Another way of looking at it is that DevOps is an evolution of Agile. I wrote recently, “DevOps is just a natural extension of Agile. It takes the guiding principles and best practices that Agile brought to developers and applies them on a much broader scale across the organization.”

    So now everything is continuous. There is no top of the waterfall that cascades through multiple layers until you reach the bottom of the waterfall. Now there’s a continuous culture of continuity. It may sound confusing when viewed through all of the various buzzwords, but when you step back it makes sense. It makes sense because that’s really the way it worked—or should have worked—all along.

    DevOps gets processes and policies out of the way to enable people to get things done more efficiently. For some reason we’ve decided that every element should be labeled continuous—which actually makes it sound a little frenetic or overwhelming. Maybe we should use “efficient” instead. That can be our new buzzword.

  • DevOps Makes Security Assurance Affordable

    DevOps Makes Security Assurance Affordable

    Despite the fact that information security is a top priority, many organizations face challenges integrating security assurance practices into their product and service development practices. All too often security problems are only detected and resolved with software patches following a security breach. Security assurance is often reactive but what is needed is a more consistent proactive and affordable approach to security. As indicated by Sean Michael Kerner in his article The DevOps model isn’t a threat to security; it’s a tool that can be used to enforce security like never before.

    DevOps with Continuous Testing (CT), when implemented according to best practices, is an opportunity for organizations to systematically and affordably integrate security assurance into product and service development. Using DevOps techniques, continuous security testing can be built into software change and deployment operations. Many refer to the application of DevOps practices to security assurance as “SecDevOps”.

    Below are some continuous testing best practices that make continuously testing security affordable:

    1) Develop expertise in SecDevOps CT: Security assurance demands that an organization develops expertise and processes to identify threats, protect against them, detect threats when they occur, respond with solutions and recover quickly. SecDevOps CT is not so different from any other DevOps CT except that it includes a security focus. Security assurance is really a special case of quality assurance and the practices for continuous testing also apply to security assurance. Other than the general continuous testing practices which I have described in my blogs on continuous testing, here are some specific skills needed for effective SecDevOps continuous testing:

    a. Include security experts in the DevOps CT team who are familiar with security domain knowledge including next generation network security protocols, understanding of BYOD security, cybersecurity attacks, in-application aware networks, streaming media and content delivery, virtualization and cloud security.

    b. Security experts should partner with and train the rest of the organization. This should be done from the beginning of the development process. This is important to ensure security is no longer an afterthought for an isolated department to worry about, but integrated at all stages of a project. This will ensure that developers and testers have a security mindset as part of their daily work.

    2) Leverage DevOps and continuous testing to make security assurance affordable: Because security assurance like quality assurance is considered a cost rather than a revenue generation activity, organizations are challenged to afford proactive security assurance. DevOps, when implemented in accordance to best practices, has a number of characteristics that make security assurance affordable as follows:
    a. DevOps infrastructures are configured as a common resource shared by the entire development, test and deployment teams and consolidate separate department test labs which allow budgets to be pooled.

    b. Orchestration tools include automated test topology re-configurations of products, test tools and interconnection components within physical, virtual and hybrid test lab deployments. This enables a wide range of security relevant production configurations to be stood up without expensive dedicated test equipment for each topology.

    3) Use Security-CT-Ready test tools which support security test protocols and flexible automation workflows: As indicated in my prior blogs not all test tools are CT ready and this is true for security tools also. Using the right security test tools that are CT-ready will make the job of integrating security into CT much easier. Here are key ingredients to look for in Security-CT-ready test tools:

    a. Use test tools with built-in security tests….no need to recreate them! These tools need to be compatible with security standards. The following whitepaper provides a framework of the impact –
    NIST Cyber Security Framework.

    b. Use test orchestrations tools that integrate well in the DevOps environments and can stand-up any production configurations consisting of a variety of platforms, mix of vendor components from the device level all the way to large scale network topologies. A good example is Spirent’s Velocity solution.

    c. Use static code analysis tools during continuous builds with built-in checkers that verify software security coding standards including CWE/SANS, OWASP and PCI DSS. For example Parasoft static analysis tools.

    4) Engage outside expertise to help accelerate your progress towards continuous security testing best practices: Many organizations make the mistake of trying to do everything themselves instead of employing expert consultants. Bringing in outside expertise can greatly improve the speed of accomplishing best practices. A good example of security expertise is Spirent’s Network Security Testing brochure.

    The above is a partial list of suggestions for continuously testing security that have been proven to yield good results for SecDevOps. At Spirent we think testing has a bright future in DevOps. You can read more about our views at Spirent.com/solutions/devops

    What do you think of these suggestions and do you have others that should be mentioned?

  • Continuous Testing System Stability

    Continuous Testing System Stability

    The effort to speed up continuous testing will be wasted if the underlying system that continuous testing depends on is not stable. In my prior blog
    Continuous Test Results Analysis – at the Speed of DevOps, I discussed the importance and practices for speeding up continuous testing results analysis. However, unless the continuous test system that the test results are derived from is rock solid, the results will at best be unreliable and at worst – results may not be generated at all!

    One of the first rules of DevOps is to “keep going and don’t stop”. Stopping is the equivalent of pushing the red button on a factory assembly line effectively causing the entire line to shut down. Then, panic ensues because every second that the line is down affects all of production leading to increasing costs not to mention restarting the line is substantial also.

    Many techniques are employed to minimize the need to stop testing despite reported product test failures. However if the underlying continuous testing system itself is not stable, then the entire DevOps system including continuous testing will immediately be interrupted until the system recovers. Development team members are looking for results, so the time savings for speeding up the availability and the total work associated with results analysis is huge for every minute of results analysis time saved on each CI/CT cycle. When aggregated over multiple code branches, a few minutes saved per test result can be man-years of time saved over a year!

    So what can be done to ensure the continuous test system itself is stable and produces reliable results even in the face of inevitable occasional system failures? After all, there is no such thing as a truly perpetual machine that never ever breaks down. Below are some suggestions in a checklist format that have proven useful in high performance DevOps deployments.

    1. Build a fast-response multi-discipline team: There are a wide variety of ways any system of interconnected computers and applications can fail whether they are virtualized, internal, or running on a cloud used remotely from a data center in the cloud. The following list of suggestions for the team will assure fast recovery when the system does experience failure.
      • Assign a continuous testing system architect with responsibility to design system stability SLAs, design high-reliability system architectures and define sufficient team workflows suitable for disaster prevention and recovery consistent with recovery time SLAs.
      • While the team members may be in-or-out sourced, full or part-time, there needs to be clearly responsible first responders and 2nd line escalation team members with definitive workflows that take care of failure events 24/7 for every hardware, software, tools, network and firewall component that the system depends on.
      • Assign product team leads to plan changes needed for new products and services.
      • 24/7 availability system designs are pricey to build, expand, and maintain, so finance and accounting team members need to be part of the team to keep clear, current cost data relevant to the continuous testing system.
      • A senior budget manager approves expenses needed for stability without delay.
    2. Anticipate failures – prepare, prepare, prepare!: Don’t wait for a failure before testing system recovery procedures! Documented disaster prevention and recovery plans or high-availability tools won’t save the continuous test environment if the team is not ready to react during a failure event.
      • Run failure simulations for each defined failure scenario at a frequency that matches the expected probability of each failure.
      • Cross-train team members on system recovery best practices so the skills will be ready when most needed – ensure no one person has essential unique recovery skills.
    3. Choose test tools designed with high stability system administration capabilities: Look for the following features designed into in test tools to ensure the continuous test environment can be configured and managed for high reliability.
      • Tools are implemented as autonomous services, controlled through RESTful APIs, that can be separately invoked and managed
      • Test system service monitoring is built-in including up-time, wait-time and usage metrics for each use case of each test system service.
      • Diagnostic capabilities are built-in to each service. When a failure occurs the failure log includes helpful descriptions including causal information.
    4. Engineer the system for stability: Typical continuous testing systems have many “moving parts” and many of them have to work in concert for the system to perform properly. It is common engineering knowledge that the reliability of a system is determined by the reliability and configuration of its parts. Here are some suggestions to improve the overall system reliability.
      • Choose powerful servers and virtual machines (VMs) with ample processor speed, fast I/O, large and fast memory and data storage. This is important to minimize the chance of system failures due to possible overloading of these resources.
      • Deploy multiple servers and VMs for each system component with enough redundancy so that no one component failure will interrupt the entire continuous testing system.
    5. Configure the test workflow to avoid false negatives: There is no point in reporting product test failures to users if the reason for the failure was an underlying continuous testing system failure instead of a failure of the product itself. Such “false negatives” are a big waste of time. Here are some suggestions to reduce the chance of false-negative test reports.
      • Assign an application to continuously verify, in-real time, that all the system components and interconnections are operating and report these results to the continuous testing report aggregation tool. These results are called CT system environment test results.
      • Use the CT system environment test results within the continuous test results analysis tool to determine if any test failure coincides with an environment failure. Mark each verdict “inconclusive” instead of “fail” if it does and include a description of the environment failure in the verdict log.
    6. Maintain the system to assure long-term stability: Every system needs maintenance at some point. Hardware components wear out and software components become obsolete. Each time a system component is replaced or upgraded to new versions there is a risk of instability. Here are some recommendations to reduce the risk of introducing instabilities to the continuous testing system when changes are implemented.
      • Create a maintenance environment that mirrors the test and production environments.
      • Define a test suite that will exercise all system components and topologies.
      • Test changes in the maintenance environment before deployment.
      • Occasionally run failure/recovery simulations in each production environment.

    The above is a partial list of suggestions for continuous test system stability that have been proven to yield good results for DevOps. At Spirent we think testing has a bright future in DevOps. You can read more about our views at Spirent.com/solutions/devops

    What do you think of these suggestions and do you have others that should be mentioned?

  • Test Results Analysis at the Speed of DevOps

    Test Results Analysis at the Speed of DevOps

    Fast continuous testing is counterproductive if the analysis of test results does not keep up with the speed of testing. In my prior blog Continuous Testing – Accelerated! I discussed the importance and practices for speeding up continuous testing. Unless test analysis is as fast as the tests, all that will happen is an accumulated debt of test results to be analyzed resulting in, at best, no net gain in time savings and, at worse, valuable results going overlooked and mass confusion that actually slows down the overall CI/CT cycles. Typically many development team members are looking for results, so the time savings for speeding up the availability and the total work associated with results analysis is huge for every minute of results analysis time saved on each CI/CT cycle. When aggregated over multiple code branches a few minutes saved per test result can be man-years of time saved over a year!

    So what can be done to ensure continuous test results analysis keeps pace with accelerated continuous testing? Below are some suggestions in a checklist format that have proven useful in high performance DevOps deployments.

    1. Determine clear responsibilities and SLAs for test results analysis:When the organization sets specific responsibilities and goals for results analysis the entire DevOps process will be accelerated.
      • Assign first responders and 2nd line escalation team members with definitive workflows.
      • Define a strategy for responding to test failures such as pre-determined actions if test failure rates exceed certain thresholds for each category of test. Examples of actions could include “continue with CI”, “revert changes”, or “stop and fix faults” depending on failure priorities.
      • Train the team on best practices test design and peer-review the scripts to ensure they define clear verdicts.

    2. Design test scripts for efficient test results analysis: When the test scripts are designed for efficient results analysis the process is accelerated. Here are some examples of good practices:
      • Each script must have a clear “Pass”, “Fail” or “Inconclusive” verdict, and a description of the verdict.
      • Each verdict should be reported together with pre-defined tags that can be used to categorize the verdicts.
      • Assign a priority code to each verdict. The highest priority verdicts should be analyzed first.
      • If a script has multiple verdicts, the highest priority verdicts should be reported first if possible.

    3. Choose test tools that are designed for fast test results analysis: Look for the following features when you are evaluating test tools. Not only will this save you a lot of headaches in the long term, it will give you the flexibility to design tests that integrate with other DevOps tools.
      • Restful APIs that do not require wait states between commands and responses.
      • Support a framework that enforces test results standards that ensure each test, when completed, reports a clear verdict including “Pass”, Fail”, or “Inconclusive” , a description of the verdict, event logs from both the test system and system-under-test POV, and tags that identify the build campaign, and functional attributes which can be used for cause analysis.
      • All timers can be sync’d to a common clock across the entire test environment and the verdicts record time at a precision that is appropriate for the fastest events relevant to the system under test.
      • Caching features that enable results data to be processed for one test while the tool proceeds to execute other tests.
      • Results data mapping and processing utilities integrated within the tool without requiring slow speed scripting to parse results.

    4. Choose and configure test analysis tools to process results as they are available:In a best practices large scale DevOps environment test cases are running in parallel across many machines simultaneously and there may be multiple code branches and versions being tested at the same time. According to best practices powerful test result analysis tools collect and aggregate test results data from all of the test machines, filter the results in real time for processing according to functional tags, analyze the aggregated data to determine probable causes and report the analyzed results together with probable cause information in priority order to the most relevant first responders, in real time “results-so-far” cumulative format.
      • Choose powerful servers for results aggregation and results processing, with ample processor speed, I/O, large and fast memory and data storage. Consider separate servers for aggregation and analysis to all result collection and processing threads to be run in parallel without interfering with each other.
      • Configure the system with fast high priority channels to collect all results data to a common results repository.
      • The results repository should be big enough to save all the results of many past test campaigns, such as a month of results data, because best practices test analysis tools will mine the past data to determine failure trends.
      • Results are collected across all test agents, processed and reported in real time as the verdicts are reported.
      • Results are reported in categories by build, tag and priority, and probable cause, giving highest priority in the reports to code areas or features that exceed pre-set failure thresholds
      • Probable causes are automatically determined from statistical trend analysis which weight the most recent build results heaviest and also use static code defect density information to point to code modules that most likely caused the failures.

    5. Configure the workflow for optimal test results analysis: When optimized, test analysis can prevent DevOps cycle interruptions significantly. The optimal workflow will use just-in-time concepts such as the following:
      • Only send results to the appropriate first responders when failure thresholds are exceeded.
      • Automatically escalate failures to the appropriate 2nd line escalation team members when the SLA time for first responders expires.
      • Analyze the highest priority verdicts first.
      • Prune or refactors tests that have a low failure history.
      • Define threshold levels that will cause an automatic revert of a change but make sure the team gets all the results for analysis and probable causes so they can fix the problem off-line before re-applying the corrected change.

    The above is a partial list of suggestions for continuous results analysis that have been proven to yield good results to complement accelerated continuous testing and DevOps.

    At Spirent we think testing has a bright future in DevOps. You can read more about our views at
    Spirent.com/solutions/devops

    What do you think of these suggestions and do you have others that should be mentioned?

  • Continuous Testing – Accelerated !

    Continuous Testing – Accelerated !

    Except for cases where test duration is the point of the test, as indicated in my prior blog “Isn’t Continuous Testing a Longevity Test Anti-pattern?”, faster continuous testing cycles are preferred because failing early and often is a DevOps goal. There are multiple reasons for this, but it should be evident that faster tests reduce the wait time to get results and since there are typically many development team members looking for results the developer time savings multiplier effect is huge for every minute of test time saved on each CI/CT cycle. When aggregated over multiple code branches a few minutes saved per test can be man-years of time saved over a year!

    So what can be done to accelerate continuous testing? Frankly a thorough answer to this topic could fill a book. Below are some suggestions in a checklist format intended to be useful for DevOps .

    1. Organize the team for speed: When team members and the organization are explicitly selected for CT speed the entire DevOps process will be accelerated.
      • A test architect familiar with the principles of best practices test environments and continuous test design is critical to design and tune continuous testing for speed. The architect establishes the CT strategy and speed goal relative to the coverage goal.
      • A test analysis team reviews test results so no time is wasted on results analysis.
      • Developers and test members are trained on best practices test design and their test scripts are peer-reviewed to ensure they comply with the strategy.

    2. Select powerful test resources: Powerful, dedicated, scalable resources will accelerate all test processes and shorten the time to results.
      • Select the fastest processors, low latency storage and I/O to initialize and execute both the test systems and system-under-test entities quickly.
      • Dedicate resources that have longer initialization times so they can be pre-initialized.
      • Virtualize resources that can be scaled horizontally without sacrificing speed.
      • If affordable, arrange enough resources to run ALL tests in parallel.

    3. Choose test tools that are designed for fast test execution: Look for the following features when you are evaluating test tools. Not only will this save you a lot of headaches in the long term, it will give you the flexibility to design tests that integrate with other DevOps tools.
      • Restful APIs that do not require wait states between commands and responses
      • Priority queuing of test cases
      • Creation of test topologies that can be pre-loaded
      • Ability to spawn out tests in parallel to run on any number of machines
      • The ability to park a test that needs to wait for an event and return to it after proceeding with subsequent tests
      • Caching features that enable initialization and results data to be processed for one test while the tool proceeds to execute other tests
      • Multiple devices, session types and results data processing utilities implemented as an embedded part of the tool and not requiring slow speed scripting
      • The ability to render commonly used test utility functions in a fast execution language.

    4. Organize test schedules and test suites to fail early: The sequence and approach for scheduling of test cases greatly affects the speed of testing. If possible the scheduler should launch all tests in parallel as soon as the build is ready. Otherwise priority schemes are needed.
      • Organize the test scheduler and test suites to avoid any unnecessary tests.
      • Cache initialization and topologies data and resources ahead of the next test execution.
      • Check the test environment before running tests, abort if invalid conditions are detected.
      • Organize tests so the most important ones are run first.
      • Set a failure escape threshold and abort the test if exceeded.
      • Eliminate dependencies between tests so that each one can run standalone or in parallel. Queue the next test while a test is running.
      • Use smart gray box testing methods to limit test selection to the most important tests according to the software change and skip tests that are low risk.
      • Run longer tests in parallel with shorter ones.

    5.Design test cases to fail early: The test cases themselves need to be designed with the strategy to get test results as fast as possible.
    • Design each test case with a single unique test purpose and make it self-contained so it may be scheduled to run standalone or in parallel with any other test case.
    • If a test case has multiple verdicts arrange the highest priority verdicts first and close the test at the first failure.
    • Prune or refactors tests that have a low failure history.
    • Shorten dead-script timers, and make any wait timers conditional so they can expire as soon as the condition is met.
    • Reduce the number of test iterations to fit a time budget.
    • Reduce communication wait times by using the fastest protocols. Optimize the most frequently used utilities.

    1. Arrange test results reduce to minimize analysis wait times: Arrange the test results to be consumable as soon as the tests are run so the system does not bottleneck.
      • Report test results as the tests are run and accumulate the results across parallel test targets.
      • Establish thresholds for aggregated test results and abort the tests and release test resources to be used by other tests if the thresholds are exceeded for any run.
      • Keep track of test timings and adjust the test schedules according the average and peak run times for each test case.
      • Separate test result report processes to persistent resources so no test process is waiting to finish a report and the test process can proceed to the next test in parallel with test results.
      • Correlate the test results to the changes and use trends analysis to help speed up diagnostics and as an input to test selection such that the most failing test cases are given higher weight in future test runs.

    The above is a partial list of suggestions for accelerating continuous testing that have been proven to yield good results for accelerating continuous testing and DevOps.

    At Spirent we think testing has a bright future in DevOps. You can read more about our views at Spirent.com/solutions/devops

    What do you think of these suggestions and do you have others that should be mentioned?

  • Isn’t Continuous Testing a Longevity Test Anti-pattern?

    Isn’t Continuous Testing a Longevity Test Anti-pattern?

    Accordingly to Jim Coplien: “an anti-pattern is something that looks like a good idea, but which backfires badly when applied. “ In DevOps context, continuous testing is an attractive solution for the narrow context of testing specific software components quickly (in cycles measured in minutes or hours) but may be “bad” if the system that the software is a component of has failure modes (E.g., effects of memory leaks) that will only be observed during long term testing over several days and if the time between releases is greater than that. In other words, the pattern of quick test cycles prescribed by continuous testing appears to be a contrary Anti-pattern to the fundamental concept of long duration “longevity” testing. So what are the well formulated positive patterns that are applicable in its stead?

    In my prior blog entitled “From QA to Continuous Testing” I indicated “QA test experts that are most skilled in the best practices of test design are valued because they will produce the best quality products.” QA test experts assert that it is unwise to release a software product that has not been tested for durations comparable to the lifecycle of a release. Only tests that are run for a significantly long time compared to the release cadence are deemed sufficient. Just how long is long enough? This is a subject for further discussion beyond the scope of this short article, but my own observation is that best practices test environments run accelerated level load and impulse load traffic patterns on a build candidate for at least 48 hours or more before a release is determined sufficiently tested for at least limited deployment. Even 48 hours is much longer than typical “continuous testing” cycles. So “what positive continuous longevity testing patterns are applicable in its stead?”

    Continuous Longevity Testing Patterns may sound like an oxymoron but in fact longevity testing is compatible within the context of continuous testing principles. Below I describe three Continuous Longevity Testing Patterns that are sometimes used.

    Pattern 1: Longevity testing time windows are reserved such as weekends. In this pattern the regular Continuous Integration/Test cycles are interrupted during the weekend and longevity tests are only run on the build that was last created prior to the weekend time window. This has the advantage of not requiring any special test resources other than the resources used for regular CT tests and does not require any sophisticated scheduling to orchestrate the test environment. The disadvantage is that the longevity test results are only available weekly, it is hard to diagnose which build caused a longevity failure and problems with release candidates will delay the release for a week until the next longevity test run validates the fixes.

    Pattern 2: Longevity parallel testing is scheduled frequently, perhaps as frequently as every few builds, or perhaps daily. This pattern has the advantages that the regular Continuous Integration/Test cycles are not interrupted any time (they run 24/7), the longevity test results are available for more builds than Pattern 1, which reduces diagnostic complexity and may reduce release delays, but has a disadvantage that separate test resources must be reserved for each longevity test in parallel with the regular CT tests. For complex systems under test with many test targets and lots of builds this can be very expensive so there is a tradeoff between test resource costs, and the cost of delays if not done frequently enough.

    Pattern 3: Intelligent longevity testing is scheduled only when certain conditions trigger the longevity tests to be started according to smart rules that correlate software changes to problems with longevity performance. This pattern has the same advantages as Pattern 2 and has the further advantage that the cost of test resources is usually much less than Pattern 2 if the rules for the triggers don’t fire too frequently. The disadvantage is the entire scheme depends on the creation of a rules engine and rules that understand the risk of changes for long duration testing. If the rules are too lenient then this pattern might allow too many longevity failures to escape but if the rules are too tight then this pattern is not any better than pattern 2 and yet more complex to implement.

    So which Continuous Longevity Testing Pattern is best? The best answer is the one that best matches organizations’ goals for release cadence, cost of testing, and risk.

    So indeed, Continuous Testing is a Longevity Test Anti-pattern but fortunately DevOps infrastructures implemented according to best practices have a variety of complementary Continuous Longevity Testing Patterns to choose from.

    At Spirent we think testing has a bright future in DevOps. You can read more about our views at
    Spirent.com/solutions/devops

    What do you think of these patterns or do you have others that should be mentioned?