devopsy.kinsta.cloud

Author: Mush Honda

  • How Continuous Testing Advances Your Business Goals

    How Continuous Testing Advances Your Business Goals

    The goals of any successful business fit into two broad categories: driving higher revenue and greater operational efficiency. Finding solutions to achieve both of these is an organization’s goldmine.

    Considering the ease of access to new, alternate products from developers across the globe, competition continues to grow. In the software industry, customer expectations magnify cut-throat competition. Users expect newer features, faster releases and ever-improving optimal performance.

    To achieve these core business goals, DevOps teams now develop software with a mindset of continuous delivery. This practice helps organizations achieve faster release cycles while delivering newer features to meet customer demand.

    For most teams, there remains a major ongoing challenge: effective testing within this high-speed delivery cycle. Traditional (non-automated) testing methods became a bottleneck in the CI/CD pipeline mindset, forcing organizations to choose between quality and speed to market.

    Continuous testing helps overcome this problem by weaving testing throughout the delivery cycle by leveraging automation. But what tests should be automated?

    In late 2018, KMS surveyed 135 professionals who successfully implemented continuous testing, for their insights on the process. Of the organizations surveyed, the following tests were always automated:

    • Unit tests (47.4%)
    • Component tests (46.7%)
    • Integration tests (45.9%)
    • UI tests (38.5%)
    • Performance and load tests (57%)
    • Security tests (52.6%)
    • User acceptance tests (34.1%)

    We also found the top three focus areas of test coverage for continuous testing were:

    • Integration tests (55.6%)
    • Security tests (48.1%)
    • UI tests (43.7%)

    As a bottom line, businesses must successfully adopt continuous testing to keep their delivery pipelines in sync with development.

    Despite several benefits arising from continuous testing, KMS found a few challenges for organizations that have adopted this methodology. Seventy-eight percent reported that continuous testing would have been easier to adopt with a directive from their executive leadership.

    What’s more, 78% indicated they would have liked more team ownership of the initiative and 86% would have liked better team collaboration.

    So, how do organizations get executive buy-in and rally the team to adopt the practice? By understanding and communicating how continuous testing directly contributes to achieving the two key business goals of increased revenue and operational efficiency.

    Let’s drill down into increased revenue generation. After all, money is what keeps any business in motion. Businesses drive revenue by:

    1. Solving a pain point for customers.
    2. Encouraging users to adopt their solution.
    3. Having the agility to shift and meet changing consumer needs, with speed.

    To achieve operational efficiencies, businesses focus on:

    1. Achieving shorter release cycles.
    2. Receiving feedback and ability to respond with agility.
    3. A more effective team, without increasing headcount.

    With these goals in mind, there are three core business cases for adopting continuous testing:

    Increased Agility and Confidence in Quality

    Agile is the ideal delivery model for all software companies. True agility means organizations can shift focus quickly and respond rapidly to customer feedback or changes in the market. It also gives teams the freedom to explore new opportunities/solutions. Continuous testing enables this agility by supporting a shift-left philosophy that uses test automation in the earlier stages of the development cycle. Shifting left enables a continuous feedback loop that allows teams to respond to defects and establish better team communication.

    By adopting continuous testing, 80% of professionals found fewer defects escaping to production. Reacting to feedback quickly allows companies to give customers the products they want. An organization can increase revenue by delivering market-disrupting features.

    Automation tools also encourage a testing mindset in developers—80.7% of companies adopting continuous testing have found their developers doing more testing. This mindset shift is critical to achieving agility with better collaboration. It makes the quality of the software the responsibility of the entire team, truly aligning with the Agile Manifesto. As a result, product quality and user experience benefit, and happy customers increase revenue.

    Faster Overall Speed

    Keeping up in today’s market requires shorter release cycles. As I mentioned earlier, adopting continuous testing enables faster sprints by eliminating the testing bottleneck at the end of the delivery life cycle. In fact, 83.7% of companies have achieved shorter release cycles by adopting continuous testing.

    Speeding up the software development life cycle pushes products to market faster. Getting features in the hands of customers before your competition gives your company a huge competitive edge. By adopting continuous testing, companies are not slowed down by antiquated testing initiatives. Speed enables them to compete on a global scale and disrupt the market.

    Lower Operational Expenses

    How do you balance speed and cost? Automation. Continuous testing automates routine tests. By using open source tools such as Katalon, teams can automate with minimal extra expense. This not only helps you test faster, but your team can be more productive—84.4% of companies have achieved better test coverage with the same team size and 67.4% were able to test with smaller teams. Either way, you get more done without investing in additional overhead.

    However, automation is not here to replace testers. Automation in pre-production environments gives your resources the bandwidth to work more strategically: 53.3% of companies that have implemented continuous testing don’t test manually in pre-production environments. Seventy-four percent now have the bandwidth for exploratory testing in production. Use your team and resources where they are most effective, doing the work automation can’t. Ultimately, smart use of resources achieves a sought-after balance of speed and cost.

    By exploring the business cases for continuous testing, companies achieve true agility. The methodology seamlessly aligns with fundamental business goals. Enabling continuous delivery, continuous testing will support revenue growth and operational success for your organization.

    — Mush Honda

  • The Art of Software Performance Testing

    The Art of Software Performance Testing

    When users turn away from applications or have critical feedback, it’s often because the application is slow, hangs/freezes or, worse, it isn’t available when needed. In this era of limited—or, zero—patience for app instability, delivery teams should ensure that their performance testing capability is sound. First though, let’s start with definitions. Despite popular opinion, performance testing is not the same as load testing. Load testing checks how systems function under a heavy number of concurrent users performing transactions over a certain period of time, while performance testing is much broader in that it checks how the system behaves and performs. Performance testing examines responsiveness, stability, scalability, reliability, speed and resource usage of your software and infrastructure.

    Performance testing doesn’t always get the attention it deserves. Testers have a tendency to focus on features and component functionality, even when verifying performance on application architectures designed leveraging APIs. As software development methodologies and application infrastructure evolve, so should performance testing. We observe that in Agile teams, performance testing typically executes at least one sprint behind other testing methods. Performance testers don’t start testing until the application’s features are deemed stable. They operate separately from the delivery team. Yet, performance testers should run tests as soon as new code is available, which in turn helps developers receive real-time feedback on issues that they then can fix immediately.

    To ensure a successful strategy in validating the performance of an application, here’s a look at three fundamental performance test categories:

    • Endurance: This involves running a transaction or scenario continuously for a period of time to detect memory leaks or anything else that causes a system to slow down.
    • Breakpoint: The goal here is to increase the load on a system until it reaches a threshold, after which it crashes. These tests are critical to understand at what point the system or underlying components will buckle.
    • Scalability: By increasing the number of concurrent users, instances or database size, you can measure how applications will handle growth.

    Performance Testing: Recommended Tools and Skills

    Performance testing is difficult without automated tools that can simulate scenarios and user activity so that you can test comprehensively. Many good tools are free or available as open source. Some of the popular ones include JMeter, Grinder and Locust.io.

    To be successful, performance testers need a DevOps mindset. Technical skills are a must, since understanding software code behaviors and interpretation of hardware and software utilization is necessary. An understanding of how to interpret the technical concerns observed from the business perspective is also a great skill to build. Tests should always answer a “what if” scenario and occur within the sprint. Hence, the ability to apply a continuous testing approach as part of the holistic testing strategy is a great skill to learn! Finally, learn how to provide real-time feedback to developers, as a form of hypothesis about changing code to support better performance tracking.

    Guidelines to Make Your Performance Testing a Success 

    These steps will  help structure your performance testing to achieve results for the business success of the delivered application:

    1. Map your tools and environment. You’ve got to understand the environment infrastructure that will host the application under test. The ability to quickly configure, build and tear down the environment to test is a crucial time-saver.
    2. Set acceptance criteria. Getting concise goals of application performance among all stakeholders will chart your path and allow you to gauge success. An example metric is page load times for your mobile application.
    3. Define KPIs. The acceptance criteria will dictate the KPI goals for performance tests. These may include percentile response times, throughputs, garbage collection performance, heap utilization and resource utilization.
    4. Set up test runs, execute and monitor. After completing the preparatory steps above, you can define test scenarios and create the test cases. Don’t forget to incorporate monitoring. This entails creating alerts in the testing environment to ensure that monitoring services are working as expected.
    5. Analyze, repair and assess. After each performance test cycle has completed you’ll want to assess the results against your acceptance criteria and fix any problems immediately. Over time, acceptance criteria will change and so should your test scenarios and anticipated outcomes.

    As with all testing activities, early involvement and collaboration with developers, analysts and business stakeholders will save time and headaches later. The more you know about the technology stack and upcoming application changes, anticipated maturity of the application and evolving business goals, the better prepared you can be for any testing need as it arises.

    In the evolving shift to a modern software development environment, performance testing has become central to building confidence in software. It’s time we give it our full attention.

    — Mush Honda

  • As Business Speeds Up, Testers Must Adapt

    As Business Speeds Up, Testers Must Adapt

    It should be no surprise to anyone working in software today that delivery speed is the name of the game. Irrespective of business sectors or B2B/B2C focus, global competition in application development and delivery has flooded the market. Due to the increased number of options in the marketplace, users expect an excellent experience from your application or site—and if you cannot meet expectations, they’ll find it elsewhere.

    A company’s ability to be highly responsive to (if not proactive about) customer’s needs is also valuable for adoption and branding. Established companies understand that delivering high quality in their products and services is core to brand value.

    This shift isn’t always easy for testers. Moving to a faster pace of work can be unsettling when your job is all about assessing the perceived quality of the software. Testers like to be cautious and check everything twice. The current work environment has made testers more germane than ever in software and IT organizations.

    Testers help build confidence in the quality of an application, and this perception of quality is what typically drives user adoption. If two applications have more or less the same functionality but one is faster, more reliable and has a better UI and design, you can guess which one will be more popular.

    Setting the Foundation

    To thrive in this new world, testers must learn to prioritize and focus on the most important needs of the team’s delivery goals versus testing every last mouse click. Testers will need to quickly incorporate feedback from multiple places, and work well with stakeholders across the development organization and the business. Efficiency is imperative, and that means increasing the use of tools whenever possible.  The following framework to develop a world-class testing organization when your company is moving faster all the time can make the transition a little easier.

    1. Understand business goals. While this may sound obvious, testers haven’t always had to think about the bigger picture. With more competition from everywhere, everyone on the development team should have a solid understanding of customer needs, business goals such as to drive new revenue or grow customer loyalty, and how the competition stacks up for the solution in development.
    2. Define a testing strategy. This exercise begins with understanding the business goals as they relate to the product or project and, at a more granular level, the goals for each sprint. The testing strategy should include a description of the overall project, priorities for testing mapped out into phases, timelines, testing approach, tools and methods to use and how progress will be measured.
    3. Develop a risk management plan. What will happen if those business goals are not achieved? For instance, the business stakeholder might say the project must go live in four months with new functionality. But is this an arbitrary goal or one that is embedded in an actual customer commitment? What will be the financial impact if the deadline slips? Which enhancements are critical and which are nice to have from the user perspective? Once those particulars are documented (or discussed), the testing team can create the appropriate plan, prioritize and determine what resources are needed to execute it.
    4. Resources. Testing teams with mandates to move faster will need support in the form of new tools, training and even new hires. Software that testers (as well as others) use can include DevOps and CI/CD tools, ALM tools, reporting and analytics, UI and API testing tools, systems for recording and logging manual testing, tools for usability feedback and more. This can get overwhelming for startups and other companies that don’t have the budget to purchase new software. Fortunately, there are many credible open source tools that can fit the bill. This reality is also why development teams must understand well enough what customers really want so they can prioritize on which features must be perfect and which areas of the application could meet a lower bar of quality.

    A Word of Caution

    Testers are well-equipped to make all the changes required for the faster pace of modern software development. This shift has been underway for a few years now, and many testers are embracing the change and the elevation of their role on engineering and IT teams. However, as testing relies on more automated tools, there are a few things to consider:

    1. Avoid tools mania. Testers, like many technical people, love to try out the newest toys, but it’s important to justify a tool’s adoption before acquiring it. With too many tools, there’s just too much noise and complexity. That will counteract any speed and efficiency gains that automation supplies.
    2. Automation tools do not replace testing. It’s not necessary nor helpful to automate every testing task. Testers should view automation tools as enablers, not a replacement for the function. Automation is wonderful for reducing effort and giving testers time to work on value-added activities that can boost the overall user experience and usefulness of an application. Be sure to baseline test objectives before recommending an automation solution.
    3. Institute process around automation. Ensure you have defined a proper process in sharing artifacts that are produced by your automation tools. Artifacts such as scripts must be versioned with proper comments and stored in a central repository instead of on individuals’ machines. Defining and enforcing good development practices such as check-in/check-out process, scripting style guides and object organization with automation will ensure a higher return on effort in the long run.

    In a few years, this point may be moot. But there are still many testers coming into new roles from the waterfall mindset who may struggle in the transition.

    Testers once lived simply in the silo of sluggish procedures. They knew exactly when they would be called upon for action and how they would complete a task. Now, though, testers must put on their business assurance hats and work side by side with developers, operations managers and others early in the process. They will have to learn new ways to test, such as with behavior-driven development and test-driven development. They need to get more technical, given the push toward test automation.

    What hasn’t changed though is the tester’s quest for quality in software. Great testers will add the most value when they can be proactive about alerting others to brewing problems before they present themselves to the user.

    — Mush Honda