devopsy.kinsta.cloud

Tag: software agility

  • Best of 2021 – Best Practices for Application Performance Testing

    As we close out 2021, we at devopsy.kinsta.cloud wanted to highlight the most popular articles of the year. Following is the eighteenth in our series of the Best of 2021.

    When done properly, software application performance testing determines if a system meets certain acceptable criteria for both responsiveness and robustness. Before you jump in to testing, though, there are some best practices to remember.

    Start by defining test plans that include load testing, stress testing, endurance testing, availability testing, configuration testing and isolation testing. Align these plans with precise metrics in terms of goals, acceptable measurements, thresholds and a plan to overcome performance issues for the best results. Make sure you can triage performance issues in your testing environment; you should analyze issues impacting application performance by examining system functionality under load, not just the indicators of poor performance on the load testing tool side. Leveraging application performance management (APM) tools, which simulate production environments, provide much deeper insights into application functionality, as well as into overall performance under stress or load.

    10 Performance Testing Best Practices

    1. Test Early and Often

    Performance testing is often an afterthought, performed in haste late in the development cycle, or only in response to user complaints. Instead, you should be proactive. Take an agile approach that uses iterative testing throughout the entire development life cycle. Specifically, provide the ability to run performance “unit” testing as part of the development process – and then repeat the same tests on a larger scale in later stages of application readiness. Use performance testing tools as part of an automated pass/fail pipeline, where code that passes moves through the pipeline and code that fails is returned to a developer.

    2. Consider Users, Not Just Servers

    Performance tests often focus solely on the performance of servers and clusters running software. Don’t forget that people use software, and performance tests also should measure the human element. For instance, measuring the performance of clustered servers may return satisfactory results, but users on a single, troubled server may experience an unsatisfactory result. Tests should take user experience into account, and user interface timing should also be captured along with server metrics.

    3. Understand Performance Test Definitions

    It’s crucial to have a common definition for the types of performance tests that should be executed against your applications, such as:

    • Single User Tests. Testing with one active user yields the best possible performance, and response times can be used for baseline measurements.
    • Load Tests. Understand the behavior of the system under average load, including the expected number of concurrent users performing a specific number of transactions within an average hour.
    • Peak Load Tests. Understand system behavior under the heaviest anticipated usage for concurrent number of users and transaction rates.
    • Endurance (Soak) Tests. Determine the longevity of components, and whether the system can sustain average to peak load over a predefined duration. Monitor memory utilization to detect potential leaks.
    • Stress Tests. Understand the upper limits of capacity within the system by purposely pushing it to its breaking point.
    • High Availability Tests. Validate how the system behaves during a failure condition while under load. There are many operational use cases that should be included, such as seamless failover of network equipment or rolling server restarts.

    5. Build a Complete Performance Model

    Measuring your application’s performance includes understanding your system’s capacity. This includes planning what the steady state will be in terms of concurrent users, simultaneous requests, average user sessions and server utilization during peak periods of the day. Additionally, you should define performance goals, such as maximum response times, system scalability, user satisfaction scores, acceptable performance metrics and maximum capacity for all of these metrics.

    6. Define Baselines for Important System Functions

    In most cases, QA systems performance don’t match production systems performance. Having baseline performance measurements for each system can give you reasonable goals for each testing environment. These baselines provide an important starting point for response time goals, especially if there are no previous metrics, without having to guess or base them on the performance of other applications.

    7. Perform Modular and System Performance Tests

    Modern applications are incorporate many individual, complex systems, including databases, application servers, web services, legacy systems and so on. All of these systems need to be performance tested individually and together. This helps expose weaknesses, highlight interdependencies and understand which systems you should isolate for further performance tuning.

    8. Measure Averages, but Include Outliers

    When testing performance, you need to know average response time, but this measurement can be misleading by itself. Be sure to include other metrics, such as 90th percentile or standard deviation, to get a better view of system performance.

    KPIs can be measured by looking at average and standard deviations. For example, set a performance goal for the average response time plus one standard deviation beyond it (see Figure 1). In many systems, this improved measurement affects the pass/fail criteria of the test, matching the actual user experience more accurately. Transactions with a high standard deviation can be tuned to reduce system response time variability and improve overall user experience.

    application

    9. Consistently Report and Analyze the Results

    Performance test design and execution are important, but test reports are, too. Reports communicate the results of your application’s behavior to everyone in your organization, especially project owners and developers. Analyzing and reporting results consistently also helps to determine future updates and fixes. Remember to consider your audience and tailor reports to each audience. Reports for developers should differ from reports sent to project owners, managers, corporate executives and even customers, if applicable.

    10. Triage Performance Issues

    Providing the results of performance tests is fine, but those results, especially when they demonstrate failure, are not enough. The next step should be to triage the code/application and system performance, and involve all parties: developers, testers and operations personnel involved. Application Monitoring Tools can provide clarity regarding the effectiveness of triage.

    Additionally, remember to avoid throwing your software “over the wall” to a separate testing organization, and ensure your QA platforms match production as closely as possible. As with any profession, your efforts are only as good as the tools you use. Be sure to include a mix of manual and automated testing across all systems.

     

  • Leadership Role in Adding Test Automation to Existing Environments

    Leadership Role in Adding Test Automation to Existing Environments

    Leadership is the foundation for all agility. I’ve written elsewhere extensively on a model that works well for the new environment in which we find ourselves: the need for tens or hundreds of very smart people, doing things that have never been done before with new tools, working with people they may not be familiar with.  These creative, expert people make dozens of decisions a day that few others see in real-time, turning ideas into code that must run flawlessly and execute complex business processes. Quite a leadership challenge.

    The leadership model I’ve articulated is simple. Leaders at all levels should focus on rigor (good decisions based on options and data), alignment (everyone rowing in same direction) and efficiency (respecting everyone’s valuable time). Getting the right people, with the right skills and the right kinds of interactions, is more valuable than having the right process and tools. 

    For software agility, we know reducing the friction between development and productive use is a powerful leverage point. This general area we now call DevOps. Coming from the application space as I do, my primary focus has been on test automation. How can we at very low cost continually ensure that our evolving software is ready to be deployed to production (or production staging, if we are doing periodic bundled software deploys)?

    I’ve recently watched a team introduce test automation into an existing application environment. Great leadership was demonstrated throughout by multiple team members. Let’s examine the steps taken and point out the leadership that made it happen.  

    Set the Target

    Team and organizational leaders laid out a clear vision of the business benefits and their support for the initiative. This was the culmination of several months of learning, communication and cost/benefit analysis. Leaders gained alignment based on rigorous analysis.

    Ensure Adequate Leadership and Technical Skills Are in Place

    In the run up to the inception of the work, leaders hired an experienced testing leader, who in turn hired a couple engineers to do the work. Because the environment was a common one in the industry (Salesforce was a big part of it), the leaders sought the best expertise they could find, and wound up engaging an external specialist firm to get them going. People over process. 

    Define the Technical Architecture

    The new team leveraged the knowledge of the consultants and some of their engineering partners to examine options and select an approach. They shared the options and analysis with their team and leaders and gained consensus and understanding. In the process the approach was fine-tuned. The team demonstrated rigor and efficiency and gained alignment. One key of their success was the “towering technical expertise,” a lean product development term, brought by the consultants.

    Deliver Incrementally

    For this item, I will describe the solution chosen, since the specifics will be of interest to this audience. The vision was to train the product owner/systems analysts to write acceptance criteria for each user story (which was the unit of development here) in the stylized language Gherkin. As the development was done during a sprint period, the test team would take the Gherkin and write automated tests for the test tool Cucumber, which would drive execution in Selenium. The execution would be automatically connected to the deployment of new code using Jenkins.

    The team wanted to avoid disrupting the existing flow until the risk was reduced based on proven results. Accordingly, they began by building automation for existing software, proving that the vision worked in practice, and gradually completed the regression suite and started to catch up to work in process that hadn’t been released. Excellent leadership for agility was demonstrated again, driving rigor through proof of the concept and laying the groundwork to get the rest of the team aligned.

    Get the Rest of the Team to Change

    Once the test team caught up to being able to do in-sprint automation, they needed to tackle the training and acceptance of a new way of writing acceptance tests (in Gherkin). The great results they demonstrated helped influence the product owners and systems analysts to give it a try.

    Continuously Improve

    As the creative period of designing the test automation approach and getting into production came to an end, a new period of measurement and continuous improvement began. Here we have another foundational element of leadership for agility: understanding where we have repeatable flows where process really is the key, and where we have a creative endeavor in which the focus on people and interactions must rule. In a sense, the team transitioned from a product development effort to a manufacturing-like flow, and they accordingly changed their approach.

    Conclusion

    Sometimes we make the mistake of thinking that getting to efficient DevOps is all about tools and processes. My experience is that to the contrary, leadership is the key. As the first value of the 2001 Manifesto for Agile Software Development states, “People and interactions over process and tools.”

    — Michael K. Levine