devopsy.kinsta.cloud

Tag: lead time

How Test-Driven Methodologies Reduce CI/CD Lead Time
Modern development environments, in which rapid continuous delivery is facilitated by automated continuous integration/continuous development (CI/CD) pipelines, require thorough and automated testing in development prior to integration. This article explains how test-driven development (TDD) and acceptance test-driven development (ATDD) methodologies reduce CI/CD lead times.

Lack of test automation in development impacts CI/CD lead time for subsequent integration, delivery and deployment stages of a CI/CD pipeline in the following ways:
- Increased defect rate: Without automated testing during the development stage, more defects and issues are likely to be introduced into the codebase, increasing the time spent on debugging and fixing issues in later stages.
- Slower feedback loop: Without test automation, the feedback loop between code changes and test results is slower, making it more challenging to identify and resolve issues quickly. This delay can impact the overall lead time in the CI/CD pipeline.
- Reduced confidence in releases: When testing is not performed adequately or isn’t automated, there is less confidence in the quality of the software being released. This may lead to more manual testing and validation, further increasing lead times.
- Inefficient use of resources: Without test automation, developers and testers need to spend more time on manual testing, reducing the time available for developing new features or fixing existing issues.
- Increased merge conflicts: When test automation is not performed early in the development stage, the risk of merge conflicts increases as different developers work on the same codebase. Resolving merge conflicts can be time-consuming and may cause delays in the integration, delivery, and deployment stages.
- Decreased stability and reliability: Inadequate test automation during the development stage can lead to less stable and reliable software. This may require additional time and effort to address issues and ensure that the software is production-ready.
- Inconsistent test environments: Manual testing may not always be conducted in a consistent environment, leading to potential discrepancies between development, staging, and production environments. This can cause unexpected issues and delays in the pipeline.
- Hindered collaboration: Lack of test automation may reduce visibility into the quality of the codebase, making it more difficult for team members from development, QA, security and operations to collaborate effectively and identify issues early.
Although CI/CD pipelines help catch issues and ensure code quality through automated builds and tests, having a strong foundation of test automation during the development stage helps minimize the potential for issues in the pipeline and ensures a smoother CI/CD process.

There are several reasons why many development teams do not automate tests prior to integration, despite the benefits of doing so.
- Lack of awareness or understanding: Some development teams may not be aware of the benefits of test automation or may not fully understand how to implement it effectively in their development process.
- Insufficient skills or experience: Test automation often requires specialized skills and knowledge, and some development teams may lack the necessary expertise to implement it effectively.
- Time constraints: Developing and maintaining automated tests can be time-consuming. Teams working under tight deadlines might prioritize feature development over creating automated tests, believing that it will save time in the short term.
- Limited resources: Test automation tools and infrastructure can require significant investments in terms of cost and resources. Some organizations may not have the necessary budget or resources to invest in test automation, especially for smaller projects or teams.
- Resistance to change: Some teams may be resistant to adopting new practices, particularly if they have been successful with their existing processes. Changing established workflows and learning new tools can be challenging and may encounter resistance from team members.
- Incomplete or rapidly changing requirements: In situations where requirements are unclear or frequently changing, development teams may struggle to create meaningful automated tests, as they may require continuous updates to stay relevant.
- Overemphasis on manual testing: Some teams may place a greater emphasis on manual testing due to a belief that it provides more accurate or thorough results. This perception can lead to a reluctance to invest time and effort into developing automated tests.
- Fear of increased maintenance: Automated tests, like any other code, need maintenance and updates as the application evolves. Some teams may be concerned about the additional effort required to maintain a test suite, particularly if they have limited resources.
The amount of test automation that should be done in development prior to the integration stage of a CI/CD pipeline depends on several factors, including the complexity of the application, team resources and the specific project requirements. However, here are some general guidelines to help you determine the right balance:
- Unit tests: Developers should aim to automate most, if not all, unit tests during the development stage. They are typically fast to execute and provide quick feedback, allowing developers to catch issues early and minimize the number of defects that make it to the integration stage.
- Component and integration tests: While unit tests focus on individual components, component and integration tests verify the interaction between different components or services. These tests should be automated as much as possible to ensure that any issues arising from component interactions are identified and resolved quickly.
- Static code analysis: Automating static code analysis checks during development helps identify code quality issues, security vulnerabilities, and potential performance bottlenecks early in the process.
- Code coverage: Strive for a high level of code coverage with your automated tests. However, it’s important to remember that code coverage is just one indicator of test quality and should be complemented by other testing strategies.
Striking the right balance between automated and manual testing is important. Automate tests that are repetitive, time-consuming or prone to human error. Manual testing should be reserved for exploratory testing, usability testing and other areas where human intuition and judgment are necessary.

Test-driven development (TDD) and acceptance test-driven development (ATDD) are development methodologies that emphasize creating automated tests before writing the actual code, helping teams ensure code quality and catch issues early prior to integration.

Test-driven development (TDD) is a software development methodology that emphasizes writing tests before writing the actual code. The primary goal of TDD is to create clean, maintainable and bug-free code by catching issues early in the development process. TDD follows a short iterative cycle consisting of three main steps: Write a test, make the test pass and refactor the code. The developer starts by writing a test for a specific feature or function, which initially fails. Next, the developer writes the minimal amount of code necessary to make the test pass. Finally, the developer refactors the code to improve its structure and readability while ensuring that all tests still pass.

Acceptance test-driven development (ATDD) is an extension of TDD that focuses on defining and validating the high-level requirements and acceptance criteria for a feature or functionality before development begins. ATDD involves collaboration among developers, testers, and business stakeholders to ensure that the application meets the needs of its users. The process begins with writing acceptance tests, which are derived from the user’s perspective and based on the acceptance criteria. These tests are written in a human-readable language to facilitate clear communication among all team members.

Once the acceptance tests are defined, the development team follows the TDD process to implement the feature, using the acceptance tests as a guide. As the team develops the code, they also create unit tests and other lower-level tests to validate the implementation. The acceptance tests are executed as part of the development process and serve as a measure of progress. When all acceptance tests pass, it indicates that the feature or functionality meets the defined acceptance criteria, and the development task is considered complete. By using ATDD, teams can improve collaboration, ensure that the application meets user expectations and reduce the risk of miscommunication or misunderstanding requirements.

Hurdles to overcome when transitioning to TDD and ATDD include:
- Mindset shift: Adopting TDD and ATDD requires a change in mindset, where testing is considered an integral part of the development process rather than an afterthought.
- Skill development: TDD and ATDD require specialized skills in writing effective tests and implementing test automation. Development teams may need to invest time and resources in acquiring these skills.
- Time and resource allocation: Transitioning to TDD and ATDD might initially lead to a perceived increase in development time, as writing tests before code can seem slower. However, the long-term benefits of improved code quality and reduced debugging time should be considered.
- Collaboration and communication: ATDD relies on effective collaboration and communication among developers, testers, security, operations and business stakeholders. Establishing a culture of open communication and collaboration is crucial.
- Tooling and infrastructure: Implementing TDD and ATDD may require investments in new tools or changes to the existing development infrastructure to support test automation and continuous integration.
In the context of TDD and ATDD, the roles of quality assurance (QA), security and operations teams are crucial for ensuring a well-rounded and robust software development process. Their involvement helps to address various aspects of the application, leading to a more reliable and secure product.
- Quality Assurance (QA): QA team members collaborate closely with developers in both TDD and ATDD methodologies. They can help define test cases, create test plans and provide input on acceptance criteria. In ATDD, they play a particularly significant role in developing and validating acceptance tests, ensuring that the application meets the defined requirements and user expectations. QA team members may also be involved in setting up and maintaining test automation infrastructure, analyzing test results and identifying areas for improvement.
- Security: Security professionals contribute to TDD and ATDD by identifying potential security vulnerabilities and ensuring that the application is developed with security best practices in mind. They can help define security-related test cases and help to embed security into the application, reducing the risk of vulnerabilities and improving the overall security posture.
- Operations: Operations teams are responsible for managing the infrastructure and deployment processes that support the application. In TDD and ATDD, they can collaborate with developers to ensure that the application is designed and developed with operational concerns in mind, such as scalability, performance and maintainability. Their involvement also helps to promote a DevOps culture where development and operations teams work together to ensure the application’s success in production environments.
A practical roadmap for teams to transition to TDD and ATDD while overcoming the hurdles mentioned earlier could look like this:

Build awareness and buy-in:
Provide training and skill development:
Start small and iterate:
Foster a culture of collaboration:
Set up the necessary tooling and infrastructure:
Monitor progress with metrics:
Continuously improve and adapt:
Summary

TDD and ATDD methodologies contribute to reducing CI/CD lead times by automating tests during development. This improves code quality, minimizes defects and promotes collaboration among team members from the outset of the development process. By writing tests before the actual code and focusing on meeting acceptance criteria, teams can catch and resolve issues early, leading to fewer delays and reduced time spent on debugging and fixing errors during the integration, delivery and deployment stages. These practices also help ensure that the application aligns with user expectations and operational requirements, resulting in a smoother and more efficient CI/CD pipeline.

Anyone who is passionate about modern quality engineering processes for building high-quality software to participate is encouraged to join new ShiftSync Community from Tricentis to engage in discussions about this topic and other development methodologies.
May 19, 2023
Practical Advice on How to Use Lead Time
Many ALM tools and Value Stream Management platforms graph lead time, but few really tell you what to do about it, how to use it and why you shouldn’t use the average. In fact, so much of what is written on this subject is rather vague and hand wavy. Part of the reason is that Value Stream Management is a big job, and it isn’t something a tool can provide. Rather, it’s a human endeavor. Another driver for the hand waving may be that the writers really don’t know much about lead time because they’ve never actually used it.

Improving the value stream is a big topic but there are definitely things we can cover in this space: how to use lead time, how to get it, prerequisites to using it well, techniques or approaches and references for further study. I’ll start with the simplest, most mechanical topic, and move to the harder topics last.

What Is Lead Time?

Lead time is how long it takes a work item to get through a system, such as your IT development value stream. If your system is under control, recently measured lead times can be a good predictor for lead time in the future. People want to know when stuff will be done so they can make plans.

Lead time is also a lagging indicator for many of the actions used to improve, well, lead time and flow. It’s what some people call an output metric—you make a bunch of improvements to the system and improved lead time is what you get out of your efforts.

Where to Start

Measure lead time separately for the few types of work where it really matters, such as production defects and epics or initiatives. For every work item, record the start and end dates. But first you have to define what you consider the start and end. If you are using the measure to communicate with a customer, use dates from the customer’s perspective. If you are trying to improve the development part of the pipeline, post planning, then start the clock when it’s scheduled for dev. In that way, some people distinguish flow time from lead time. Cycle time is sometimes used for an even smaller slice of the value stream, usually just one step—I’m not going to debate here whether that’s a proper or official use of that term.

Whatever you do, for heaven’s sake, don’t use the average lead time. Instead, use a percentile. I use the 80th percentile. For illustration, the 25th percentile is the point at which 25% of the observations fall below that point. The 80th percentile of your lead time observations (measures) is the point at which 80% of your historical lead time observations fall below that point. The 50th percentile is not always the median, but for the sake of this article we can say it’s close.

Average, however, could be materially off. It’s worse to use the average than the median because the average can be thrown off by outliers more than the median and 80th percentile. If you are good enough with statistics to correctly identify and remove outliers, that’s great, but few people do that at all (much less with statistical precision) and it’s really not necessary for most IT work.

The only thing I’d use the median for is to monitor the trend—to see if the median is improving. But using it for forecasting or expectation setting would be bad. You see, when using median, 50% of observations took less time, but 50% took longer. You wouldn’t want to tell a customer that he has a 50/50 chance of getting a fix in two weeks. Telling them they have an 80% probability of getting a fix in three weeks, in my experience, is more palatable.

You want to be able to tell your customers (or marketing, management, support or program management) that 80% of the time you resolve this kind of issue in “n” weeks. Most people are happy with those odds. Anything higher takes in so much of the long tail of the distribution that it makes forecasts not terribly useful for planning.

How to Get the Graph

Use the lead time graph in your existing tooling if you can; get the data into a data warehouse and use a BI tool if you must; or use Excel only as a proof of concept or if your organization is small. More on this below. Compute your 80th percentile lead time expectation every sprint or month, and graph it in a bar chart so you have a new bar every period.

How to Get the Data

It’s easy to find the 80th percentile in your dataset. If your tooling can give you your choice of percentiles for lead time, that’s fantastic. If not, you’ll need to gather all of your lead time observations. You can do it in Excel, but then you’ll be forever updating Excel with fresh data or re-importing the data. That’s workable if you have a small number of teams and observations to input each sprint. Otherwise, you’ll want to get it into a database and make a BI dashboard.

How Much Data to Get

You don’t need a whole lot of data.

“Data beyond 24 weeks is most likely out of date,” said agile data and probabilistic forecasting guru, Troy Magennis.

How to Compute It

In Excel, use the percentile.inc function. Generally, order your lead time data from smallest to largest, multiply your number of observations by 0.8, round up to an integer, then identify the observation at that location. Technically, you should average the two observations at that point if you didn’t have to round, but I’ve not found that materially significant in any of my data for these purposes so I don’t do it.

What to Do with It

I publish my 80% lead time expectation. I talk about it with the people who are anxiously waiting for the delivery. I talk about it with my engineering team. I talk about it with my management team, PMO, project managers and program managers. I talk about it with my lean and agile coaches and consultants and scrum masters. I talk about it with my team leads. I want everyone in the loop and on board with the improvement goal. I use it to explain how certain behavior, such as expedites and high work-in-progress (WIP), work against improving the lead time expectation.

Look at your bar chart showing the changes in your lead time expectation over time. See if it’s moving in the right direction. Use A3s, Toyota Kata, lean principles and systems thinking to improve the system. Engage your upstream and downstream neighbors in the improvement process and in making process policies explicit.

What if the Data Isn’t in One Tool?

Change your process before you change your tooling, and don’t tool up a bad process. A common problem is when an organization doesn’t have the beginning and ending lifecycle states of a given class of work in a single tool. I didn’t say value stream here because you don’t need the whole value stream mapped for every class of work in order to get lead time. You can combine data from multiple tools in order to get the end-to-end, but the simplest thing that could possibly work is to have your top-level item reflect or approximate that end-to-end flow.

Before deciding you need a new tool, there are a couple of easier approaches to try first:
- Add the starting or ending date to the items of interest, whichever of those is missing. Sometimes I see those planning an initiative only track the item until it’s approved and released to engineering. It may be easier to add another date to that item and track it to delivery to production than to change tooling.
- Add more steps to an existing kanban. Likewise, it may be easier to add additional columns to a kanban and keep the items on the board until they are in production. Or get them on the board earlier in their lifecycle.
- Add another kanban. It may be that you don’t have a kanban. Your initiatives or your production requests aren’t tracked well. Get them in a kanban. This is not the same as changing tooling. You probably already have a tool that can give you a kanban. Use it. Every organization should have an epic kanban.
Prerequisites

You can use lead time as a lagging indicator of your process improvement efforts without any of these prerequisites. However, if you are using lead time to set completion date expectations, you should attend to these matters first: stable teams, predictable teams (stable throughput), a stable system (few process changes) and low WIP.

If you are trying to make predictions, being predictable is helpful. A leading indicator for predictability is velocity variance or throughput variance. I compute that as the standard deviation divided by the average, and I want that to be 0.2 or lower. That 0.2 is the point at which the numbers are more reliable, in my experience with many organizations’ data.

You need to have stable teams. Your organization will not be predictable if your team makeup is changing all the time, or if you are moving people around between projects or if you are allocating people to multiple projects.

You need to control your WIP at every level. Put all of these on your metrics dashboard:
- The number of items being worked on per individual. This should be one per individual, or less. It’s usually easier to gather this data at a team level. At a team level, the number of items being worked on should be less than the number of individuals. Encourage working together. If you are pair-programming most of the time, your WIP should be less than the number of pairs. With TDD, good test coverage and good continuous integration practices, you should be able to get multiple pairs on one user story.
- The number of epics or initiatives each team is working on. This should be on your metrics dashboard, and list teams with more than one epic in progress. But remember, it’s not the team’s fault. Fix the system. Don’t blame the team.
- The number of open sprints. This should be 1 per team. On your dashboard, list the teams with more than 1 open sprint.
- The number or epics or initiatives the organization is working on. Graph this measure each sprint on a bar chart so you can see the trend. Work to improve (lower) the trend.
- The number of releases being maintained (fixed, patched, level 3 support). Graph the trend and work to lower it.
- The number of releases being supported (help desk, service desk, level 1 and 2 support). Graph the trend and work to lower it.
Conclusion

Lead time, if tracked and used correctly, is a great predictor for how quickly work will be completed, allowing you to more accurately forecast delivery dates. You should measure it, talk about it and review it often to see what improvements have been made, and can be made. And remember:
- Measure lead time separately for the few types of work where it really matters.
- Use percentiles rather than averages for more accurate forecasting and expectation setting.
- Adjust your process before you change your tools.
— Andrew Fuqua
April 16, 2020
Iterative Indicators: Measuring DevOps ROI

It’s well-established that application of ROI measurements to IT investment remains an imperfect science, and the same goes for ongoing adoption of DevOps methodologies.

The fuzzy math of vendor-driven ROI calculators aside, even the most respected industry watchers have long struggled to create concrete parameters for assessment of technology spending. This, in large part, has been related to the fact that the requirement for continued investment has outweighed the value of short-term metrics.

At the same time, it’s hard to put a price on the influence that paradigms including mobility, virtualization and the cloud are having on today’s business world. The scope of the so-called Applications Economy is only truly beginning to be realized.

For in-depth discussion of DevOps ROI, register today for the upcoming webcast “Maturity Metrics – Analyzing the 2016 State of DevOps Report” with noted industry expert Gene Kim.

With DevOps, the challenge of ROI measurement evokes similar characteristics as with these other bellwether movements. The affect that leveraging DevOps to become more agile and respond adeptly to changing customer expectations will have, long-term, ultimately will be realized as much in organizational viability as in any immediate cost-savings or other factors.

That said, the recently published 2016 State of DevOps Report, assembled by the researchers at DevOps Research and Assessment (DORA), goes further down the path toward creating meaningful ROI metrics than any other such previous effort.

For starters, the 2016 report builds on previous year’s conclusions that so-called “high-performing organizations”—those already deeply engaged in DevOps practices—are “decisively outperforming their lower-performing peers in terms of throughput,” and this gap continues to widen.

Moreover, DORA contends that high performers deploy applications code 200 times more frequently than slow performers and achieve 2,555 times faster lead times. Perhaps even more importantly, in particular as it relates to the ability to improve applications performance, the report estimates that leading DevOps practitioners lay claim to 24 times faster recovery times and three times lower change failure rates.

The State of DevOps Survey, completed by roughly 4,600 technology practitioners, also found that high-performing teams spend far less time on so-called “unplanned work and rework”—most often required to address existing problems—allowing them to spend nearly 50 percent more of their time on new work. The latter brand of effort is typically aimed at adding or improving feature functionality—or, in simple terms, innovation.

As increased software velocity and reduction of MTTR represent the most critical differentiators in today’s environment, where an end user may wait only seconds for an app to load before abandoning your brand altogether, these conclusions obviously carry impressive weight.

Finally, based on a framework established by research analysts at IDC which suggests that hourly application downtime costs can range from $1.25 to $2.5 billion for a Fortune 1000 firm, and that the average cost of a critical application failure is $500,000 to $1 million per hour, the State of DevOps Report allows for the conclusion that high-performing organizations save an average of more than $91 million per year.

Much of this advantage is found, the researchers note, in the form of lower applications change failure rates, or those updates to software code that result in subsequent performance issues or outages. While high-performing DevOps practitioners experience an average change failure rate of 7.5 percent, medium-performing organizations see that figure rise dramatically, to a whopping 38 percent.

If the undeniable adage that “time is money” applies in today’s Applications Economy as much—if not more than ever—than these ROI calculations would appear to infer substantial meaning.

Later this month, Gene Kim, one of DORA’s founders and a key contributor to the State of DevOps Report, will host a webcast with CA Technologies expert Aruna Ravichandran to discuss DevOps ROI and many other related topics. Register for that event today.

July 8, 2016