Caveat: I currently cover DevOps – including test – and security – including the *AST. This does give me a viewpoint that others may not have, and also potentially gives me blinders that others may not have. I am not generally a predictor but a reader of direction. Proclamations like this are rare for me and are shorter-term/more tactical than the grab bag of predictions usually seen this time of year.
I was thinking about this one this morning; the direction of travel of traditional and security testing is very similar at the moment. We are seeing the vast complexity of the overall space and the rapid growth of automation come together to once again make all testing a topic worthy of consideration.
I loved test-driven development (TDD). I can’t speak for others, but RPC/API is just a cool way to implement centralized functionality, and TDD is almost mandatory to develop APIs and the clients that call them. There is no UI to distract or have separate requirements in API development; there is a stated function and correct or incorrect behavior. To a computer scientist, this is predictable beauty. TDD made it more than predictable—it made it a trail through the woods. “We know it needs to do X, Y and Z, with the only side effects being A,” can be detailed with tests to prove that is what it does, and then look to prove the “It doesn’t have other side effects” part. And you can do it all while developing—no need for dev/test iterations. Like any IT solution, it isn’t a great fit for everything, but even if it isn’t called TDD, it is nearly impossible to develop API-based solutions without some level of TDD.
The thing is, TDD is absolutely a developer’s tool. Developers and DevOps pretty much must use it to turn out usable APIs. The test team only has a use for a subset of those tests. Because if the dev team did it right, they passed all the TDD tests, and the test team’s role in that scenario is to look closely at what the developers might have missed. Integration is often beyond the control of an individual dev and/or API, for example, because many different clients may be integrating with it. Test teams, in a true TDD shop, fill the gaps with meaningful tests that evaluate the overall application quality.
Security testing—at least static application security testing (SAST) and increasingly dynamic application security testing (DAST)—are also becoming a core part of the development process. This is as it always should have been. It is far easier to fix a security flaw (or any flaw) at the design/early code stage than in a testing process after deployment. The quality of security testing inputs is also increasing, making it far more viable. And finally, the tools are being effectively integrated. Being a developer who did a lot of security work because we often needed it and had no one to do it, I will say I get a little rush about being told right in my IDE about the security risks in my code. I write, I see, I revise where necessary, and I turn out better code. All without having to pause for security or bang heads over policy.
To date, all this testing is separate, but I suspect, having vaguely the same location in the process of development, that testing will all occur together and the markets may even start to merge a bit—though merger of security testing and functional/performance testing tools is a bit of a stretch, it is more possible than in the past. In most orgs, even though developers have an increasing stake in both sets of testing, the target purchasers are still two different groups—security versus test—with different priorities. That’s why I think they aren’t likely to merge any time soon.
While AI in most technology spaces is currently experiencing the same level of FUD as XML, Java, JavaScript or low-code was in the past, all super-hyped tools in tech do have a sweet spot. Security and testing are in that sweet spot for AI. Both in test generation and in results evaluation, it does it better, faster and with more coverage while giving staff a concrete set of results to pore over and improve security posture and/or code quality. I cannot say loudly enough that in terms of enabling the steps in the SDLC that have suffered from a lack of resources and timeline to write, execute and evaluate tests, AI is a game-changer.
So, all of these things together mean that 2024 is a good time to take a hard look at your testing environment(s) and architecture, both traditional testing and security testing. Now that AI has made advanced automation a fact, it is time to consider implementing the level of testing we always knew we should have but never had the resources. Build it right into the build process and make it part of the organizational DNA—just like Agile is.
If you could hire a specialist who would paw through your code and give you a list of the 25 things you need to fix first, you would do it. That is, essentially, what modern testing tools are. They can do more tests, evaluate more issues and side effects and filter down to the “No—this is really important!” list without costing the team a ton of time.
And keep rocking it. We’re all sitting on bugs and vulnerabilities. I do hope we are way past denying that fact. Apps are big old complex monsters; these days, sometimes built from generations of technologies cobbled together. Of course they have issues. Even yours. So let’s take steps to clean it up, just a little bit. And keep clipping along.
Modern development environments, in which rapid continuous delivery is facilitated by automated continuous integration/continuous development (CI/CD) pipelines, require thorough and automated testing in development prior to integration. This article explains how test-driven development (TDD) and acceptance test-driven development (ATDD) methodologies reduce CI/CD lead times.
Lack of test automation in development impacts CI/CD lead time for subsequent integration, delivery and deployment stages of a CI/CD pipeline in the following ways:
Increased defect rate: Without automated testing during the development stage, more defects and issues are likely to be introduced into the codebase, increasing the time spent on debugging and fixing issues in later stages.
Slower feedback loop: Without test automation, the feedback loop between code changes and test results is slower, making it more challenging to identify and resolve issues quickly. This delay can impact the overall lead time in the CI/CD pipeline.
Reduced confidence in releases: When testing is not performed adequately or isn’t automated, there is less confidence in the quality of the software being released. This may lead to more manual testing and validation, further increasing lead times.
Inefficient use of resources: Without test automation, developers and testers need to spend more time on manual testing, reducing the time available for developing new features or fixing existing issues.
Increased merge conflicts: When test automation is not performed early in the development stage, the risk of merge conflicts increases as different developers work on the same codebase. Resolving merge conflicts can be time-consuming and may cause delays in the integration, delivery, and deployment stages.
Decreased stability and reliability: Inadequate test automation during the development stage can lead to less stable and reliable software. This may require additional time and effort to address issues and ensure that the software is production-ready.
Inconsistent test environments: Manual testing may not always be conducted in a consistent environment, leading to potential discrepancies between development, staging, and production environments. This can cause unexpected issues and delays in the pipeline.
Hindered collaboration: Lack of test automation may reduce visibility into the quality of the codebase, making it more difficult for team members from development, QA, security and operations to collaborate effectively and identify issues early.
Although CI/CD pipelines help catch issues and ensure code quality through automated builds and tests, having a strong foundation of test automation during the development stage helps minimize the potential for issues in the pipeline and ensures a smoother CI/CD process.
There are several reasons why many development teams do not automate tests prior to integration, despite the benefits of doing so.
Lack of awareness or understanding: Some development teams may not be aware of the benefits of test automation or may not fully understand how to implement it effectively in their development process.
Insufficient skills or experience: Test automation often requires specialized skills and knowledge, and some development teams may lack the necessary expertise to implement it effectively.
Time constraints: Developing and maintaining automated tests can be time-consuming. Teams working under tight deadlines might prioritize feature development over creating automated tests, believing that it will save time in the short term.
Limited resources: Test automation tools and infrastructure can require significant investments in terms of cost and resources. Some organizations may not have the necessary budget or resources to invest in test automation, especially for smaller projects or teams.
Resistance to change: Some teams may be resistant to adopting new practices, particularly if they have been successful with their existing processes. Changing established workflows and learning new tools can be challenging and may encounter resistance from team members.
Incomplete or rapidly changing requirements: In situations where requirements are unclear or frequently changing, development teams may struggle to create meaningful automated tests, as they may require continuous updates to stay relevant.
Overemphasis on manual testing: Some teams may place a greater emphasis on manual testing due to a belief that it provides more accurate or thorough results. This perception can lead to a reluctance to invest time and effort into developing automated tests.
Fear of increased maintenance: Automated tests, like any other code, need maintenance and updates as the application evolves. Some teams may be concerned about the additional effort required to maintain a test suite, particularly if they have limited resources.
The amount of test automation that should be done in development prior to the integration stage of a CI/CD pipeline depends on several factors, including the complexity of the application, team resources and the specific project requirements. However, here are some general guidelines to help you determine the right balance:
Unit tests: Developers should aim to automate most, if not all, unit tests during the development stage. They are typically fast to execute and provide quick feedback, allowing developers to catch issues early and minimize the number of defects that make it to the integration stage.
Component and integration tests: While unit tests focus on individual components, component and integration tests verify the interaction between different components or services. These tests should be automated as much as possible to ensure that any issues arising from component interactions are identified and resolved quickly.
Static code analysis: Automating static code analysis checks during development helps identify code quality issues, security vulnerabilities, and potential performance bottlenecks early in the process.
Code coverage: Strive for a high level of code coverage with your automated tests. However, it’s important to remember that code coverage is just one indicator of test quality and should be complemented by other testing strategies.
Striking the right balance between automated and manual testing is important. Automate tests that are repetitive, time-consuming or prone to human error. Manual testing should be reserved for exploratory testing, usability testing and other areas where human intuition and judgment are necessary.
Test-driven development (TDD) and acceptance test-driven development (ATDD) are development methodologies that emphasize creating automated tests before writing the actual code, helping teams ensure code quality and catch issues early prior to integration.
Test-driven development (TDD) is a software development methodology that emphasizes writing tests before writing the actual code. The primary goal of TDD is to create clean, maintainable and bug-free code by catching issues early in the development process. TDD follows a short iterative cycle consisting of three main steps: Write a test, make the test pass and refactor the code. The developer starts by writing a test for a specific feature or function, which initially fails. Next, the developer writes the minimal amount of code necessary to make the test pass. Finally, the developer refactors the code to improve its structure and readability while ensuring that all tests still pass.
Acceptance test-driven development (ATDD) is an extension of TDD that focuses on defining and validating the high-level requirements and acceptance criteria for a feature or functionality before development begins. ATDD involves collaboration among developers, testers, and business stakeholders to ensure that the application meets the needs of its users. The process begins with writing acceptance tests, which are derived from the user’s perspective and based on the acceptance criteria. These tests are written in a human-readable language to facilitate clear communication among all team members.
Once the acceptance tests are defined, the development team follows the TDD process to implement the feature, using the acceptance tests as a guide. As the team develops the code, they also create unit tests and other lower-level tests to validate the implementation. The acceptance tests are executed as part of the development process and serve as a measure of progress. When all acceptance tests pass, it indicates that the feature or functionality meets the defined acceptance criteria, and the development task is considered complete. By using ATDD, teams can improve collaboration, ensure that the application meets user expectations and reduce the risk of miscommunication or misunderstanding requirements.
Hurdles to overcome when transitioning to TDD and ATDDinclude:
Mindset shift: Adopting TDD and ATDD requires a change in mindset, where testing is considered an integral part of the development process rather than an afterthought.
Skill development: TDD and ATDD require specialized skills in writing effective tests and implementing test automation. Development teams may need to invest time and resources in acquiring these skills.
Time and resource allocation: Transitioning to TDD and ATDD might initially lead to a perceived increase in development time, as writing tests before code can seem slower. However, the long-term benefits of improved code quality and reduced debugging time should be considered.
Collaboration and communication: ATDD relies on effective collaboration and communication among developers, testers, security, operations and business stakeholders. Establishing a culture of open communication and collaboration is crucial.
Tooling and infrastructure: Implementing TDD and ATDD may require investments in new tools or changes to the existing development infrastructure to support test automation and continuous integration.
In the context of TDD and ATDD, the roles of quality assurance (QA), security and operations teams are crucial for ensuring a well-rounded and robust software development process. Their involvement helps to address various aspects of the application, leading to a more reliable and secure product.
Quality Assurance (QA): QA team members collaborate closely with developers in both TDD and ATDD methodologies. They can help define test cases, create test plans and provide input on acceptance criteria. In ATDD, they play a particularly significant role in developing and validating acceptance tests, ensuring that the application meets the defined requirements and user expectations. QA team members may also be involved in setting up and maintaining test automation infrastructure, analyzing test results and identifying areas for improvement.
Security: Security professionals contribute to TDD and ATDD by identifying potential security vulnerabilities and ensuring that the application is developed with security best practices in mind. They can help define security-related test cases and help to embed security into the application, reducing the risk of vulnerabilities and improving the overall security posture.
Operations: Operations teams are responsible for managing the infrastructure and deployment processes that support the application. In TDD and ATDD, they can collaborate with developers to ensure that the application is designed and developed with operational concerns in mind, such as scalability, performance and maintainability. Their involvement also helps to promote a DevOps culture where development and operations teams work together to ensure the application’s success in production environments.
A practical roadmap for teams to transition to TDD and ATDD while overcoming the hurdles mentioned earlier could look like this:
Build awareness and buy-in:
Start by educating the team on the benefits and principles of TDD and ATDD. Obtain buy-in from team members and stakeholders to ensure a smooth transition.
Provide training and skill development:
Organize workshops, training sessions or pair programming exercises to help team members develop the necessary skills for writing effective tests and implementing test automation.
Start small and iterate:
Begin the transition by applying TDD and ATDD practices to small, manageable parts of the project or a pilot project.
Foster a culture of collaboration:
Establish regular meetings or workshops for developers, testers, security, operations and business stakeholders to collaborate on defining requirements and acceptance criteria. Encourage open communication and shared responsibility for test creation and maintenance.
Set up the necessary tooling and infrastructure:
Allocate time and resources to select, set up and configure these tools and ensure they are properly integrated into the development process.
Monitor progress with metrics:
Monitor key metrics, such as code coverage, test execution time, defect rates and lead time to track progress and identify areas for improvement.
Continuously improve and adapt:
Regularly review the team’s progress, gather feedback, and adjust the process as needed.
Summary
TDD and ATDD methodologies contribute to reducing CI/CD lead times by automating tests during development. This improves code quality, minimizes defects and promotes collaboration among team members from the outset of the development process. By writing tests before the actual code and focusing on meeting acceptance criteria, teams can catch and resolve issues early, leading to fewer delays and reduced time spent on debugging and fixing errors during the integration, delivery and deployment stages. These practices also help ensure that the application aligns with user expectations and operational requirements, resulting in a smoother and more efficient CI/CD pipeline.
Anyone who is passionate about modern quality engineering processes for building high-quality software to participate is encouraged to join new ShiftSync Community from Tricentis to engage in discussions about this topic and other development methodologies.
The whole concept of Agile and DevOps was to iterate development faster and deliver results in a more timely manner. As we learned more about both methodologies, processes and policies were put into place that improved the quality of what was created. Early ideas of quality like, “We can just do another iteration,” still exist, but are largely not used. Rollbacks are a thing in most orgs, and though the environment is more complex, rollbacks are easier to manage in the smaller development changes that iterations generally represent.
And once we got past the whole, “Anything that slows the process down is bad!” mentality that permeated early implementations of both DevOps and Agile development, we started seeing amazing things. But including things like security and functional testing were not viable in the fast-iteration world that Agile and DevOps created. And they really weren’t viable at most orgs in the slow-iteration world of waterfall, either. Staffing and time constraints made them fall far short of optimal in the vast majority of organizations. But putting them into the DevOps process challenged vendors to find a way to do things faster and challenged organizations to look at quality and security a little differently.
And now, we have tools that can test—be it functional, integration or security—at the speed of DevOps. Tools that enable Agile development with an eye toward security. A far larger number of developers now acknowledge their part in application security and are creating more secure code.
Keep an eye on how AI/ML is being used in testing—all forms of testing—and how it evolves in the next year or two. The same with full-on automation. I’ve made no secret that I view AI/ML in the testing space as advanced automation because of what it enables, so maybe “Watch what AI/ML-enabled automation is available,” is more accurate. The ability to create, run, process results and even modify code to resolve issues are all out there today, and getting better. Faster, more accurate, more automated.
Much like back when intrusion detection became intrusion prevention, I would not just turn it on and walk away, but think about the ability to upskill developers; “Here is a list of things the automated system found, and a list of fixes it proposes.” They are literally being shown how to do their craft better.
I’ve been a developer, so I’m not being offensive, here. If you get something that tells you, “This is an issue; here is why it happened and here is what to do to fix it,”take the opportunity to get better at coding. Look closely at the what, the why and how to fix it and incorporate that into your work.
And keep rocking it. We’ll be making more stable and secure code, but you’ve kept systems running without that feedback and fixed the errors along the way. This just makes it easier to find the errors early—before they crash production and make a fire drill.
I have long noted the trend of pendulums in IT, particularly in organizations with longer histories. Centralized IT will be achieved, its weaknesses noticed and a movement will begin to decentralize. Decentralized IT is achieved, and people remember its weaknesses, causing centralization.
The same is true with speed of delivery versus quality. Faster will be more important for a while, then the reality that faster causes quality concerns will surface, and reliable becomes the keyword, causing slower rollouts of better product.
This has been going on since I started in IT, and while it is more evident in enterprises, any company that lasts long enough will see these cycles run through.
Agile and DevOps, as the Great Disruptors, stood a chance of redefining the tensions that cause these pendulum swings. Centralized versus decentralized is less important if stakeholders are all truly involved and informed—something that the culture side of the DevOps crowd aimed to achieve. Similarly, fast and quality were promised, as the ability to recover from inevitable mistakes would be rapid, while integrated testing would uncover most issues.
We have enough years under our belts to know at this point that these issues are, at best, resolved with a resounding asterisk that is clarified sometimes.
One nice thing about DevOps (more than agile) is that it has not stopped evolving yet, which means there is still hope that rapid delivery and high quality can both be answered while involvement of business owners and all the various IT groups will reduce the centralized/de-centralized cycle. But we’re not there yet.
Split.io—purveyor of feature flagging/tagging tools—has a pretty good blog about what they discovered when they went out and talked to hundreds of shops. It shows that failures are happening at an alarming rate, and at some number of shops, for your average bug, they will still be fixing this released issue when the next release goes out. That is asking for trouble, in my experience.
Better integrated testing is certainly part of the answer. TDD is good, but not enough, since it is more granular than some tests require. Split believes that feature flags are part of the answer, and I believe they’re right. But this approach is a band-aid to the problem.
By the time you flip a feature flag to turn off a bad feature, you’ve already released a bad feature. And you will be spending time researching and fixing it. I believe feature flags should be used to catch it when this (inevitably) happens in a complex coding environment, but at the same time the source of issues should be looked into. An ounce of prevention is indeed worth a pound of cure.
Figuring out what is allowing bugs to slip through into production rather than counting on remediating them (using precious time to do re-work) should be a priority moving forward. Is there a process/policy that is at the source? Is it simply a training issue? Training issues are repetitive—teaching today does nothing for new algorithms/tools put to use after today, and turnover can create a training gap if you’re not staying on top of it. For some shops, is speed of delivery faster than some piece (normally testing, but some shops suffer for development on certain platforms also) of the delivery chain can adequately respond to? Do you need to re-evaluate toolchain and delivery timelines?
We are delivering more code, faster than ever, and that is exciting. But quality matters, and anyone who relies on end-users to test is playing a very dangerous game. Find ways to ensure quality without slowing down the process more than necessary. At this point, I honestly would implement runtime-flippable feature flags as a starting point, simply to be more responsive to issues. From there, work to resolve the underlying reasons that bugs are cropping up.
Until then, keep cranking it. Keep working with the business to get quality systems in front of users, and enjoy doing some great work.
At John Hancock, we use Test-Driven Design (TDD) for all our new initiatives. TDD has three simple steps:
Before you write functional code, write a test that proves if the implementation works or fails.
Write the code and watch the test pass.
Check if the added functionality changes your holistic approach and if you want to consolidate/refactor some of your code. If yes, refactor these sections.
That sounds simple enough, so why write an article about a three-step process? Why do we, at John Hancock Digital Enablement, get so excited about TDD? We have seen major impacts to our developers, products and culture:
Increased motivation and self confidence.
Shorter Release-Cycles.
Increased Code Quality.
Massive Reduction of Bugs in Production.
The Foggy Path
Imagine your development process like a foggy path, where you can see directly in front of you, but anything further away becomes foggy and imprecise. Imagine it like a journey into the unknown.
Obstacles may appear suddenly, forcing you to rethink your approach (refactoring), or you may get above the fog after a long journey and realize that there is a much simpler way to reach your goals (also refactoring).
This uncertainty and constant change may appear to be frightening and trigger anxiety, but TDD is the safety net, because you know that what you’ve built holds up against all the risks that have been identified. As a developer, I can take much higher risks and try different approaches and get quick validation.
The test cases built out as part of the TDD flow are ideally integrated into a continuous integration/continuous development (CI/CD) Platform, giving fast and reliable feedback to the engineers about unexpected behaviors their code change may have caused. Because the feedback to the engineer is near instant, the developer is familiar with the change that caused the error and is able to resolve it rapidly.
TDD Is the Opposite of “Never Change a Running System”
A well-known issue is that the later a bug is found, the more costly it is to resolve. TDD enables fast feedback loops so bugs can be resolved quickly and cost efficiently.
As an application grows, TDD gives you a solid foundation of everything that has already been built. You can be confident that what you have built is solid and that new functionality can be added to it easily and without taking the risk of breaking everything. In other words, TDD is the opposite of “never change a running system” and enables us to build even better software.
TDD Always Provides Up-to-Date Code Documentation
TDD provides excellent documentation toward the requirements and behaviors of the application. While documentation always lags behind the functional code and typically is written after a functionality is implemented, in the world of TDD everything starts with a test. The test documentation is always up to date and gives developers an easy and understandable way to figure out what the application is achieving without dissecting the entire code.
The following quote from Terry Pratchett explains it best: “If you do not know where you come from, then you don’t know where you are, and if you don’t know where you are, then you don’t know where you’re going. And if you don’t know where you’re going, you’re probably going wrong.”
As an application grows, evolves and is built up over time, and as it’s passed down and handed over, TDD gives it documentation and describes its solid foundation. The act of writing a test case forces a developer to stop and think about the functionality that is to be implemented and describe its desired outcomes.
TDD Provides Early Feedback
Through TDD we receive early feedback about the path we are about to set foot on. Early indicators can help us correct our assumptions or help rephrase or rewrite a requirement so that we deliver customer value.
Early indicators that we’ve encountered:
Functionality cannot be measured: Why are we implementing this if we cannot measure its outcome?
Functionality does not add any value.
Functionality cannot be tested: Similar to the first bullet, if a functionality cannot be tested, how can we accept it’s delivery?
Functionality can be easily embedded into an existing part of the application that needs refactoring: This can trigger the refactoring before we start the implementation.
As described, it is very helpful to think about the steps ahead before jumping in, which can be difficult for engineers to do. TDD does so in a language that is natural to engineers.
TDD Embraces an Ever-Changing World and Removes Fear of Change
TDD as a part of eXtreme Programming (XP) accepts and embraces an ever-changing world we cannot control. Changes may cause discomfort and anxiety as planning becomes difficult/different.
In an unstable, disorganized software project, changes are thrown at it, a solid foundation is missing, testing is skipped or postponed to the end and refactoring is skipped all in the name of on-time delivery. The first delivery of the application may be on time, but as time goes on the anxiety grows, there are random bugs everywhere, the more bugs that are fixed, the more issues pop up and you essentially end up playing a game of whack-a-mole in which you will never be in control.
In a project that embraces XP principles, such as TDD, we try to remain in control from the beginning, facing challenges early so they do not bite us later on. A solid test base gives developers, and everybody else on the team, confidence that the delivered solution will work, but most importantly it removes the fear of change.
Without fear of change, we can try all sorts of crazy things, test out ideas and see them turn out or fail without the code failing, without losing a customer, without getting that phone call in the call center.
Traditional QA and Release Cycles Are too Slow for this New Era of Ever-Changing Needs
As software grows, its testing becomes more intense. So, with each new piece of functionality there must be a corresponding test case. The manual execution of those tests becomes lengthy, the feedback loop to the developers slower, releases get held up, customers do not get new functionality in time and, in the end, money is lost.
Traditional QA and release cycles are too slow for this new era of ever-changing needs. What if I want to release daily or weekly? Do we have to execute the test procedures every time? What if bugs are found and repaired? Do we need to rerun the entire set?
The answer is yes—but if a machine does it in a few minutes, who cares?
While I am not a proponent of no human testing, what I am saying is that 95% of manual QA can be automated and executed by a machine, and TDD provides a fantastic test set to get started with. Humans may catch errors that are hard for a machine to identify or imagine since human behavior is hard to predict, so spot testing as well as exploratory and behavioral testing are still needed.
We Are Left with a Team that Is Hungry for More and Only Afraid of the Enormity of the Possible
What are we left with at the end of a TDD project? We have a well-documented, easy to understand, easy to change and stable code base lead by a team that is not afraid of change.We are left with a team that is hungry for more and only afraid of the enormity of the possible.
Unit testing is like exercise: Companies know it’s good for you and you should do it, but given their druthers, they’d rather avoid the effort and just get to the benefit.
I wish I could say it’s otherwise. After all, making unit testing a standard practice in a company’s software development process is probably the first, best thing a business can do to significantly boost code quality. To go one step further, adopting the best practices of Test Driven Development (TDD) can not only speed up release velocity in the long term but also has the benefit of making code maintenance less costly. For example, if you want to know what a piece of code is supposed to do, all you need to do is review the unit tests. This reduces the need for volumes of support documentation.
It’s almost a no brainer. Yet, there’s still a lot of perception out there that TDD’s dictum to write the test first slows down the development process. The thinking goes that if a developer is going to write a test for the code before writing the code, just save time and write the code anyway and then write the tests only for critical parts of the code.
This might be a sound rebuttal of the TDD process, except for two things. First, can you name a part of your codebase that should not be tested? And second, as those of us out on the terrain have discovered, when release time approaches and the corners start to get cut–and they do get cut–many times the first thing to go is unit testing. More than one development manager has uttered the words “We’ll catch it in QA” when faced with an approaching deadline and an understaffed development team. Sadly, when given the choice between getting so-so code out the door on time and delaying the release in order to create great code, it’s been my experience that in companies that have not embraced unit testing to an acceptable code-coverage threshold, so-so code will win every time. Those who have taken a swig of the TDD Kool-Aid don’t have to make that choice. The tests are already written.
It’s true that all the unit testing in the world will not make poorly engineered code better. But, if the engineering is sound, unit testing works. Still, companies resist, even those who program in languages from which unit testing as a way of life emerged. There are a lot of businesses out there programming Java, C#, Python, etc., that want the benefit of unit testing and yet don’t want to pay for the effort. When it comes to the mainframe community, resistance be as strong as that found in the French Underground in WWII.
Why? Well, when it comes to mainframe development, unit testing is much harder to implement than in the x86 world. There’s the problem of language compatibility. There is still a lot of COBOL out there supporting the 80% of the world’s data that still resides on a mainframe. Unit testing as a way of life is a fairly recent occurrence in the history of computing. There’s a boatload of testing libraries out there for Java, C/C++, .NET, Python. But until recently, there weren’t any unit testing frameworks for COBOL. Also, in traditional mainframe programming, you can have situations in which many developers work on the same body of code. Test isolation can be hard.
Finally, there’s the matter of culture. Remember, the thing that makes unit testing and TDD work is that designing and executing tests are the responsibility of the developer. Pushing testing onto developers is a notion at odds with traditional mainframe sensibilities. In traditional mainframe shops in days of old, developers wrote the code and QA tested it. It’s a process that worked for years as long as those years meant you could release code every six months. Today it’s all about getting code revisions out weekly, if not daily. Still, the Dev/QA segmentation is there in the older companies that are mainframe driven.
But that was then and this is now. I am happy to report it’s a new day in the land of software development of the mainframe. Adopting TDD in general and unit testing in particular as a way of life in mainframe software development is not only possible but it borders on being, as I describe above, a no brainer. The tools and processes are there. Let’s take a look.
Unit Testing Mainframe Code Using IBM Z Open Unit Test
Writing unit tests for the mainframe has had a difficult history. If you’ve started your professional career in software development within the last ten years, you’ve had the pleasure of enjoying the benefit of TDD and CI/CD as a standard way of doing business. Tools such as Jenkins, JUnit, Selenium and the like have always been around. But in the mainframe world, for many, this is all new stuff.
As a result, for many mainframe developers, the notion of writing a unit test let alone integrating one into a CI/CD process means entering a strange new world. In the pre-tool world, writing any test, let alone a unit test, meant spending days doing tedious, laborious work in some sort of jury-rigged testing environment. Or, in some cases, the developer might not be responsible for testing the code at all. Rather, the developer writes the code and gets it working to his or her satisfaction by any means necessary. Then he or she gives the code the old “toss over the wall” to the QA department for testing. What those testing processes are might be known to outsiders, but in many cases they’re not. So, the development process becomes at best a slow, grinding exchange of test results and remedies between groups that have limited insights into the processes and priorities of the other.
The good news is those in the known have come to understand there’s a better way. The better way involves three things. First, making the developer responsible for testing the code he or she writes. Second, removing the cultural, and in many cases, the organizational wall between development and QA. And third, putting tools at the forefront of testing activity.
In other words, the new thinking is to understand testing must be done as early as possible in the software development cycle (hence, the Shift Left Movement), everybody is responsible for the quality of the software and tool adoption is critical in the TDD process.
The construction industry caught onto this notion a long time ago. A bulldozer driven by a competent operator is a lot better for moving earth than a legion of human beings with shovels. The same is true in software development. It’s better to give a single developer a tool to create and apply tests to his or her code than have a legion of other employees sit around with text editors writing one-off units tests for their code and applying them by hand.
Which brings us to IBM Unit Test.
IBM Z Open Unit Test
IBM Z Open Unit Test is a tool designed to make writing and executing unit tests on the mainframe fast and efficient. Also, IBM Z Open Unit Test facilitates integrating unit tests directly into a company’s CI/CD processes under a tool such as Jenkins.
The tools support automated unit testing for batch and CICS COBOL programs. Also, IBM Z Open Unit Test provides recording and playback functionality for CICS COBOL programs, as shown in Figure 1. This tooling allows developers to easily record external calls to CICS and import that recorded data for modification in playback. The ability to record, modify and playback test data at execution time means developers can run their automated unit tests as part of a build (or, as a batch job). There is no need to deploy to CICS, as it is now stubbed out.
IBM Z Open Unit Test aligns with TDD methodology in that the tool allows developers to record, view and modify the input and expected output as assertions, within an easy to use interface. This can be done before making a change to a program, and developers can understand what kind of data is flowing in and out.
Figure 1: IBM Z Open Unit Test has data recording and playback capabilities for Batch and CICS COBOL programs.
As mentioned above, IBM Z Open Unit Test can be run from within an IDE or executed as part of the enterprise’s CI/CD process. Also, the product supports test debugging and importing tests written in mainstream formats such as JUnit and SonarQube. These are good and important features. But the real kicker is support for code coverage.
Why Code Coverage Counts, Even for Mainframe Testing
Code coverage is the elephant in the room in terms of testing administration and management, particularly around unit testing. One of the key principles of test driven development is that every line of code that gets written gets tested. Yet this is the dream rather than the practice. I’ve been in more than a few companies that say they support TDD, yet when asked if they test every line of code written, which translates into 100% code coverage, the answer usually involves something along the lines of “we’re getting there.”
Many companies are tickled pink that their developers are doing any unit tests at all. In such companies, asking a developer to get to 100% coverage is akin to asking him or her to go to the moon and back during lunch.
Some companies that have matured a bit set the coverage standard at 80%. This means that as long as eight out of every 10 lines of code written get touched by a unit test, things are just fine. Or are they? What if the untested two lines of code is a short else clause in an if-then statement that eats memory and blocks I/O? Are these the two lines of code you want to leave untouched in a test? Remember this: Typically it’s not the tested code that wreaks havoc, it’s the code that’s untested.
Thus, the mission critical importance of 100% code coverage.
Mainframe development has lagged a bit embracing the imperative to ensure 100% code coverage. This is mostly due to the culture that went with legacy software development practices and the immaturity of tools at the time. The good news is both culture and tools are evolving.
Getting back to IBM Z Open Unit Test, the tool not only provides test coverage reporting, which is critical for providing the metrics necessary to get to 100% coverage, but also a feature that supports negative testing.
Negative testing is also known as an unhappy path testing. It’s an important, yet often overlooked aspect of unit testing. Just getting a piece of code to work under the right conditions is hard enough work. To subject that code to every bad condition (unhappy path) that can happen and make sure it still works, requires significantly more work. Thus, when crunch time comes, and it always comes, typically the developer will let negative, unhappy path, testing go. It’s not malicious intent. It’s about meeting a deadline.
Fortunately, this dynamic is a known problem in the industry. IBM Z Open Unit Test addresses the problem by providing a feature that automatically generates negative tests. This is a big deal. In fact, if this is all the tool did, it would be a boon to the practice of TDD on the mainframe. Writing negative tests is a tedious, laborious activity. Using a tool such as IBM Unit Test not only saves time but helps achieve the ultimate goal: 100% code coverage.
Putting It All Together
Unit testing is an essential part of the practice of TDD. While TDD has taken hold throughout the x86 based distributed computing, there has been a lag to embrace the discipline in mainframe computing.
Mainframe computing is becoming very cool again, if for no other reasons that it presently controls 80% of the world’s data and in many ways was the cloud before the cloud. In the past, TDD has been hard to implement in the mainframe world due to a lack of tools to support the discipline. Fortunately, with modern tools such as IBM Z Open Unit Test, the benefits of TDD can be easily enjoyed in the mainframe development community.
It’s almost a no brainer. Yet, there’s still a lot of perception out there that TDD’s dictum to write the test first slows down the development process. The thinking goes that if a developer is going to write a test for the code before writing the code, just save time and write the code anyway and then write the tests only for critical parts of the code.