Tag: code coverage

Unit testing is like exercise: Companies know it’s good for you and you should do it, but given their druthers, they’d rather avoid the effort and just get to the benefit.

I wish I could say it’s otherwise. After all, making unit testing a standard practice in a company’s software development process is probably the first, best thing a business can do to significantly boost code quality. To go one step further, adopting the best practices of Test Driven Development (TDD) can not only speed up release velocity in the long term but also has the benefit of making code maintenance less costly. For example, if you want to know what a piece of code is supposed to do, all you need to do is review the unit tests. This reduces the need for volumes of support documentation.

It’s almost a no brainer. Yet, there’s still a lot of perception out there that TDD’s dictum to write the test first slows down the development process. The thinking goes that if a developer is going to write a test for the code before writing the code, just save time and write the code anyway and then write the tests only for critical parts of the code.

This might be a sound rebuttal of the TDD process, except for two things. First, can you name a part of your codebase that should not be tested? And second, as those of us out on the terrain have discovered, when release time approaches and the corners start to get cut–and they do get cut–many times the first thing to go is unit testing. More than one development manager has uttered the words “We’ll catch it in QA” when faced with an approaching deadline and an understaffed development team. Sadly, when given the choice between getting so-so code out the door on time and delaying the release in order to create great code, it’s been my experience that in companies that have not embraced unit testing to an acceptable code-coverage threshold, so-so code will win every time. Those who have taken a swig of the TDD Kool-Aid don’t have to make that choice. The tests are already written.

It’s true that all the unit testing in the world will not make poorly engineered code better. But, if the engineering is sound, unit testing works. Still, companies resist, even those who program in languages from which unit testing as a way of life emerged. There are a lot of businesses out there programming Java, C#, Python, etc., that want the benefit of unit testing and yet don’t want to pay for the effort. When it comes to the mainframe community, resistance be as strong as that found in the French Underground in WWII.

Why? Well, when it comes to mainframe development, unit testing is much harder to implement than in the x86 world. There’s the problem of language compatibility. There is still a lot of COBOL out there supporting the 80% of the world’s data that still resides on a mainframe. Unit testing as a way of life is a fairly recent occurrence in the history of computing. There’s a boatload of testing libraries out there for Java, C/C++, .NET, Python. But until recently, there weren’t any unit testing frameworks for COBOL. Also, in traditional mainframe programming, you can have situations in which many developers work on the same body of code. Test isolation can be hard.

Finally, there’s the matter of culture. Remember, the thing that makes unit testing and TDD work is that designing and executing tests are the responsibility of the developer. Pushing testing onto developers is a notion at odds with traditional mainframe sensibilities. In traditional mainframe shops in days of old, developers wrote the code and QA tested it. It’s a process that worked for years as long as those years meant you could release code every six months. Today it’s all about getting code revisions out weekly, if not daily. Still, the Dev/QA segmentation is there in the older companies that are mainframe driven.

But that was then and this is now. I am happy to report it’s a new day in the land of software development of the mainframe. Adopting TDD in general and unit testing in particular as a way of life in mainframe software development is not only possible but it borders on being, as I describe above, a no brainer. The tools and processes are there. Let’s take a look.

Unit Testing Mainframe Code Using IBM Z Open Unit Test

Writing unit tests for the mainframe has had a difficult history. If you’ve started your professional career in software development within the last ten years, you’ve had the pleasure of enjoying the benefit of TDD and CI/CD as a standard way of doing business. Tools such as Jenkins, JUnit, Selenium and the like have always been around. But in the mainframe world, for many, this is all new stuff.

As a result, for many mainframe developers, the notion of writing a unit test let alone integrating one into a CI/CD process means entering a strange new world. In the pre-tool world, writing any test, let alone a unit test, meant spending days doing tedious, laborious work in some sort of jury-rigged testing environment. Or, in some cases, the developer might not be responsible for testing the code at all. Rather, the developer writes the code and gets it working to his or her satisfaction by any means necessary. Then he or she gives the code the old “toss over the wall” to the QA department for testing. What those testing processes are might be known to outsiders, but in many cases they’re not. So, the development process becomes at best a slow, grinding exchange of test results and remedies between groups that have limited insights into the processes and priorities of the other.

The good news is those in the known have come to understand there’s a better way. The better way involves three things. First, making the developer responsible for testing the code he or she writes. Second, removing the cultural, and in many cases, the organizational wall between development and QA. And third, putting tools at the forefront of testing activity.

In other words, the new thinking is to understand testing must be done as early as possible in the software development cycle (hence, the Shift Left Movement), everybody is responsible for the quality of the software and tool adoption is critical in the TDD process.

The construction industry caught onto this notion a long time ago. A bulldozer driven by a competent operator is a lot better for moving earth than a legion of human beings with shovels. The same is true in software development. It’s better to give a single developer a tool to create and apply tests to his or her code than have a legion of other employees sit around with text editors writing one-off units tests for their code and applying them by hand.

Which brings us to IBM Unit Test.

IBM Z Open Unit Test

IBM Z Open Unit Test is a tool designed to make writing and executing unit tests on the mainframe fast and efficient. Also, IBM Z Open Unit Test facilitates integrating unit tests directly into a company’s CI/CD processes under a tool such as Jenkins.

The tools support automated unit testing for batch and CICS COBOL programs. Also, IBM Z Open Unit Test provides recording and playback functionality for CICS COBOL programs, as shown in Figure 1. This tooling allows developers to easily record external calls to CICS and import that recorded data for modification in playback. The ability to record, modify and playback test data at execution time means developers can run their automated unit tests as part of a build (or, as a batch job). There is no need to deploy to CICS, as it is now stubbed out.

IBM Z Open Unit Test aligns with TDD methodology in that the tool allows developers to record, view and modify the input and expected output as assertions, within an easy to use interface. This can be done before making a change to a program, and developers can understand what kind of data is flowing in and out.

As mentioned above, IBM Z Open Unit Test can be run from within an IDE or executed as part of the enterprise’s CI/CD process. Also, the product supports test debugging and importing tests written in mainstream formats such as JUnit and SonarQube. These are good and important features. But the real kicker is support for code coverage.

Why Code Coverage Counts, Even for Mainframe Testing

Code coverage is the elephant in the room in terms of testing administration and management, particularly around unit testing. One of the key principles of test driven development is that every line of code that gets written gets tested. Yet this is the dream rather than the practice. I’ve been in more than a few companies that say they support TDD, yet when asked if they test every line of code written, which translates into 100% code coverage, the answer usually involves something along the lines of “we’re getting there.”

Many companies are tickled pink that their developers are doing any unit tests at all. In such companies, asking a developer to get to 100% coverage is akin to asking him or her to go to the moon and back during lunch.

Some companies that have matured a bit set the coverage standard at 80%. This means that as long as eight out of every 10 lines of code written get touched by a unit test, things are just fine. Or are they? What if the untested two lines of code is a short else clause in an if-then statement that eats memory and blocks I/O? Are these the two lines of code you want to leave untouched in a test? Remember this: Typically it’s not the tested code that wreaks havoc, it’s the code that’s untested.

Thus, the mission critical importance of 100% code coverage.

Mainframe development has lagged a bit embracing the imperative to ensure 100% code coverage. This is mostly due to the culture that went with legacy software development practices and the immaturity of tools at the time. The good news is both culture and tools are evolving.

Getting back to IBM Z Open Unit Test, the tool not only provides test coverage reporting, which is critical for providing the metrics necessary to get to 100% coverage, but also a feature that supports negative testing.

Negative testing is also known as an unhappy path testing. It’s an important, yet often overlooked aspect of unit testing. Just getting a piece of code to work under the right conditions is hard enough work. To subject that code to every bad condition (unhappy path) that can happen and make sure it still works, requires significantly more work. Thus, when crunch time comes, and it always comes, typically the developer will let negative, unhappy path, testing go. It’s not malicious intent. It’s about meeting a deadline.

Fortunately, this dynamic is a known problem in the industry. IBM Z Open Unit Test addresses the problem by providing a feature that automatically generates negative tests. This is a big deal. In fact, if this is all the tool did, it would be a boon to the practice of TDD on the mainframe. Writing negative tests is a tedious, laborious activity. Using a tool such as IBM Unit Test not only saves time but helps achieve the ultimate goal: 100% code coverage.

Putting It All Together

Unit testing is an essential part of the practice of TDD. While TDD has taken hold throughout the x86 based distributed computing, there has been a lag to embrace the discipline in mainframe computing.

Mainframe computing is becoming very cool again, if for no other reasons that it presently controls 80% of the world’s data and in many ways was the cloud before the cloud. In the past, TDD has been hard to implement in the mainframe world due to a lack of tools to support the discipline. Fortunately, with modern tools such as IBM Z Open Unit Test, the benefits of TDD can be easily enjoyed in the mainframe development community.

— Bob Reselman