For nearly the entire history of application development—certainly since application development became available to a larger market with the adoption of the personal computer for expansive business use—testing has taken a back seat across most of the industry. This is not because we don’t care about code quality but because it takes resources away from developing cool new tools—or even products—and slows down release cycles. Historically, both of these issues have been huge. Manual testing staff grows quickly from a few people to a massive department and waiting for that process to complete could take forever.
So, we’ve not done a lot of it. Oh, when executives or someone in leadership demanded more thorough testing, IT management would promise to do so and then—to fit within budget and time constraints—would tell developers to test their own code. There are several areas where you don’t ask the person responsible for creating the final product to review that final product—because if they knew they were doing something wrong, they’d fix it. Writing is one such area—that’s why good editors are worth their weight in gold, even in the age of the internet where anyone can (and too many do) spew onto the digital page whatever they like without the benefit of an editor. Another is testing application code. I test as I write my code, and like every person who has spent time as a professional developer, I have made a couple of stupendous errors. If I’d known that the particular weakness I left in the system was there, I would not have said it was done.
When DevOps came along, I expected things to change—until I understood that DevOps (and Agile, in lockstep) was all about delivering faster, not better. Anything that slowed down the process—testing, security, even compliance, to an extent, that cannot be avoided—was frowned upon. When test-driven development (TDD) came along, I instantly developed a love/hate relationship with it. We used it for a project I was on and, frankly, it shifted quality concerns to the tests—the code could be forced to pass all of the tests, but how thorough and accurate the tests themselves were—particularly as the project grew and changed—became the question. Done right, TDD would solve a lot of our testing problems. Heck, even done wrong it would have taken care of most problems that simple unit testing could cover.
But after decades, I think we are finally at the point where we can demand and require thorough testing with acceptable and growing coverage numbers. And you need to consider it. Test automation has come a long way and far too many of you haven’t come along with it. If you struggle to justify investing in test suites, I get it. I’ve been the manager that had to choose between improving DB performance and testing. But the modern iteration includes security testing. So if, out-of-the-box, you could get a tool that checked for valid inputs and alerted to places where invalid data might break the application, that might help. And that’s just the simplest case. I’m currently working in an adjacent space, so I don’t want to get too crazy selling you here. I’ll just say, “Testing isn’t what it was even a couple of years ago. Check it out.” Both application and security testing. And the changes will continue to come as claims of using AI/ML turn into applying actual algorithms that your AI or data science people can compare to help choose a high-quality product.
We could debate all day whether DevOps made apps more or less stable and secure, but we don’t need to. Finally, continuous testing is real. Check out the vendors doing this, compare them and see how easy it is to fit them into your environment. More stable, more secure, not the hold-up you have stuck in your head. What’s not to like? You are already rocking it; automated testing could make you look even better.
Testing has long been a problem child of IT in general, AppDev in particular, and now it is DevOps’ problem. There are things that DevOps can do to improve the chances of tests actually occurring for your application(s). Interestingly, there are some that think we no longer need testing; the argument is that because new releases can be cranked out so quickly if something needs to be fixed that it is irrelevant. Combining feature flags with the ability to roll out a new version and, well, there may be some truth to that—at least for a limited subset of testing.
If an organization implemented feature flags for user acceptance and new feature testing and then standardized on test-driven development for DevOps development teams, that would leave systems and security testing. Systems testing could, theoretically, be replaced by the ability to roll out a new version. Although systemic issues can be complex and difficult to fix, this is a riskier proposition than extending test-driven development to the systems level. For the sprint, assign a person responsible for the overall integrated system and have the rest of the team support them through the process of making certain all works as expected.
Testing—particularly systems testing—slows releases. I get that. The rate at which most of us can iterate once DevOps systems and processes are in place is fast enough that you can afford a bit of slowdown. This leaves security testing as the last standalone bit. But some security validation can happen during the development process if DevSecOps tools are being utilized. That still leaves some security issues to test for but does remove a large chunk.
The key to these last bits is to get them integrated into the process/toolchain. DevOps runs straight through, over and over. Any bit worked into the “runs straight through” will be executed. If today’s iteration falls for the “We ran over, we’ll make the deadline with less testing” mentality; not an issue—it’s one iteration. Just don’t let that become the default answer to someone not completing an iteration’s work on time. Development can have unexpected speed bumps that delay delivery and acknowledging that and documenting what it was for future reference is important. Covering it up by reducing testing time risks a less stable product and does not help overcome the issue in the future.
So start thinking in terms of DevOpsTest. Make sure the product you are delivering represents the massive time investments you’ve made producing and maintaining it. You’re rocking it—and something like integrated testing ensures that users see how great you are rocking it and not an error page.
I have long noted the trend of pendulums in IT, particularly in organizations with longer histories. Centralized IT will be achieved, its weaknesses noticed and a movement will begin to decentralize. Decentralized IT is achieved, and people remember its weaknesses, causing centralization.
The same is true with speed of delivery versus quality. Faster will be more important for a while, then the reality that faster causes quality concerns will surface, and reliable becomes the keyword, causing slower rollouts of better product.
This has been going on since I started in IT, and while it is more evident in enterprises, any company that lasts long enough will see these cycles run through.
Agile and DevOps, as the Great Disruptors, stood a chance of redefining the tensions that cause these pendulum swings. Centralized versus decentralized is less important if stakeholders are all truly involved and informed—something that the culture side of the DevOps crowd aimed to achieve. Similarly, fast and quality were promised, as the ability to recover from inevitable mistakes would be rapid, while integrated testing would uncover most issues.
We have enough years under our belts to know at this point that these issues are, at best, resolved with a resounding asterisk that is clarified sometimes.
One nice thing about DevOps (more than agile) is that it has not stopped evolving yet, which means there is still hope that rapid delivery and high quality can both be answered while involvement of business owners and all the various IT groups will reduce the centralized/de-centralized cycle. But we’re not there yet.
Split.io—purveyor of feature flagging/tagging tools—has a pretty good blog about what they discovered when they went out and talked to hundreds of shops. It shows that failures are happening at an alarming rate, and at some number of shops, for your average bug, they will still be fixing this released issue when the next release goes out. That is asking for trouble, in my experience.
Better integrated testing is certainly part of the answer. TDD is good, but not enough, since it is more granular than some tests require. Split believes that feature flags are part of the answer, and I believe they’re right. But this approach is a band-aid to the problem.
By the time you flip a feature flag to turn off a bad feature, you’ve already released a bad feature. And you will be spending time researching and fixing it. I believe feature flags should be used to catch it when this (inevitably) happens in a complex coding environment, but at the same time the source of issues should be looked into. An ounce of prevention is indeed worth a pound of cure.
Figuring out what is allowing bugs to slip through into production rather than counting on remediating them (using precious time to do re-work) should be a priority moving forward. Is there a process/policy that is at the source? Is it simply a training issue? Training issues are repetitive—teaching today does nothing for new algorithms/tools put to use after today, and turnover can create a training gap if you’re not staying on top of it. For some shops, is speed of delivery faster than some piece (normally testing, but some shops suffer for development on certain platforms also) of the delivery chain can adequately respond to? Do you need to re-evaluate toolchain and delivery timelines?
We are delivering more code, faster than ever, and that is exciting. But quality matters, and anyone who relies on end-users to test is playing a very dangerous game. Find ways to ensure quality without slowing down the process more than necessary. At this point, I honestly would implement runtime-flippable feature flags as a starting point, simply to be more responsive to issues. From there, work to resolve the underlying reasons that bugs are cropping up.
Until then, keep cranking it. Keep working with the business to get quality systems in front of users, and enjoy doing some great work.
As opposed to the traditional software development lifecycle (aka waterfall), today’s Agile and DevOps-based software development process is more outcome-oriented. The unrelenting pace of the changing market landscape has made the hybrid, Agile plus DevOps way of software development absolutely critical in addressing customer demands and staying competitive.
The focus on Agile and DevOps is necessary due to the high failure rates of traditional software development methodologies. These failure rates are mainly attributed to the glitch-infested, low-quality applications churned out by following legacy methodologies.
The failure rates had left many companies to bite the dust and brought the key role played by quality in ensuring customer satisfaction and ROI for the business into sharp focus. Also, the IT landscape is witnessing developments such as mergers and acquisitions, increased focus on risk and compliance, data and analytics, the fast rollout of apps and mobile commerce, among others. These have necessitated the software development process to become increasingly agile, streamlined, collaborative and flexible. The reshaping of customer expectations is letting organizations to go beyond shift-left testing or DevOps testing by transforming the QA process into one driven by quality engineering.
What Is Quality Engineering?
To meet the growing demand for quality software applications in double quick time, a mere tweaking of the software development process is not enough. What is needed is the adoption of software quality engineering process comprising optimal quality assurance, predictive analytics and monitoring of QA elements to reduce glitches to negligible levels. The process would involve a continuous cycle of feedback and quality improvement based on the same. The quality engineering process offers maximum test coverage in the shift-left scheme of things and ensures outcomes like functionality, security, accessibility, usability, performance and reliability of the software application. It complements Agile and DevOps methodologies and ensures their outcomes to remain consistent with the desired business objectives. Quality engineering, instead of identifying the inherent glitches in the SDLC, focuses more on preventing them altogether.
Shift Left and Strengthen Right
Quality engineering services cover the entire spectrum of the SDLC thereby yielding the maximum test coverage and delivering the best quality product. With DevOps testing shifting left, the quality determinants are incorporated right into the product build during the development phase. At the same time, DevOps-based QA focuses on the shift-right approach to ensure the last stage of the SDLC i.e., delivery goes on unhindered and customer feedback is taken in right way and worked upon. The ultimate aim of both shift-left and right is to create and deliver a glitch-free product that meets the requirements of the end-users. Here, since end-users are the ones who ultimately assess quality of a software product based on parameters such as usability, functionality and performance, among others, the feedback mechanism should be responsive enough to understand the issues faced by the end-users and remedy them at the earliest.
Quality Engineering as an Aid
Quality engineering (QE) helps to bridge the gap between development and deployment processes, which can arise due to the lack of agility. It harps on the successful strategy of “test early, test often,” to pre-empt challenges that businesses face when testing a software product. Quality engineering ensures the quality of a software application throughout the SDLC–right from the designing stage to its final delivery. QE can be incorporated into the development process by using methods such as Behavior-Driven Development (BDD), Test-Driven Development (TDD) and Acceptance Test-Driven Development (ATDD) respectively.
Let’s find out what these methods are all about:
BDD: In this software development method, the behavior of both the software and business outcomes are taken into consideration. Here, the right sets of tests are automated, leading to an increased collaboration among developers, testers and other business stakeholders.
TDD: In this software development process, the QA team writes unit level tests before the development stage. This results in getting an early feedback and helps the development team to refactor its requisites.
ATDD: This software development process focuses on writing an acceptance test even before the development phase. This helps the software to meet the acceptance criteria beforehand.
Quality Engineering and DevOps
QE, being an end-to-end testing methodology, can offer optimum solutions to businesses struggling with their legacy IT infrastructure. The solutions in terms of continuous testing and integration as mandated by DevOps, can be arrived at by integrating the QA process with the DevOps pipeline. This results in executing processes such as build, test, deploy and deliver in a seamless manner. QE complements DevOps and Agile by detecting glitches right at the beginning of the development process.
Moreover, QE’s focus on end-to-end testing helps the product to be developed and tested simultaneously. This approach adds teeth to the DevOps’s requirement for continuous integration (CI) and continuous delivery (CD). No wonder, the successful implementation of quality engineering is underpinned on fulfilling the requirements of DevOps and Agile. It further consolidates the benefits achieved from implementing DevOps testing services and Agile testing services.
QE riding on DevOps and Agile can help businesses to scale up their operations without investing much resources and time. By implementing DevOps test automation and Agile test automation using a test automation software such as Selenium, Robitium etc., businesses can boost their SDLC. QE can help businesses to develop and deploy glitch-free software speedily and deliver enhanced customer experience.
Conclusion
The fast-changing competitive business landscape of today requires organizations to streamline their operations, increase productivity and fast forward the delivery of glitch-free software applications. To achieve these objectives, they need to make quality engineering as part of their development and testing cycles. However, QE ends up complementing the Agile and DevOps approach by achieving outcomes such as CI and CD. Furthermore, it leads businesses to get high ROI, better productivity, reduced cost of operations and enhanced customer experience.
Unit testing is like exercise: Companies know it’s good for you and you should do it, but given their druthers, they’d rather avoid the effort and just get to the benefit.
I wish I could say it’s otherwise. After all, making unit testing a standard practice in a company’s software development process is probably the first, best thing a business can do to significantly boost code quality. To go one step further, adopting the best practices of Test Driven Development (TDD) can not only speed up release velocity in the long term but also has the benefit of making code maintenance less costly. For example, if you want to know what a piece of code is supposed to do, all you need to do is review the unit tests. This reduces the need for volumes of support documentation.
It’s almost a no brainer. Yet, there’s still a lot of perception out there that TDD’s dictum to write the test first slows down the development process. The thinking goes that if a developer is going to write a test for the code before writing the code, just save time and write the code anyway and then write the tests only for critical parts of the code.
This might be a sound rebuttal of the TDD process, except for two things. First, can you name a part of your codebase that should not be tested? And second, as those of us out on the terrain have discovered, when release time approaches and the corners start to get cut–and they do get cut–many times the first thing to go is unit testing. More than one development manager has uttered the words “We’ll catch it in QA” when faced with an approaching deadline and an understaffed development team. Sadly, when given the choice between getting so-so code out the door on time and delaying the release in order to create great code, it’s been my experience that in companies that have not embraced unit testing to an acceptable code-coverage threshold, so-so code will win every time. Those who have taken a swig of the TDD Kool-Aid don’t have to make that choice. The tests are already written.
It’s true that all the unit testing in the world will not make poorly engineered code better. But, if the engineering is sound, unit testing works. Still, companies resist, even those who program in languages from which unit testing as a way of life emerged. There are a lot of businesses out there programming Java, C#, Python, etc., that want the benefit of unit testing and yet don’t want to pay for the effort. When it comes to the mainframe community, resistance be as strong as that found in the French Underground in WWII.
Why? Well, when it comes to mainframe development, unit testing is much harder to implement than in the x86 world. There’s the problem of language compatibility. There is still a lot of COBOL out there supporting the 80% of the world’s data that still resides on a mainframe. Unit testing as a way of life is a fairly recent occurrence in the history of computing. There’s a boatload of testing libraries out there for Java, C/C++, .NET, Python. But until recently, there weren’t any unit testing frameworks for COBOL. Also, in traditional mainframe programming, you can have situations in which many developers work on the same body of code. Test isolation can be hard.
Finally, there’s the matter of culture. Remember, the thing that makes unit testing and TDD work is that designing and executing tests are the responsibility of the developer. Pushing testing onto developers is a notion at odds with traditional mainframe sensibilities. In traditional mainframe shops in days of old, developers wrote the code and QA tested it. It’s a process that worked for years as long as those years meant you could release code every six months. Today it’s all about getting code revisions out weekly, if not daily. Still, the Dev/QA segmentation is there in the older companies that are mainframe driven.
But that was then and this is now. I am happy to report it’s a new day in the land of software development of the mainframe. Adopting TDD in general and unit testing in particular as a way of life in mainframe software development is not only possible but it borders on being, as I describe above, a no brainer. The tools and processes are there. Let’s take a look.
Unit Testing Mainframe Code Using IBM Z Open Unit Test
Writing unit tests for the mainframe has had a difficult history. If you’ve started your professional career in software development within the last ten years, you’ve had the pleasure of enjoying the benefit of TDD and CI/CD as a standard way of doing business. Tools such as Jenkins, JUnit, Selenium and the like have always been around. But in the mainframe world, for many, this is all new stuff.
As a result, for many mainframe developers, the notion of writing a unit test let alone integrating one into a CI/CD process means entering a strange new world. In the pre-tool world, writing any test, let alone a unit test, meant spending days doing tedious, laborious work in some sort of jury-rigged testing environment. Or, in some cases, the developer might not be responsible for testing the code at all. Rather, the developer writes the code and gets it working to his or her satisfaction by any means necessary. Then he or she gives the code the old “toss over the wall” to the QA department for testing. What those testing processes are might be known to outsiders, but in many cases they’re not. So, the development process becomes at best a slow, grinding exchange of test results and remedies between groups that have limited insights into the processes and priorities of the other.
The good news is those in the known have come to understand there’s a better way. The better way involves three things. First, making the developer responsible for testing the code he or she writes. Second, removing the cultural, and in many cases, the organizational wall between development and QA. And third, putting tools at the forefront of testing activity.
In other words, the new thinking is to understand testing must be done as early as possible in the software development cycle (hence, the Shift Left Movement), everybody is responsible for the quality of the software and tool adoption is critical in the TDD process.
The construction industry caught onto this notion a long time ago. A bulldozer driven by a competent operator is a lot better for moving earth than a legion of human beings with shovels. The same is true in software development. It’s better to give a single developer a tool to create and apply tests to his or her code than have a legion of other employees sit around with text editors writing one-off units tests for their code and applying them by hand.
Which brings us to IBM Unit Test.
IBM Z Open Unit Test
IBM Z Open Unit Test is a tool designed to make writing and executing unit tests on the mainframe fast and efficient. Also, IBM Z Open Unit Test facilitates integrating unit tests directly into a company’s CI/CD processes under a tool such as Jenkins.
The tools support automated unit testing for batch and CICS COBOL programs. Also, IBM Z Open Unit Test provides recording and playback functionality for CICS COBOL programs, as shown in Figure 1. This tooling allows developers to easily record external calls to CICS and import that recorded data for modification in playback. The ability to record, modify and playback test data at execution time means developers can run their automated unit tests as part of a build (or, as a batch job). There is no need to deploy to CICS, as it is now stubbed out.
IBM Z Open Unit Test aligns with TDD methodology in that the tool allows developers to record, view and modify the input and expected output as assertions, within an easy to use interface. This can be done before making a change to a program, and developers can understand what kind of data is flowing in and out.
Figure 1: IBM Z Open Unit Test has data recording and playback capabilities for Batch and CICS COBOL programs.
As mentioned above, IBM Z Open Unit Test can be run from within an IDE or executed as part of the enterprise’s CI/CD process. Also, the product supports test debugging and importing tests written in mainstream formats such as JUnit and SonarQube. These are good and important features. But the real kicker is support for code coverage.
Why Code Coverage Counts, Even for Mainframe Testing
Code coverage is the elephant in the room in terms of testing administration and management, particularly around unit testing. One of the key principles of test driven development is that every line of code that gets written gets tested. Yet this is the dream rather than the practice. I’ve been in more than a few companies that say they support TDD, yet when asked if they test every line of code written, which translates into 100% code coverage, the answer usually involves something along the lines of “we’re getting there.”
Many companies are tickled pink that their developers are doing any unit tests at all. In such companies, asking a developer to get to 100% coverage is akin to asking him or her to go to the moon and back during lunch.
Some companies that have matured a bit set the coverage standard at 80%. This means that as long as eight out of every 10 lines of code written get touched by a unit test, things are just fine. Or are they? What if the untested two lines of code is a short else clause in an if-then statement that eats memory and blocks I/O? Are these the two lines of code you want to leave untouched in a test? Remember this: Typically it’s not the tested code that wreaks havoc, it’s the code that’s untested.
Thus, the mission critical importance of 100% code coverage.
Mainframe development has lagged a bit embracing the imperative to ensure 100% code coverage. This is mostly due to the culture that went with legacy software development practices and the immaturity of tools at the time. The good news is both culture and tools are evolving.
Getting back to IBM Z Open Unit Test, the tool not only provides test coverage reporting, which is critical for providing the metrics necessary to get to 100% coverage, but also a feature that supports negative testing.
Negative testing is also known as an unhappy path testing. It’s an important, yet often overlooked aspect of unit testing. Just getting a piece of code to work under the right conditions is hard enough work. To subject that code to every bad condition (unhappy path) that can happen and make sure it still works, requires significantly more work. Thus, when crunch time comes, and it always comes, typically the developer will let negative, unhappy path, testing go. It’s not malicious intent. It’s about meeting a deadline.
Fortunately, this dynamic is a known problem in the industry. IBM Z Open Unit Test addresses the problem by providing a feature that automatically generates negative tests. This is a big deal. In fact, if this is all the tool did, it would be a boon to the practice of TDD on the mainframe. Writing negative tests is a tedious, laborious activity. Using a tool such as IBM Unit Test not only saves time but helps achieve the ultimate goal: 100% code coverage.
Putting It All Together
Unit testing is an essential part of the practice of TDD. While TDD has taken hold throughout the x86 based distributed computing, there has been a lag to embrace the discipline in mainframe computing.
Mainframe computing is becoming very cool again, if for no other reasons that it presently controls 80% of the world’s data and in many ways was the cloud before the cloud. In the past, TDD has been hard to implement in the mainframe world due to a lack of tools to support the discipline. Fortunately, with modern tools such as IBM Z Open Unit Test, the benefits of TDD can be easily enjoyed in the mainframe development community.
It sounds like an oxymoron: Create test cases before coding. In the process of writing and using test cases before writing functionality, a developer can produce higher-quality software. That is the concept behind test-driven development (TDD), and it works.
TDD Snapshot
TDD has been around since at least 1999, and it is part of the emerging test-first approach to development associated with extreme programming. In 2003, American software developer Kent Beck “rediscovered” TDD and applied it as a way to create simple designs and inspire developer confidence. Fast forward to today’s agile development world, TDD is a software development process using a very short feedback loop in which developers do the following:
Create a test that will fail immediately.
Write the necessary code to pass the test as quickly as possible.
Refactor the code written in step two, using the created test as a reference.
TDD Benefits
So why might a long-in-the-tooth software development approach have tangible benefits in an agile development world? Because put in its simplest terms, TDD is a way to develop highly usable software.
Following the TDD process, a developer must focus on the test cases before actually coding anything. That meansa developer is thinking more about the use of the software and the design of user interfaces to achieve that. As a result, the developer is more interested in the interface than in the implementation—and that leads to more usable software.
There are several other tangible benefits for taking a TDD approach, including:
Creating a metric for the code quality that can be easy and quick.
Allowing for a quick visualization to determine if the code base has any functional issues.
Living documentation of functionality with new code written.
Allowing for safe refactors of code, whether based on attempts to improve code quality or a changed requirement.
That final point is worth dwelling on for a moment. The TDD approach requires that the evolving code base is cleaned up frequently so that new tests and code are easy to introduce. That usually means that code moves from its current place to where it more logically belongs in the software. That has the effect of reducing any extraneous duplication of code and enforces a strong discipline around objects, classes, modules, etc. In this way, the overall maintainability of the software increases incrementally.
The improvement in readability and maintainability will pay huge dividends during the expected life cycle of the software. Following a TDD methodology requires developers to focus on writing smaller units of testable code. By following the TDD methodology, this leads to more modularized, flexible and extensible software.
Where Can TDD Fit?
A TDD approach is appropriate for both new greenfield software and legacy systems. For a development team that has to deal with existing legacy software, the key is to start small and begin with working on bug fixes. A good practice is that for each bug reported, create a test that addresses the broken bug and then fix the functionality. After several iterations of this, the development team has created a repeatable working test to address bug fixes. When applying this methodology to new software applications, make a point of understanding the testing tools used for the technology stack.
For example, when working in an Angular application that generally uses the Jasmine testing framework for unit testing, and when using the Angular CLI for creation, unit tests are created alongside modules of code. Using a TDD methodology, the approach would be to:
Determine a portion of the functionality to be created with this component.
Create a unit test that will fail immediately targeting this portion of functionality.
Run the test runner to confirm the failing test (it may be useful here to leave the test runner on to run after each source file save, which speeds the process).
Write code in the Angular component that will make the written test pass.
Make any refactoring changes for the Angular component after confirming passing, using the test as a guideline to ensure code refactoring does not break functionality.
Measuring Testability with Code Coverage
Another essential consideration when improving testability in code is to use a code coverage tool. Code coverage is a metric to show the percentage of code that has a unit test written for it. Angular applications use Istanbul to calculate code coverage through the application. Running a single run of code coverage in an existing project gives the following output:
The output provided by Istanbul gives a metric on the overall test coverage and the areas of code that need improvement in testing. Code coverage can be useful in a few ways:
Provides an idea of the overall testability, allowing for a threshold to ensure the total software testability does not drop past a certain point.
Identifies areas in the code base with poor testing, making them opportunities for refactoring.
However, as effective as code coverage can sound, it’s important to understand that it’s just a metric. Writing good unit tests is a matter of following what the code will do, and metrics such as these should not drive significant decisions.
Considerations When Using TDD
It’s important to note that TDD does not solve all problems. There are many different types of testing required to create a fully comprehensive testing strategy, including acceptance testing. In TDD, the focus is on a single unit of code at a time. A sophisticated software application may have many thousands of units of code and their corresponding tests. That’s why it’s critical to make sure that testing quality stays high when following the TDD methodology. Tests cannot become something bypassed in the pursuit of more functionality or expediency. Avoiding testing generates the risk of test creation becoming a hindrance for developers.
For example, ignoring failing tests makes it difficult to determine the actual state of the application. It’s also important to have buy-in for a TDD approach form all the teams involved in the effort. Buy-in is particularly true on the business side. Time must be spent up front discussing the nature and benefits of a TDD approach, and the belief that using TDD will improve the resulting software. Otherwise, business management sees writing tests as an activity that doesn’t contribute to the bottom line.
Conclusion
TDD emphasizes the importance of an effective and sustainable testing approach. TDD also contributes directly to the overall quality of software. It’s a truism for small or large system development that often goes missing in the day-to-day hustle to get new functionality into production. Quality software gets built when there’s an acknowledgment that quality test code should receive the same amount of attention and resources as quality production code, as they are equally essential in development.
Software testing and QA is sometimes given low priority during development. The idea of treating software testers as adversaries by the developers seems old-fashioned and is short-sighted. It’s hard to believe that such an attitude can still exist, and yet it still persists, even in a modern DevOps team. But a DevOps-focused software team benefits when it considers QA/testing as an integral part of developing high-quality applications.
In a business setting, the hidden costs of buggy or unsecure software are lost productivity, revenue and extra work by the development team (and even extra work by users, if the software isn’t running right). QA/testing helps test for system flaws, vulnerabilities and security issues, so why wouldn’t a DevOps team be supportive?
QA/testing can help an enterprise avoid big mistakes. In fact, the DevOps Institute notes, “With the advent of DevOps and the movement to break down silos between developers and operations, it becomes critically important that all members of an IT team—regardless of what tools they use, or role they play—understand the essentials of testing.” And, according to Puppet, a standard for automating the delivery and operation of software, high-performing enterprises build quality testing into each stage of their software development process.
Make QA/Testing a Priority
Whether your DevOps-focused developers are handling the QA/testing themselves or passing it along to QA/testing specialists, it’s something that must be done. Ideally, developers themselves would handle QA/testing in an integrated DevOps environment, and that’s why some developers see it as an unwelcome chore. It’s not something they’re usually trained in, and they might struggle with it or see it as a burden. That could be the reason why some DevOps teams hit a “software testing bottleneck.” As data from a Vanson Bourne survey commissioned by Appvance shows, 73 percent of 200 enterprise IT decision makers in the United States have adopted DevOps processes, but 57 percent of those adoptees experienced a bottleneck during software testing that impacts how quickly new and updated software can be released.
On the flip side, the developer’s company may have QA/testing professionals in-house, so the developers are hands-off in that area. Needless to say, the relationship between developer and the QA/testing pro could get a little rocky. Just think about it: The role of the QA/testing specialist is to find problems and point out issues with the developer’s code. Or, perhaps the developer or engineering staff is lagging behind schedule, so the team inevitably needs QA/testing to rush its processes, further creating a tense or adversarial relationship between the two groups.
At one of my startups a while back, we worked for months on our first software release of a web application for an online marketplace. Late on a Wednesday night, we let the CEO of our company try it out—and the app immediately crashed. “Gee,” I said, “It was working on my computer.” Of course, it really wasn’t. We had synchronization problems, which are common in multi-threaded applications that access the same information for multiple users. And we made the mistake of letting the CEO be the first simultaneous user to try it out (creating a bad impression), instead of making organized QA/testing a priority at our company.
QA/testing in a DevOps environment can help your team avoid mistakes, so passion for app functionality doesn’t supersede quality and robust bug-free software. In the end, what you need from your team is passion for both functionality and quality/testing. Otherwise, you’ll deliver a faulty product and lose customers or users—unnecessarily—because of a bad first impression by your software.
Two Approaches for Integrating QA/Testing with DevOps
When you’re integrating QA/testing into your DevOps environment, there are two different approaches or procedures to ponder. First is the more traditional test of the full system, sometimes called integration testing. The second is test driven development (TDD), which is popular in Agile and Scrum methodologies. The main difference between the two is in how software code is tested. During integration testing, developers review and test the code. Then, for critical facets of the software, a peer review of code is completed. Afterward, testers create scripts and deploy an array of integration and system tests. These could include scalability testing, load testing, stress testing and very importantly, security testing. Code vulnerability, SQL injections and malicious code would be things to think about.
Under TDD, the order in which testing appears in the cycle is reversed. Each developer first writes “unit tests” and then writes the code. This is done iteratively in such a way that every discrete section of code is automatically tested. When new code is sent to the integration server, the tests go with it. The server itself runs unit tests for all the code that developers deposit in the server. On top of this, testers design system and integration tests to run automatically by the integration server. Finally, testers perform any manual tests that are impossible to automate. In this way, the integrity and quality of the code is constantly guaranteed.
Academic studies and field experience suggest that TDD increases development time by approximately 15 percent … but hold on. Experience also shows that total cost of ownership of TDD-developed software is lower than software using integration testing only. It’s worth noting that by using TDD and integration testing, a company can deploy software of world-class quality that exhibits less than 0.3 errors per 1,000 lines of code during 18 months of use.
QA/Testing: A Partner in Your Software’s Success
If you’re hesitant about integrating QA/testing into your DevOps team and instead tempted to outsource it, you may want to push pause. It’s actually quite challenging if you do development in-house and expect to simply outsource what you think is independent QA with cheap testers. Outsourcing QA/testing and assuming that piece is done is not really outsourcing—it’s staff augmentation at a distance and it usually doesn’t work. You don’t want a “siloed team” to handle a critical component such as QA/testing. Instead, outsource QA/testing and bring your in-house and outsourced teams together around a common DevOps culture and mindset.
DevOps experts Bob Aiello and Leslie Sachs, in an IBM report, say quality software “… requires not only a shift in behavior, but a dramatic shift in the organizational culture as well.” So whether you outsource or have an in-house squad, your developers and the entire team should take ownership of QA/testing, and remain open to new ways to test code efficiently.
Ultimately, QA/testing should not be viewed as a time-consuming, disruptive chore by anyone on your team or in your enterprise. Think of it as a partner in your software’s success—not an adversary.
We’ve gotten a lot of good out of agile development methodologies. I’m never a “One True Way” kind of person, but agile has a ton of benefits that development needed (and Ops increasingly needs). Extending that to the Dev side of DevOps is painless, since much of agile is wrapped in DevOps clothes these days.
From my perspective, the most impactful bit to come from Agile was test driven development. By writing the test to fulfill the requirement, then writing the code to pass the test, our small team (I was on a team of developers with no Ops people supporting us—or more accurately, we were the Ops team when I first used TDD) needed the assistance it provided. Without dedicated QA staff, we were able to perform as rigorous a test suite as our changes warranted, simply by running the tests that we’d used for development.
Of course, this is old news to some of you, TDD has largely been consumed in the larger Agile/DevOps movements and you’re at it daily. But the shift to having developers do more of their own testing is something that I’m still pretty iffy about, to be honest.
Devs Aren’t Necessarily Testers
While it can work, devs certainly can understand the weak points in their code, I would argue that the person writing the code is not the correct person to write the tests for it. Particularly as you get into more the integration phase of testing, and the complexity of code inter-relationships heats up. Some developers will do well at it. My experience is, many will not. Why? Because if they don’t know of a weakness in their solution when writing the solution, they won’t know of the weakness when writing the test case. And it is a truly rare shop that has 100 percent code coverage for testing, so those weaknesses could end up being time bombs.
Simply put, QA needs to work with Devs to write the tests. Two heads are indeed better than one when looking for faults with solution designs. But the reality of development often doesn’t allow for this type of cooperation. When you’re on a sprint and this one bit is slated for half a day, setting aside time to get together can be difficult for a problem the developer has already written the test for and solved in their head. After all, the whole point of Agile and DevOps was to speed delivery, and making the simple more complex isn’t generally part of that equation.
Have a Plan, and Use it
I know of a couple of shops where integration testing is pretty much an afterthought. One has a QA team that fights to make certain it is done and done well (though still as an afterthought), rather than cobbled together as a PoC (I was thinking Proof of Concept; take either use of that acronym, both are accurate) not too long before release. The others muddle through, figuring out what doesn’t work in fits and starts (too often in production), rather than following a stream. The result is, unsurprisingly, missed deadlines and ever-extending sprints.
For all of the good that DevOps in general, and CI/CD in particular have brought us, there are a few areas—one of them being testing—where speed and/or quantity has replaced quality. We run tests a LOT. But are they the right tests? For unit tests in a TDD environment, yes—for lines that have test coverage. Outside of that narrow space, I think the answers are much more iffy, and it depends upon a lot of things.
The best hedge is to have QA involved in writing the tests for TDD environments, even for small bits. It’s not an overhead if it shortens the overall timeline to delivery; even if it slows a piece here or there, it’s the timeline to finished product that really matters, not the timeline of a given piece of it.
If you don’t have a QA team—as many shops do not—I recommend devs doing it for each other. That’s not to say that QA is not needed, but it does utilize the “Two heads are better than one” mantra, and keeps the ideas flowing. Otherwise, the Ops members of a DevOps team are great for that. Historically, production woes fell on them, so no doubt they’ve honed their skills in detecting problem areas.
Don’t Be That Person
No matter how you go about it, do not go light on the testing. Do TDD, and keep it up. Make every change request start with test modification, and then fix what fails. Document the tests, since they’re code also, and stressing what is being tested and why is as important as the code talking about what it’s doing and why.
After all, you don’t want to end up with the difference between ‘==’ and ‘>=’ placing you in the headlines for a week, mostly with bad news. That difference is an obvious TDD test case if ever one was written; don’t be that person.
Continuous testing works, test driven development works. You just have to think about it when implementing them, and stick to a plan that tests beginning to end. It’s about quality code quicker, not about any code quicker. Note that I’ve been focused on the process end of testing, not the smoke/load/regression end. Those are important too, but their issues are different, so I opted to throw them into a separate blog.
So go forth, iterate quickly, but test thoroughly. I doubt most of us will achieve 100 percent code coverage anytime soon, let alone 100 percent integration nested coverage … But test what’s important, and bring what’s important to test.
Editor’s Note: This post was updated Aug. 4, 2018, to reflect an updated slide presentation at the end of the article.
Infrastructure as Code is a truly powerful concept whose significance is best described with the words of Adam Jacob, CTO at Chef, in Web Operations: Keeping the Data On Time: “Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare metal resources.” And so it’s not for nothing that treating infrastructure as code is regarded a key principle for reducing cycle time and maximizing flow in Continuous Delivery and DevOps, respectively. Even if your plans are less bold as you read this article, wouldn’t you agree that consequent validation of what you automate on the infrastructure level — long before it goes into production — would be extremely powerful?
So, bear with me for a while and please don’t rush head over heels into automation by simply picking one of the established infrastructure automation tools such as Chef or Puppet, getting everyone trained in Ruby, and blindly start to “automate all the things”. If that’s how you do it, then you are most probably doing it wrong. In You’re Doing DevOps Wrong. Automation in the Enterprise., author Alan Sharp-Paul suggests that, before doing any automation, you first have to understand precisely the requirements and how to validate them to gauge success and maintain quality of your automations over time:
Are the configuration files in place? Do they contain the right settings?
Is this port open? Is that port closed?
Why Agile Infrastructure?
Honestly, you and I wouldn’t expect anyone to come up with reliable software merely by choosing an arbitrary programming language and a nifty IDE without having some sort of engineering process in place, right? Likewise, the (automated) provisioning of a solid production environment demands a no less methodical approach. Luckily, Infrastructure as Code enables us to treat the provisioning of infrastructure as well as automating deployments and configuration as an Agile engineering discipline. Although the concepts of Agile Infrastructure are not entirely new, testing against live infrastructure has only recently been facilitated with the consolidation of tools such as Ansible, Chef, Puppet, Docker, Vagrant and Serverspec under the hood of Test Kitchen (see more on that below). Here are some of the key implications of applying Agile software development practices to the infrastructure level:
Alignment of Devs and Ops
Agile software development teams are driven by customer demand and aim to produce outcome in short, regular intervals. Enabling IT Operations and Infrastructure departments to work in small iterations and react to changes quickly allows them to align with and effectively become driven by software development. The outcome of these joint efforts will be both working and deployable software at the end of each iteration.
Fail Fast: Feedback Loops
Executing infrastructure tests as part of your Continuous Delivery build pipeline (before an application is deployed and long-running tests are executed) establishes another important feedback loop that enables you to “fail fast” and get the right people involved. The underlying principle is simple: by expressing your intentions twice — once in the code and once in a test — you’ll know you’ve caught a bug whenever these assumptions don’t match.
Regression Safety
Believe it or not: whether you are using Ansible, Chef, Puppet or any other — you’re running on code! Their respective issue lists are full of unresolved issues of various severities with new issues coming in regularly. Having a sufficient safety net of tests for your infrastructure in place allows you to quickly identify bugs as they creep in (from either side) and protects you against unwanted regressions. After all, a bug in your infrastructure can be very hard to track down and can be much more severe than a bug in your application.
Refactoring
Code refactoring becomes a breeze when you have tests in place that immediately alert you when you broke the desired state of your infrastructure (and all you wanted was to make your code simpler and more maintainable). This is invaluable when you want to recompose your existing infrastructure from 3rd-party components offered on public hubs such as Ansible Galaxy, Chef Supermarket or Puppet Forge, and especially when you are migrating away from your existing automation stack.
The Red, Green, Refactor Cycle of Test-Driven Development
Test-Driven Development (TDD) is a popular software development process that involves writing code in short, repeating development cycles with the goal to enforce good design and enhance confidence. While it is certainly no silver bullet and has been controversially discussed in the past, the Red, Green, Refactor Cycle, which is at the heart of TDD and whose principles I want to quickly summarize here, hopefully gives you some inspiration on how to incrementally build up your infrastructure:
Red: Understand the requirements and the state you want to implement. Then think about how to capture this state in a test. Implement the test, then run your entire test suite and see the added test fail — and if it doesn’t, something is really wrong.
Green: Implement just enough functionality to make the test pass. Don’t care much about readability, simplicity and design for now — just make it work. Run your tests again and watch all tests pass. If they don’t, either fix your test or your implementation, or take a step back (revert to a point in your code that’s known to work) and start over with a smaller increment.
Refactor: Now that your tests pass, improve your implementation and make sure, by running your tests again, that you didn’t break anything. Once you are done refactoring, start the cycle over again.
Keep in mind that each repetition of the cycle refers to a small increment. In terms of infrastructure, such an increment could be the installation of a user, a package or the placement of a configuration setting. When you arrive in a spot where you broke something, you can easily revert to a point in your code that’s known to work. Leveraging such practices really makes the difference between Infrastructure Automation and Infrastructure as Code, as described in Infrastructure as Code – Automation Is Not Enough by Keith Morris.
Test Kitchen, Terraform, AWS and awspec
Now, how to get there? Test Kitchen is an easily extensible test harness that allows you to test your code written in Ansible, Chef, Puppet (and others) on various cloud providers, virtualization providers and operating systems with tests written in a variety of test frameworks.
In the following presentation, I give hands-on examples for you to quickly dive into the topic of using Test Kitchen with awspec as a framework to test Terraform deployments on AWS:
Supporting Swift Cyber Business Ramp Ups Around the World with Translation and Localization Sped by CD
Smartling technology enables the business to quickly translate content in websites, apps, and documents into any language in the world. “We streamline all the non-linguistic work associated with the translation,” says Andrey Akselrod, CTO / Co-Founder, Smartling, a translation management platform provider for localization anywhere, globally. Using the Smartling platform, companies upload content, making it instantly accessible by a Smartling translator partner or a translator that the company acquires independently.
I viewed the Smartling demo and I can tell you that the platform offers many innate, seamless, intuitive abilities. Briefly, it allows the translator to largely automate translation, localization, and content re-use and editing against a backdrop of how it will appear on a website, in an app, and in electronic documents. The platform includes workflow automations to speed translation along to the next stage in the project. “This allows for a reduction in complexity, cost, and time for the translation process, without sacrificing the high-quality translation that only a human can do,” says Akselrod.
The TDD Testing Secret Sauce
Smartling developers first write automated regression tests for the code they are about to create. Then they confidently code their software knowing how the tests will challenge the code because they wrote the tests themselves AND did so before they wrote the new feature or innovation. This process accelerates development because the programmers write the code knowing what it takes to ensure that it won’t break anything. This enables Smartling to deploy live code as many as ten times per day, according to Akselrod.
Test-Driven Development (TDD) saves on refactoring and validates the tests. Smartling discovered this when certain test and development issues disappeared after they moved to TDD. When you code the tests after writing the software that you need to test, you may have to refactor your code so that it is suitable for testing. “You find yourself in a position of doing refactoring without tests to validate that the code still works,” says Akselrod. There’s an old carpenter’s saying that you should “measure twice and cut once” to make sure you don’t waste what you cut or have to cut again. But this sounds like the reverse, expecting the coder to cut (code) twice and then measure. The absurdity in that is self-evident.
Developers who must create tests after they innovate their software don’t test whether the tests could fail. “This results in low-quality tests. We’ve discovered this the hard way—we had pretty good test coverage, but a number of bugs in the code base led us to discover that our tests were failing,” says Akselrod. TDD resolved all this, saving time, energy, and investment, and satisfying customers.
Though these automated tests run much faster than manual regression tests do, Smartling gauges test performance over time and refines testing efficiencies to ensure that tests keep pace with accelerating deployments. “We distribute test execution among multiple servers to make sure that the tests run as quickly as possible,” says Akselrod.
Can Get Enough of That TDD
Developers find themselves obsessed with the TDD development process for the prompt props they receive as they deploy code directly rather than wait to see it run successfully. “TDD makes for quick customer feedback. Customers are thrilled because we react to their requests quickly. This fosters an exciting and fulfilling conversation between developers and customers,” adds Akselrod.
TDD’s competitive advantage wets the appetite for market penetration. TDD moves Smartling through the OODA decision cycle at an accelerated rate. “Modern software development cycles through an OODA loop such that developers Observe, Orient, Decide, and Act. If your OODA loop is faster than your competitor’s is, you win,” says Akselrod. TDD enables Smartling to rapidly deploy, respond to the customer base, and instantiate innovations, expediting aggressive market growth.
Results
Customers are seizing the benefits that savvy platform provider Smartling serves. As this throng of growing businesses increasingly hitches its wagons to Smartling, Smartling in turn sees three-billion page views per month in traffic flowing through its Global Delivery Network of translated dynamic pages, according to Akselrod. No matter how you slice it, three-billion is a number you simply cannot explain away.
It’s almost a no brainer. Yet, there’s still a lot of perception out there that TDD’s dictum to write the test first slows down the development process. The thinking goes that if a developer is going to write a test for the code before writing the code, just save time and write the code anyway and then write the tests only for critical parts of the code.
TDD Benefits
The output provided by Istanbul gives a metric on the overall test coverage and the areas of code that need improvement in testing. Code coverage can be useful in a few ways:
Steve Mezak is founder and CEO of
