We have reached an interesting point in the automation of IT in general and testing specifically. Our tooling now allows us to generate a list of all of the possible permutations of a given test set. This is interesting because we can generate all of the possible permutations of a given test. We can also automatically run all of the tests so generated. And we can automatically filter those results, only raising what needs greater attention to a human for evaluation.
And that’s not necessarily a good thing.
Nothing is free. Nothing. In the case of pervasive testing that goes beyond the wildest dreams of the most stringent QA manager, it costs a lot. The CPU cycles and storage space to create, execute, evaluate and notify the volume of tests we are talking about is exorbitant and unlikely to be adopted by your average organization. Truth is, it is unlikely to be adopted by any organization, but there is always some corner case, so I’m being a little vague to keep the conversation focused on the technical issues.
Testing vendors are already working on filtering tools to test only what has changed in the product, but this implies that, at a minimum, this level of testing should not be applied when testing is most needed–during initial development. I might be bringing my baggage along, but I personally prefer focused and curated testing to a firehose that is always on. It doesn’t take much imagination to see a scenario where a full-on automated test filtering tool is off just a little bit, and we get flooded with alerts that are not necessarily false positives but that include information we don’t want to see and consume man-hours needlessly.
Like everything in our recent past, this process will be rolled out too early and then will get tweaked and fixed on the fly, with you and I paying for the privilege of being beta testers, in fact, if not in name. So, I strongly advise caution in adopting new testing automation. I’m super excited by the possibility that all application testing can get the attention it deserves. Still, I am not at all interested in being the experiment on the way to that panacea. So watch the space, listen to other tech folks, and pay attention to how it is working out for them. We have had a tiny percentage of test coverage for so long that waiting for the early adopters to work out the kinks is not going to cause too much additional risk.
As a couple of glaring scenarios have recently reiterated, high-tech vendors are more than happy to do to you in the age of recurring revenue rather than for you. Use caution; make sure that any test automation tool you use aligns with your corporate goals and that the ones you hold subscriptions to are not rolling out massive test automation behind your back. Know your product, and pay attention to any changes in release or license—now more than ever.
And keep rocking it. Soon, we will have effective 100% or near 100% coverage in testing; just don’t rush it. Let the work mature and keep your eyes focused on serving your customers the best product you can without adopting bleeding-edge ideas that might create liability via technology or implementation immaturity. Your customers should thank you, but far too many won’t, so I will for them. Be they internal or external, those who use your systems owe you big thanks for the quality and massive uptime they take for granted.