If we do not change the way we teach, 30 years from now we will be in trouble. Because the way we teach, the thing we teach our kids are the things from the past 200 years. It is knowledge based. And, we cannot teach out kids to compete with machines, they [machines] are smarter.

We have to teach something unique so that a machine can never catch up with us.

—Jack Ma, CEO Alibaba

tl;dr synopsis Given the complexity of modern DevOps, technical certification is becoming more granular. Also, its use is exploding as a requirement for employment. Most technical certification is awarded by passing a test composed of multiple choice questions. Passing a test with multiple choice questions requires the test-taker to absorb a significant volume of knowledge for a short period of time. This acquired knowledge does not need to be retained. Hence, the certification can only attest to the bearer’s knowledge of a technology at the time the test was taken. And, essentially the test measures only the bearer’s ability to take the certification examine. Finally, testing by multiple choice does not ensure the certification bearer’s qualification in terms of required experience or higher-level thinking capabilities. Technical certification is valuable in the current commercial landscape. However, at some point, as machine intelligence is able to perform more of the activities presently done by humans, the format and methods by which certification testing is administered will need to change. Future certification testing methods will need to put greater emphasis on analysis and creative problem-solving rather than the current focus on knowledge acquisition and rote application. Also, what and how we teach humans will need to be unique, beyond the capabilities of artificial intelligence.

Over the last 10 years, DevOps has become a much more complicated landscape. Regardless of the service provider of choice—Amazon Web Service, Microsoft Azure, Google Cloud, OpenStack, IBM, Joe’s Homegrown Cloud Service—you gotta know a lot to play.

At a high level, service providers offer products that are surprisingly similar. For example, they all provide some sort of machine virtualization, storage services, database services, queuing technology, authorization service, provisioning technology and API management. But the way you manipulate the conceptual knobs and switches to get any of these products to work can vary dramatically between providers. There’s a lot of minutiae involved. Absorbing the amount of detail required to be productive in a provider’s product line takes time. And, ensuring that someone can actually work competently with a service provider’s products is a crapshoot. For example, a deployment engineer may understand the concepts behind dynamic provisioning, but getting a Kubernetes orchestration to work in Google Cloud is where the rubber meets the road. Doing is always much more difficult. As I said, there’s a lot of detail involved.

So, how do companies ensure that the people they hire to fiddle with their multimillion dollar, mission-critical technology stack will deliver the goods without blowing things up? Most have an extensive technical interview process accompanied by a stringent reference check. Some just hope. Others require certification.

The Rise of Micro Certification

The notion of certifying that a person has the skills and experience required to be competent in a profession has been around since 1885, if not before. That was the year Massachusetts implemented the first written bar exam. Before that time, all a person needed to do lawyering was to get a note from a court saying the person could be a lawyer. That’s how it went for Abe Lincoln. Eventually the profession matured. Professional qualification was determined through a test-based approach. Early bar exams used essay questions. In 1972, with the addition of the Multistate Bar Exam, multiple choice questions were included.

Going to multiple choice questions may seem like a trivial change, but it’s not. Whereas a knowledgeable professional is required to grade an essay question, multiple choice answers can be evaluated by anybody with the answer key. Hence, the popularity of multiple choice questions on the SAT. It’s the most efficient way to grade millions of high school test-takers each year. Evaluation is fast, cheap and a machine can do it.

As I mentioned at the beginning of this piece, IT in general and DevOps in particular has become a place of increasingly granular complexity. There are a lot of technologies on the landscape. Each of those technologies has a significant learning curve to be overcome to achieve mastery. The conventional wisdom is that the best way to determine mastery is to test a person’s knowledge of the given technology and then measure the results of the test—high score, good; low score, bad. As with early testing practices, the fastest, cheapest way to make an evaluation is to administer a test made up of questions with multiple choice answers. This is sort of like the SATs, except most technical certification is geared to a particular vendor’s product line, not a general body of knowledge. (You can view a sample of test for AWS Certification as a DevOps Engineer Professional here).

This type of professional testing paradigm is exploding. It’s infiltrating every corner of the technology landscape. And, more employers are requiring certification as conditions of employment, both for prospective employees and those already on staff. This would be OK, except for one thing: After you scrape away all the preconceptions and assumptions about the value of test-based certification, when it comes down to the bare-bones validity of test measurement, the only thing a test really measures is the ability to take the test. For example, a deployment engineer might be sitting for a certification test after having been up all night working to get the company back online after a system failure. The taker, who is exhausted, ends up failing miserably. Yet, some slacker just out of high school who has never seen in inside of a data center can take the test using stolen answers and pass with flying colors. Tell me, please: Who is really the most qualified professional?

Testing by Regurgitation

Certification is not the problem. After all, I really do feel more comfortable knowing that the surgeon about to open up my wrist to alleviate my carpal tunnel syndrome is certified by the American Board of Surgery. Rather, the problem is the way that certification testing is conducted. In my opinion, a certification testing process that emphasizes the multiple choice question-and-answer format is flawed. Even when questions are answered correctly, the test format does not really verify the competence of the taker, only that some facts are known. It’s just testing by regurgitation.

Such testing requires the taker to absorb a lot of knowledge and then dump it out according to the test requirement at hand. There is no guarantee of long-term retention. It’s the age-old process of cramming for the test. Once the “time’s up!” bell rings, the chances are that the test taker will forget most of what she absorbed. Those with photographic memories might remember it all, but for most of us mere mortals, it’s a one-shot deal.

Few people in tech work using the cram-it-in method. Most of us just use this thing called the internet to get the answers we need, when we need them. It’s what Bill Gates calls “information at your fingertips.” Yet, we still test for the knowledge in your head at the moment.

There is a better way.

A Better Way

Certification is good provided it ensures that the bearer knows what he or she is doing. When certification is about nothing more than checking the right checkbox on an online test, there’s a problem. Put a monkey in front of a multiple choice test for eternity and eventually the animal will score 100 percent.

So what do we do?  We do the following:

Support Open Book Testing

If the purpose of the test is to ensure that a person has a certain body of knowledge at the time he or she takes the test, make the body of knowledge required available at test time. It’s called open book testing. That’s how it works in real life. For example, you need a NodeJS package to provide mock objects. Do you rely upon your existing knowledge about the packages you know? Great, but what if you don’t know? Does work come to a grinding halt? No. You go to NPM to find one that meets your need. That’s how it is in real life. A competent professional is expected to have a baseline, high-level knowledge of his or her professional domain. For the day-to-day details, a pro has the ability to find accurate answers to questions quickly.

Instead of having a certification testing process based on cramming, maybe we should eat the dog food we make in real life and go open book. There is little downside to researching what you need, when you need it. Besides, the test question created a week ago might be best answered by a new technology created today.

Put the Emphasis on Essay Questions

The actual mechanics of testing vary according to the level of cognition being examined. I wrote an article about this a few years back. The best way to test higher-order thinking—analysis, synthesis, evaluation—is by using essay questions. After all, what is a Ph.D. dissertation if not a very big essay question? Essays require a level of thinking that is hard to capture using lower-order test methods such as multiple choice. Essays reveal not only what the test taker thinks, but also how the test taker thinks. There’s a reason why your high school math teacher wanted you to show your work when taking an algebra test. Sometimes how you get the answer is just as valuable as the answer itself.

A good certification test will provide a question format that allows test takers to illustrate their thinking along with the way they arrived at an answer. Thus, the need for a bias toward essay questions. Again, how you think is just as important as what you think.

Put a Human Back in the Examination Process

I’ll share an intimacy with you: I’m not very good at answering multiple choice questions. When confronted with one, I spend a lot of time trying to figure out what the question is really asking. You’d be surprised at the number of test questions out there that are vague and confusing.

This was a real problem for me back when it was time to go to college and do the SAT dance. Fortunately, my alma mater focused more on my essays than my SAT score. Also, a lot the exams during my collegiate years were administered as essays or conducted as interviews with a professor. Later, when it was my time to be the Guy Awarding the Grade, my process was to evaluate students by interviewing them about one or many of the projects they did. I based my grade on the quality of the project itself and the thinking that went into the work. I tested my students by talking to them and reviewing their work. I like to think that I got a pretty accurate idea of a person’s competence, more than a multiple choice test could ever provide.

Providing the opportunity for human interaction in the examination process can go a long way to providing the information required for making an accurate evaluation of a person’s competence.

Require Testimony of a Candidate’s Experience, Character and Competence

Think about this: It’s possible to be an AWS Certified Solutions Architect Professional without ever having done a day’s worth of paid, real-world work. All you need to have done is pass the certification exam. Yes, of course passing the exam is meaningful. It’s a grueling test. At the least, the test might prove that should be able to work the conceptual dials and switches that go with a given AWS service. But, as far as having the real-world experience, competence and character to play a critical path role in a mission-critical enterprise, some other instrument of certification is required. This instrument is sometimes called a reference—or, as the Wizard said to the Tin Man in “The Wizard of Oz,” a testimonial.

As you read above, back in Abe Lincoln’s day, a letter from a court attesting the good moral character of the bearer was all that was required to practice law. You can think of good moral character as the ability to show up on time, do reliable work, demonstrate a commitment to excellence and get along with others. Such a description might sound melodramatic, but it counts.

While it’s true that some certifications require that bearer to have a modern version of “good moral character,”  in addition to good test scores, many do not, particularly in tech. An employer wants to hire people who know what they’re doing and have the wherewithal to work effectively in an organization. Potential candidates want to provide evidence that they have what it takes to be a good employee. Building a testimonial of the bearer’s experience, character and competence into a technical certification will add a great deal of value for all involved. Also, the testimonial needs to affirm the bearer’s ability to adapt to change. In the not too distant future, your commercial value will not be about what you know today, but rather how fast you can work with technologies that will appear tomorrow.

How to Win the Race

For better or worse, we’re in a race with machines to keep our jobs. They don’t really want our job. They’re incapable of desire. But, that’s not to say they can’t have our job should the machine prove to be able to do it. Sadly, the current practice of technical certification is focused on passing tests concerned with the acquisition of knowledge.

Right now, we’re winning the race. Machines still have problems with knowledge-based tasks involving advanced image recognition and reading comprehension. We still know how to differentiate between a dog, cat and an apple tree better than a machine can.

But, it’s not always going to be this way. Keep in mind that talking machines were very primitive 20 years ago. Their voice had an artificial, staccato rhythm that was distinctly robotic. Today, it’s hard to tell the difference between a human or machine when a robocall disturbs your evening meal. It’s only a matter of time before the machine can do mid- to high-level knowledge-based work better than any human. Jack Ma’s quote at the beginning of this article supports this assertion.

We need to change. The commercial value we’ll bring to the workplace in the future will be about analyzing, synthesizing and evaluating. Machines will be able to to the lower-level thinking faster and more accurately than we ever can. The realm of human excellence will be about better understanding the world around us and bringing new things into that world. The way we start is by teaching something so unique that the machines can never catch up. This thing is creativity. The paradox is that creativity can always be learned, but rarely be taught. Still, we must try. We have little choice not to.

— Bob Reselman