Tag: devops

“His job was one simple thing—press a button. That’s all he had to do; press one button and sit back.” – Joe Barbera

I wanted to be George Jetson when I grew up. Back in the days when watching Saturday morning cartoons in footed pajamas was the high point of my week, George’s life looked as sweet as a bowl of Super Sugar Crisp cereal. Flying cars, robots, and a two-hour-a-day, three-day work week lightened even further by 100% push-button automation.

Contrariwise, my first tech job out of high school was a classic study in manual labor. From 9pm to 5am, I was solitary “Night Backup Operator” for a small data storage company, lone man responsible for continuously swapping reel-to-reel tapes on and off a literal wall of old magnetic drives.

It was repetitive, monotonous and boring, and the tapes were prone to jamming and tearing. A lot.

One night I was thumbing through a tech catalog and came across a listing for a new cassette backup system. I mentioned to my boss the following day that our reel drives could potentially be replaced by a unit the size of a large book. He took it to heart, made a purchase, and one evening shortly afterwards I came to work, popped in a blank cassette, hit a button, and kicked back for the next eight hours as 100% of my work was done for me. It was a George Jetson moment. The future had finally arrived. I was blissfully complete.

They fired me, of course. It actually took them a few weeks to realize that I’d been automated completely out of a job. To their credit, they were appreciative and regretful in parting, but there was hardly any point in paying me to sit on my can reading pulp SF for eight hours every night.

Regardless, I’ve been chasing that George Jetson moment ever since.

The magic behind the button

In the decades since, with a number of (slightly more professional) IT automation projects behind me, I’ve discovered that the “automate-everything” button is usually a lot tougher to achieve, and near-impossible to buy off the shelf. To even get close, you often end up building a significant amount of infrastructure up from scratch.

But say you could buy a “George Jetson” button; a self-contained, perfect automation solution off the shelf; what do you look for? Setting aside the nuts and bolts of a specific IT problem or project, what does an “automate-everything” button need to do, in essence?

1.   “Do No Harm”

At the first sign of an incident, Mr. Spacely (voiced by Mel Blanc, “Man of a Thousand Voices”) pops up and begins motivating with the bellow “JETSONNNNN!”. (I once worked outage calls with an IT director who had the same management style, I kid you not). We don’t want this to happen. So our “automate-everything” button needs, first and foremost, to mitigate risk.

Security should be the first consideration in risk management, yet it’s an area where (oddly enough) a number of automation solutions fall flat out of the gate. Managed user accounts (ideally, integration with established account management systems), role-based permissions (for pushing the button in various situations, IT environments, and scheduled times), and sensitive information handling should all be covered.

The ideal automation solution’s back-end infrastructure should be robust, flexible, and professionally engineered. If any pieces are open-source or community-designed, what they do and don’t do should be well-documented, including an understanding of potential situations likely to result in unintended consequences.

We also want the automation button, once pushed, to never perform the action again if there’s no need to do so. This is idempotence, a valuable concept that’s been around for a long time but has risen to prominence over the last year. Idempotence is particularly important once you start to consider auto-remediation (why push the button manually if it can run automatically?), and efficiency (why spend effort building an automation action that can only be run once?).

2.   “CYA”

“Cover Your Automation.” George isn’t particularly smart for a computer operator. But maybe he doesn’t need to be. Let’s say he’s supported by a lower floor full of smart engineering robots. The robots set up the automation actions and push them into an approval queue with just enough information to make a decision on whether an action is needed or not.

George evaluates each situation and pushes the button if everything looks good to him. Now, the “automate-everything” button becomes more of an “approval” button in a formal change management system. There might be more approvers besides George, each laying eyes on the proposed automation action and passing it along to the next approver as appropriate.

Most operators aren’t as dumb as George, but everybody has “dumb moments”. Change management needs to be a core part of any automation system.

Accountability supported by thorough logging is also a key “CYA” aspect of the ideal automation system. “What” happened, “when” did it happen, “who” did it, and so on… if your automation system doesn’t track all essential information, you might not be able to return to your job after Spacely mistakenly fires you.

3.    “KISS”

“Keep It Simple, Stupid.” Dealing with complex automation workflows is tricky if they involve a lot of step-by-step actions, as they traditionally have in the past. When complexity clashes with modern, large-scale IT environments, situations arise that are literally impossible to manage manually. “Scale x Complexity > Effort” is the equation desperately slashed across whiteboards in planning meetings these days. Modern automation systems often attempt to reduce complexity and equalize scale through a form of orchestrated states.

The idea depends on two parts. A desired state is a recent Microsoft-championed concept where the steps required to reach an end result are black boxed in favor of defining a system’s state itself. The usual example: rather than considering a step-by-step recipe for a birthday cake, you describe “a white cake with pink frosting” as your desired outcome and pop your request into an automated system to make it happen. Recent configuration management solutions have made excellent headway in this area.

But desired states aren’t enough in and of themselves. They almost always need to be orchestrated to reach the final goal. An orchestrated states effort runs more along the lines of: “I need a white cake with pink frosting at the same time as a bowl of orange punch, before the time of the birthday party, but after the house cleaning. Also, yellow balloons.” Real-life IT infrastructure requires at least this level of complexity and timing.

For a complete “automate-everything” solution, our ubiquitous button needs to orchestrate the states of all necessary systems and services in a way that’s easy for George, our hapless operator, to process, and considers both transitory states and final states as well as the interactions between them.

When evaluating whether an IT automation solution is right for you, you may not be able to find the ideal “automate-everything” button, but make sure you consider it from these three perspectives. The perfect solution may, in the end, just be a dream to be chased by the laziest as we move from one IT automation project to the next. But it sure is fun seeing how close we can come to that blissful, futuristic George Jetson moment.