devopsy.kinsta.cloud

Author: Don Macvittie

Automation Relation
One thing that is absolutely true: This latest iteration of IT improvement is about automation. Whether making DevOps more automated, working on any of the various “X-as-code” movements or implementing AI, the idea is to create what one vendor portrayed as a “human-free zone.” A space where automation does everything that needs doing.

And the one question we all need to be asking now is simple: Where do we want that human-free zone in our organization?

The answer is inevitably, “In more places than you think,” for a wide variety of reasons. Let’s cover what you should be thinking about when answering this question – because this change is already upon us and we’re doing a lot more without so much as an approval.
1. Anywhere that compliance is on the line and the action doesn’t risk crashing systems.
2. Anywhere the process being performed is so rote that any member of the team can do it and everyone thinks of that process as a distraction.
3. Anywhere that time is critical. Think of incident response to a DDoS or zero-day attack.
This is a very basic starting point. From here, we want to move into “anywhere we have a static list of steps to perform.” So, the above points would include things like, “If a new employee is in the marketing department, they belong in the Marketing security group. If they are further working on marketing for new products, they belong in the Marketing for product development group.” But eventually, you want to get to, “This app communicates on port 12345, so that port must be accessible, run the script to open that port.” The decision about “Should this port be open?” can then be shifted left to have more involved discussions about options and necessity. Our current world already does some of that; automation will make it standard.

And that’s what we’re getting. We’re in a world where some standardization is enabling automation, and automation is demanding that we figure out more standardization. And it is all good. There is a lot we do in IT that a trained monkey could do, and we need to let a trained system do it. The more time we have, the more problems we can solve … So let’s keep freeing up that time.

And keep rocking it. Your time is valuable precisely because you keep the organization running. Freeing up that valuable time just makes you a greater asset to the organization and makes your job less boring as a bonus.
April 12, 2023
Things We Should Acknowledge, Part Two: Skills Are Less Relevant

This is perhaps the toughest issue for all of us to acknowledge: The longer you are in the space, the more skills you have filled out on LinkedIn (or wherever you are tracking your career for HR reps to look at) but the skills you possess today are, by definition, a subset of the skills you will possess even a couple of months from now if you change roles. Note that I said “change roles,” not “change employers.” The fact is that we use an insane number of tools in the development, DevOps and operations environments, and we’re addicted to adding new ones (that might be a hint about part three of this series). Changing roles within a company can involve minor changes, or it can take you to a whole new environment where no one cares that you are a Java genius; they want to know how fast you can pick up Python and be truly productive. Or they nod at your talk of configuring servers and ask if you can do that with Ansible on containers. It’s just not as important anymore for most employees to have a specific skillset. That’s a major contributor to the problems I talked about in part one.

Of course, there are roles in IT that involve far less change. Email administration, for example, isn’t all that different from the way it has been since, well, forever. We have changed how/where the server is configured and many orgs have added SSO or multifactor authentication, creating new issues, but it’s all evolutionary. When NoSQL systems came out, it looked like a major change in DBA work, but it really wasn’t. Within a year, reality put SQL over top of NoSQL stores and DBAs kept chugging along.

But DevOps has impacted both development and operations in profound ways. We pretty much standardized on Java, with all of its warts, because it was object oriented, general purpose and could be compiled if needed. Though embedded largely stayed with C, there was no serious competition in the enterprise for Java and Java-like (.NET) languages. Knowing them meant you could do everything but real-time/embedded development jobs—and even some embedded work shifted to Java as prices for miniaturized compute power dropped.

But that changed. The inflection point was JavaScript, and from there, we went a little bit crazy. When JavaScript took off, full-stack developers needed to know one front-end and a couple of backend languages. Other languages still existed—the staying power of both Perl and PHP is somewhat surprising to me; they’re both good languages but have limitations. But still, no serious competition. But Node.js took JavaScript to the server, Ruby was popular for backends, Python came along for quick scripting … And now we have apps written in those and a dozen more languages floating around.

I chose languages to fit the whole story into a blog because they see less change than the operations side, where we’ve gone from annual big releases to rolling deployments updated frequently and all the support tools changed in the process. Infrastructure has changed, data stores have changed, mobility has changed, UI targets have changed—basically, it’s a chaotic mess.

And you’re in the heart of it, keeping the data flowing and the org running. It’s astounding, and you deserve more kudos than you’re likely getting. So approach hiring differently. Some orgs are already there, but skill-based hiring is still too prevalent. Hire broad technical knowledge and a desire to know more. As someone I know honestly said in a job interview once, “No, I never used it, but it is just another programming language. I’ll be productive in a week.” She got the job and delivered as promised. For all but those few corner-case roles, hire for potential and achievements and skills will follow.

April 5, 2023
The Right Tool for the Job: Alerting Edition

In high-tech, we have a terrible predilection for finding a good tool and abusing it until we hate it. I’ve written several “Use the right tool for the job” articles and posts over the years, most notably an examination of trying to make one programming language (or another) into an all-purpose solution. We’re thankfully past that … for now. But we do the same thing across all of IT in general and DevOps in particular. Consider what we did to Jenkins when DevOps was coming of age. It went from a great innovative tool to a bloated monster we tried to use to automate just about everything short of making coffee in between integration runs. Fact is, Jenkins is still a good tool in the CI space; everyone started hating it because it was over-used. It’s a terrible tool when used for things other than what it’s good at. Imagine that. We can make the same statement about, well, everything. “A car is a terrible tool when used as a life preserver,” is a good example of some of the absurdity that makes us hate good tech.

The 2023 edition of this phenomenon came up on a for-fun video call with some friends this weekend, and talking about it made me realize how bad it has become. Thus, I give you the point of this post:

Slack is not your documentation tool. It is not anything but an efficient alerting tool and maybe a stand-in communications tool for short-term convos.

I see it all over the place. “Oh, we discussed that on Slack.” That’s great, where did you document it? Because honestly, some of the companies I deal with have busy enough Slack channels that if the conversation happened more than an hour ago, it will never be seen again.

Slack, and a few similar tools, are astounding alerting channels, offering an opportunity to notify when events occur, and we really should continue to reap the ease-of-use and massive alerting benefits that these tools offer. But don’t confuse that with long-term anything. Because it is not.

Alerting to Slack needs to be one piece of a larger puzzle. Making sure that whatever was alerted on was also logged somewhere (either by the alert mechanism or by whatever tool spawned the alert), and for a whole array of issues that are alerted on and many of the conversations that happen in Slack, long-term documentation needs to be created.

So this is a lot of words to remind you of what you already know: Slack can enable quick communications, alerting and innovative solutions by having broader discussions without cramming everyone into a conference room. But it is not a documentation tool in any way, and it needs to be part of an integrated solution that uses the right tool for the job. Security alerts, for example, better be ending up in the SIEM (they are in every organization I’m aware of; just using that as a glaring example).

And keep rocking it. Just make sure while you’re running at 500 miles per hour to keep up with Agile and DevOps speeds, you are capturing the stuff that will be important in six months or a year somewhere that people can actually find it. And use each tool to do what it’s best at. Convenience is almost never the best selection criterion for a tool to solve a given problem.

March 29, 2023
Things We Should Acknowledge, Part One: Hiring Sucks

I was talking to some friends about hiring the other day, and a common thread seems to run through all of their hiring activities: It is hard to find “good” DevOps people.

This thought got me thinking that we assume a lot in DevOps, and that we should stop assuming and openly discuss those things. For all the streamlining and shift left it can do, DevOps could use some streamlining and shifting left, itself. So, this is the first of what I think will grow into a series of blogs about DevOps assumptions we make and openly discuss them. At least if we acknowledge our assumptions and the realities that we try to ignore, we can adjust.

Today’s blog is about finding “good” people. I put good in scare quotes because while there are common threads, the definition of good is definitely dependent upon the organization and even on the team. If a team needs someone with Azure chops to rev up their work and help the rest of the team become more proficient in Azure, then a GCE expert, no matter how great they are, is not “good.” This is true of every field of endeavor, but we in DevOps often pretend it is not true for us. Particularly the pundit class, but management is also severely guilty of “It’s DevOps, everyone is everything,” which any active practitioner will tell you is not true. There are still specialties, we just hand wave and blather about “It’s all DevOps.”

Unfortunately, there is often a disconnect here. DevOps roles are not well-filled by filtering on technology and they’re not well-filled by just using “DevOps” in the job title. That leaves HR struggling to find candidates to interview that techs will find appealing. How does HR filter on “can learn on the job, and do so rapidly”? Simply put, they don’t. That’s what interviews are for. But if HR filters on “must know Jenkins,” that is diametrically opposed to “must learn on the job.” Which is it? Must know certain skills and have X years of experience in them, or must be able to pick up new things on the fly? Because filtering on already having the skill would seem to imply you don’t really care about the ability to learn fast, would it not? Because there are great fast learners that have never worked in a Jenkins shop, but they could, if they’re truly fast learners, pick up Jenkins.

So ‘good’ is relative, but beyond that, what I hinted at above is also impactful. We are–nearly all of us–on-the-job trainees. There are certifications and universities are starting to catch up, but most of us? We learned either on the job or on our own time, trying to do some pet project. This, too, hurts HR filtering. Demanding a four-year degree is something that HR can measure. Demanding on-the-job training? Not so much. So, if HR gets filters they can measure, they will not be filtering based upon reality.

This is how we get things like, “Must have 30 years of Rust experience,” even though Rust is 20 years old if you count from inception. Far less if you count from the start of its usable life. HR is not an efficient or effective filtering system for DevOps, and we are somewhat flailing. Until we have improved systems and processes, I would suggest having HR send too many applicants through, and DevOps engineers or managers–those who will have to work with (and count on) those new hires–do more of the filtering than normal. There are honestly people out there with no relevant degree and no certifications that are better than most of us. Project work is the only way to gauge that measure, but project work cannot be effectively evaluated by HR because they’re not technical specialists. I know—the org is hiring people because they need help, so offloading some of HR’s duties on the team that needs the help is not a great solution. It is better than going through an entire outreach/interview/hire process only to settle for ‘good enough’ (or, worse, ‘not good enough’) because the process is flawed based upon the current environment.

This brings us to the scenario we see today–where more technical people have hit the market than in recent history, but hiring managers are still struggling. Have the DevOps team take a more active role in the process until new standards can be settled on that suit the needs of your organization. You’ve worked hard to make an astounding infrastructure and build apps that can handle the needs of your organization, make sure the people you put next to you are the best you can find and the organization can afford.

And keep rocking it. Teach them about the systems you shepherd and learn from them what they bring to the team.

March 22, 2023
Modern DevOps is a Chance to Make Security Part of the Process
I’ve mentioned before, and many of you have lived through, the slowly changing beliefs around DevOps versus security. We are past the days of “Security slows us down” and into “How can we bake security into the development process?” which is essential. Indeed, I would say it’s more essential every day as the number of threat actors and their use of automation to perform more attacks increases.

So where can we slip in a little security to protect those sweet, innocent apps from the Big Bad Wolf? Let’s make a list, then chat about it.
- Source code
- Inclusions–libraries, shared objects, etc.
- Container/cloud images
- Network
- Running applications and APIs
This is honestly not an exhaustive or detailed list, but we’re writing a blog here, so let’s stop there so we have time to discuss.

Source code security is greatly enhanced by putting in any of a dozen development security tools, which include static and dynamic security scanners along with a variety of other features.

Inclusions are often nested and confusing. A mod that developers include relies on other mods and libraries—sometimes so many that the developer can’t even list them all. So, this type of security starts with software composition analysis (SCA), then uses the list that generates to pursue vulnerabilities—normally by looking for the included libraries in a vulnerability database. Several prominent source code security solutions bundle at least the SCA part, and some also help nail down included vulnerabilities. Check with your vendor if you’re just getting started here.

Container images are akin to the “golden images” of old. Build one that has what most applications need, add on the security that an organization wants most applications to support and then have every project start there and only add what this specific app needs. That allows for scanning of the base image so all applications start out at least somewhat secure. Cloud images can be done much the same, but beginning at a higher level. Look into registry security or image scanning tools to meet this need. It is worth asking your source scanning vendor if they offer this functionality; some do.

The network on which an application will be exposed needs to be locked down. While the access points into the org probably are protected, portability of applications means that the application itself should be locked down wherever it uses that network. Tools are specific to the platform and networking in use, so look into it—I can’t offer much more generic advice.

Security has long tried to lock down running applications—generally with pushback from developers and even ops, a bit. Between web application firewalls (WAFs) and API security, we have the ability to stop a whole host of attacks on their way in … But you have to use WAFs and API security best practices. If you’re starting from scratch, consider tools in the application and API security space that protect against both the web UI and API attacks with a breadth of technologies. For those with a WAF, look at API security tools that protect APIs. For those with API security covered, consider a next-generation WAF.

My usual encouragement applies. You are rocking it, and all of these tools are increasingly integrated into the DevOps toolchain, so it’s worth the time investment to get them set up and let them run. If you’re just starting out, start with source code security. Honestly. It will make the overall app more secure by tightening its source. But don’t ignore the ops side, as that is part of the attack surface. Protect those apps you built and/or maintain that are keeping the org running.
March 15, 2023
Raise Those (Feature) Flags

I’ve written about feature flags before, but I think it’s time for a solid bit of advice: “If you are not yet using feature flags for DevOps, it is past time to reconsider.”

I say this because feature flags have been a slow-growth item, with a few spikes here and there in usage, but mostly on the development side. While many of you are using feature flags to do dynamic toggling of features–definitely an advanced use case, but still very development heavy–few organizations are using them for deployment options–blue/green without separate code, environment changes for different targets (deploy this to the cloud provider, but do this for local installation type stuff), etc.

Feature flags are starting to be supported by DevOps tooling for this precise use case, and it is glorious. While the ability to say, “run this code,” and if the new code fails, change something like an environment variable to say, “stop running that bit,” is huge. We have enough time with this process that the industry is well aware of the need to clean up as soon as the correct code options are known, and that is the biggest weakness; it makes spaghetti code that must be cleaned up. But the benefits are well worth a bit of cleanup at the end of a project, even if far too many of you don’t bother.

Now we can use them in deployment, and in full life cycle toolsets, we have them available for both–so a set of source code can be gated behind a feature flag and deployment can use the same feature flag. This enables the ability to say, “If our FF environment variable is ‘TestDeployment,’ connect to ‘Test Database’ … then use the same variable to say, “If our FF environment variable is ‘TestDeployment,’ spin up the container with test data instead of real data.” This simple example is something that could be (and has been) done manually, but doing it this way makes it part of the process. Automated testing can then follow without the need for human intervention, and from “commit” to “test results,” IT can focus on other things instead of spinning up the right containers and making sure the right DB is accessible.

After all, we are in a time of massive automation, which should not worry us. There are globally too many IT positions and too few people to fill them; automation should be exercised as much as possible to address that simple fact. And there is a good chance that you are already using tools in the DevOps toolchain that support feature flags–a growing number do support them. If not, it is easy to research and find a good tool to do so and while it doesn’t offer as much functionality, a stand-alone feature flag product will still offer more options than not having any.

And keep rocking it. The applications you wrote years ago are still powering the enterprise, even if you are at a different organization today. We leave a legacy of applications behind us that is a testament to quality. Keep leaving quality, and use all of the tools you reasonably can.

March 8, 2023
In DevOps, Chatbots are a People Problem

By now, you are undoubtedly aware that you can call up a chatbot or install an AI assistant into your IDE and generate entry-level code and deployment scripts.

Indeed, if you just go to a site like ChatGPT and enter “generate Python code to do X” where X is something you need to be done–or an example like “read a file and print out the contents one line at a time,” it will do it. And if you keep it simple, it will do it well. If you say “again, in Java,” it will comply and do it well.

Then say “new chat,” and tell it to “write a bash script to deploy an app named X written in Java and using Gradle to a cluster named Y,” it will do that also, informing you of the names of the deployment and service YAML files. Then you can say, “generate the deployment file mentioned,” and it will. And then you can say, “parameterize the Java version,” and it will do that, too. This iterative revision can continue until the deployment code, containers and deployment files are complete. If you have not yet, I urge you to spend the half hour going through these steps once required, so the potential is obvious.

It is not yet to the point where I would (or you should) trust it with complex systems, but all of that busy work we’ve been trying to overcome with templates and preconfigured containers? We can actually get around the bulk of it now. The same is true for testing of all kinds. There are tools that will auto-generate test scripts for a given set of source code. They will also generate test execution code. That means that the only step left—and the one that has held testing back because of the volume of work—is reviewing results … But wait, there’s more! In security testing, AI can help eliminate false positives. In performance or QA testing, it can filter out all but the most egregious issues (as defined by the org or by a larger pool of apps).

The problem we need to address today is the people issue. What we have traditionally used entry-level people for—and trained them in Dev/Ops/DevOps with—is no longer necessary. Because the AI can do it. That is stunning and really very cool, if you think about it.

But it heralds a danger that every org needs to be aware of. What we do and how we do it is changing. For the better, but change always brings uncertainty, and the foreseeable bit we need to confront is, “Now that we don’t need entry-level people for these steps, how do we recruit and train them?” I do not yet have a definitive answer for this question, because I am a product of my time, so I see things in terms of learning the bits that make up the whole—advising people interested in DevOps to learn at least one Linux command shell, to understand how networking works in containers, to write the scripts to get containers (and/or public cloud instances) up and running, etc.

I do think that every org should start using these tools at a pace that makes them comfortable because it allows us to continue our “more with less” growth without significant pain. But we need to make certain we are training the next generation of DevOps engineers in the process. For the foreseeable future, we’ll still be the ones bug fixing and troubleshooting, so we need to understand the environment and the applications. We are starting to get some sweet tools to help us at a higher level, though. Anyone who has inherited complex systems knows that the learning curve of tons of source files that make up many subsystems and all of the spaghetti code that these can be comprised of is painful. I don’t normally mention brand names here, so I’ll just say there is at least one tool out there that you can ask to “describe what this code does” and it will run with it and give you an answer in plain English. This might be the training tool of the future, with the ability to compare what you see in code with what the AI describes in English.

But for now, even if you don’t hunt down app description AIs, having entry-level people review the files that the AI generates will serve two purposes: First, to validate what the growing AI space is capable of and second, to get entry-level engineers into the entry-level bits of the systems.

And keep rocking it. Make sure you have a path for DevOps in the AI future, but remember that the creation of the calculator did not statistically change the number of mathematicians, just made the bulk of them more productive. That’s all we’re doing here, just a bit more flexibly … But in the end, like a calculator, the AI has an equation that it fills and reports back with.

March 1, 2023
Managing Risk

We have built some beautiful toolchains that crank out a finished product on the fly without needing anything close to the level of intervention that was historically required. The most advanced organizations on an automation journey could change a line of code and then wait for the new version to hit production without doing a thing. I say “could” because I don’t know of a single org that does this; there are, of course, checks in place and it’s not single-line changes but projects that release. But the infrastructure is there that would allow them to do this. It’s a high probability that one day we all will.

The problem is that most organizations only look at risk when building these toolchains and don’t bother with contingency plans. But over the long life of any given toolchain—toolchains that are made up of many moving parts—there will inevitably be the need for contingency plans.

And it’s certainly not only toolchains—remember when popular libraries were purposely corrupted by their author to make a point? And unless you’ve done your best to ignore it, you’re also aware of faked versions of popular libraries that insert code before continuing on to the original…

Most organizations do not consider a scenario in which critical libraries are suddenly unavailable or a scenario where a tool is crippled by movement in the market—a vendor making harsh licensing changes, the underlying tech making changes that impact the tool, etc. While it is comforting to think that open source protects us from many changes (and it forking does), that is not complete coverage that entirely protects our systems and toolchain. We need to know what is absolutely critical and how the organization will keep delivering in case that absolutely critical element goes away.

When I have done disaster planning—of which this is an adjunct—I have done so with a vague generality in mind. We have plenty of work to do, so all I ever wanted was a list of critical parts and an idea of how we would adapt without them. I worked at a place that kept a backup data center in case of total regional disaster—the backup data center was intentionally located on the other side of the country—but most of us don’t need that kind of DR. Certainly not if we can go, “This OSS library is used in 60% of our apps; if we have to, we could adapt this other one,” instead of building infrastructure.

For OSS dependencies, we have two great options that work together. Generate SBoMs, then correlate usage from those across the entire app portfolio. Then set up and maintain a local repository with point-in-time copies of those libraries that are deemed critical.

A similar, “Our CI tool is critical, but we could easily port processes and YAML to this other one,” is enough, too—it is absolutely not going to be easy if you ever have to do it, but “We looked at this six months ago and we know what to do,” is far better than “Oh crap, now what?”

You built it, you operate it, you are rocking it. Just take some time to protect it and you are covered.

February 22, 2023
Back to Basics

We here at devopsy.kinsta.cloud do sometimes suffer from The Curse of Knowledge. We write about DevOps regularly for an audience that is doing or managing DevOps. We assume knowledge, but there are still a fair number of people and organizations who are unclear about some of the terminology/technology that we talk about.

So if you are a regular reader of my blog, this might be a redundant post—but read on because it might have info you haven’t had a reason to consider yet.

I’m going to start with the obvious—how most readers got here to begin with: DevOps. DevOps is automating the build process so that builds are automated and repeatable. Teamed with Agile development, DevOps means that development teams can rapidly iterate through development and generate regular updates to their applications. Everything else in this blog is based on the premise that it is running in a DevOps environment.

Continuous integration: Typified by Jenkins, continuous integration (CI) is the part of DevOps that runs the automated builds. It includes reporting and defect resolution only in the sense that it feeds into other systems that track defects. With CI in place, a developer or DevOps engineer can say, “Okay, ready to test a build,” and with one command, can kick off a build that will result in a finished artifact. Or result in a built product, anyway. ‘Finished’ is optional, and the target of CI is not generally a running application.

Continuous delivery: One of the most confusing things for new DevOps people is CD. The way that the space evolved, there are actually two things that CD could stand for; continuous delivery–automated build of the complete application —or continuous deployment—delivery of a running application onto infrastructure. Continuous delivery is straightforward and means all the bits of the application are code complete–or at least the current state of the application is complete enough to build an application. Continuous deployment is much more convoluted. It might mean deployed to a development environment for developer-level testing or a test environment for running automated (continuous testing) tests on or in production. It might be a staged rollout, where users are siphoned over, or it might be a hard rollout where the old version is replaced by the new in a manner similar to ripping a Band-Aid off.

Continuous test: Automated testing is at the heart of continuous test (CT), and it is only now becoming a real possibility for most short-staffed enterprises. Advancements in machine learning (ML) mean that tests can be run, results filtered to eliminate false or duplicate results, and IT staff is only notified of the important bits. Unlike traditional testing, staff is required only to create the tests and to look at meaningful results.

Infrastructure-as-Code: Infrastructure-as-Code (IaC) is the ability to define all relevant elements of infrastructure with text files. Often this is done in YAML and sometimes in a variety of other formats. The idea is that infrastructure provisioning can be on-demand and adaptable. It is certainly more agile to change a text file from port 80 to 8080 and then run a command to process the text file–now the port is documented and whenever needed the command can be re-run to move to a known state. IaC largely relies upon software implementations over physical. While the idea of software-defined networking (SDN) has fallen out of favor, the underlying principles are everywhere and part of IaC, for example.

GitOps: All the rage these days, GitOps is probably best defined as applying DevOps to operations. That does sound weird, but I like to break it up as “traditional DevOps is DevOps, while IaC (to a lesser extent) and GitOps (to a huge extent) are DevOps.” The point being that GitOps uses the text files generated in IaC and worker nodes generally designed for continuous deployment and then performs operations tasks based on those text files. If IaC defines the entire environment that an application needs and DevOps defines all of the sources that an application is composed of, bundling the two together will enable full-on automation from conception to decommissioning. When a new version of a source file or configuration file is checked into Git, the system then makes certain that every instance of the application is updated with the new version. Of course, this process, like every one listed here, is far more complex than that in implementation; we are talking about ‘what, not how’ at this level.

All of this really is the very tip of the iceberg, but that was the goal here – to offer readers a place to start if they’re finding themselves looking at DevOps for the first time. As you dig deeper, things like DevSecOps and test driven development (TDD), or defect tracking and Agile methodologies start to play into the landscape, and maybe I’ll write an intro bit for those also. But this time, we’re out of space.

As always, keep rocking it. No matter which or how much of the above you are using, if the organization’s applications are up and running, that’s you. And you deserve more kudos than you are likely getting. But keep grinding through—the rest of us get it, and thank you.

Image source: Photo by Towfiqu barbhuiya on Unsplash

February 15, 2023
We Are Living in an Ephemeral World

We are lucky to live and work in the times that we do. Oh, we have our challenges, and I talk about them a fair amount here on these pages. But the things we can do in IT today have not been possible in the past, and the change isn’t over. Soon you will be able to relegate pretty much all of the busywork of IT to automated systems, Note that I did not say “AI/ML”–some of that automation will be, but we throw the term around too much right now, and not all automation will be based on AI/ML. In fact, quite a bit of the best automation is simply rule-based systems that we finally have things standardized enough to write up. And I intend to talk specifically about automation that is not AI/ML today.

We have reached the point in DevOps where we can automate the entire build/release process using containers. The CI tool, running in a container, can spawn a container that performs the build. Then, when it is done, if it passed, it can spawn another container to run tests, do security scans, etc. Finally, the product can be promoted and released … in a container. Human involvement is only required for approvals and then only for organizations that want checkpoints on the way to production (which is definitely most but not all of us).

If you take the totality of what is going on out there in DevOps, it is finally catching up to DevOps. The biggest issue will be making it efficient and doable for the DevOps team. To date, by focusing on DevOps, teams have been able to avoid the simple fact that complex deployments on infrastructure that spans environments and uses external resources is hard. By keeping the DevOps side simple (or not part of the DevOps process), developers can crank out code faster, facilitated by DevOps tools and Agile methodologies, and operations can continue to wrangle the beasts of networking, storage, etc. to get the app running in production. Once ops has completed the process, then it can be automated based on steps taken.

But we now have Software-Defined Everything ™ combined with containers. That means that for many–though not anywhere near all–applications, a commit that updated a constant in a single source file can spawn the entire process in iterative containers all the way through release. That is very cool, and we should take a second to appreciate how far we’ve come in a short time.

Then we need to talk about those worker containers. They need a ton of centralized logging and reporting. Spinning them up, letting them run and then automatically destroying them is fine—as long as the organization has access to the information it needs to review and analyze the results. In-application tooling will cover the released product and even the test environment, but not an ephemeral build process. By not keeping a ton of build tools running all of the time, the enterprise is, by definition, more secure—the blast radius is reduced if build tools are only available when they’re active, and logging of who/what kicked of a build is kept. But don’t forget the reporting. At build time, our tools know exactly which error failed a build; the container needs to submit a ticket before being destroyed or log heavily. I say, “Always create a ticket,” but I’ve worked in enough environments to know that your reality may not make that possible. As long as notifications occur, it doesn’t really matter how they occur.

So crank it up; start super-automating. Tool up those build environments, then make them into disposable container instances. Others have been doing this for a while, so there is help available for setting it up, but it is just now taking off, so it is a good time to dive in. And keep rocking it. How the apps are built and deployed is only important in the sense that you are driving the company forward. Keep doing it.

February 8, 2023