devopsy.kinsta.cloud

Author: Don Macvittie

  • The Right Tool for the Job: Container Edition

    The Right Tool for the Job: Container Edition

    I have written about “the right tool for the job” repeatedly. Because (again) we are absolutists in IT. My father was an antiques restorer that dabbled in flipping houses and was rather forceful about the “right tool for the job.” When I left the service and joined IT, it was something of a shock to me that the “right tool for the job” was considered situational.

    At the time, there were a large number of Microsoft zealots that loved Visual Basic, and a larger number of non-zealots that preferred C or Java. That was how we got to my first “right tool” article. VB was very strong with regard to UI and record crunching, while to-the-metal was C’s strength and Java fell in between the two—not being perfect at either. So my article stated that. Front-ending databases? Use VB. Writing drivers and low-level processes? Use C/C++. Need a bit of both and easy web page generation? Go with Java.

    It is many years later and time for an article that seems just as obvious to me, but in a different IT space. We have all heard of the three architectures: Data center, cloud and mixed/multi-cloud. We have also heard of how, in this rapidly changing world of DevOps, Agile and containers, different organizations are using different deployment methodologies. That’s all good so far. Just as the variety of programming languages available today is a bonus in many ways, so are multiple deployment and operations options.

    Where it gets sticky is when people include the “We’re an X shop” in conversations. Or try to do everything under the sun with one of these methodologies. For many of us, container infrastructure is perfect. Use an orchestrator like Kubernetes, build systems and security practices, then not have to care much if the Kubernetes instance it runs on is on-premises, hosted or cloud. Of course you have to care some. Networks have to connect, etc. But most of the work is done.

    Then there are the people that have use cases that don’t fit this mold and have not acknowledged that fact yet. This blog is for you. We’ve all heard of companies that have containers whose life expectancy is measured in seconds. Beyond that, I was told about a company that is using containers as message queues. Honestly. Creating a container with a startup command and putting it on a queue. When it is pulled off the queue, it executes that command and then goes away. We’ll call this crowd the “Over Users”.

    Finally, we have the people that cannot or will not be moving their systems anywhere. There are a variety of reasons for this, but these are companies that know their applications will continue to run in the data center for the foreseeable future. We’ll call this group the “No Needs”.

    For Over Users, that is literally what serverless is for. Create a queue with a (much smaller) call to an API in a serverless environment and be done. Fewer configuration issues, less maintenance, less everything but code. Add a trigger, remove entire OS/network/configuration generation and maintenance. The most likely resistance to this advice is “But we have an investment…” Yes. And for most orgs it is an investment in unnecessary technical debt. Divest. Keep the good parts—a container architecture meant to shuffle through many containers that exist for a short time is well-architected.

    For No Needs, before you port everything to containers—which I am generally in total support of—you should consider if you need to; think about if it offers any benefits and what the ROI of such a move is. Honestly. For DevOps-heavy shops or orgs that require highly portable solutions, the ROI of containers is quick and obvious. For orgs whose applications are not changing a ton and are highly unlikely to move anywhere, the ROI is a whole lot murkier. “But it gives us options!” is the most likely negative response to “We don’t need that right now.” But that’s the problem—you have those options now. What it gives you is a shifting of cost from some possible future where it might be needed to today’s already constrained budget. If there is a verifiable need—normally in portability, but any verifiable need—then make the move. Until then, spend that time and money making those apps rock.

    You’re doing it. The lights are on and the applications are useful. Use the architecture that makes sense for what you are trying to achieve. Right tool for the job, as my Dad often said (he said it a bit more forcefully when I thought a chisel was a screwdriver, but that’s a different story). Keep rocking it, and stay focused on need.

  • Focus on What You Can Do

    Focus on What You Can Do

    Here we are in the second half of 2022 and the writing is on the wall. No matter how much budget you have had in the last few years, be it large or small, it is going to shrink. This is cyclical, and should not be a surprise. The amount of budget shrinkage may be larger, but budgets have fluctuated since before any of us were born. I would argue that they fluctuate in IT more than any other part of the business, simply because much of IT is still viewed as overhead and, honestly, we’re expensive. Our salaries, our equipment, our software and our SaaS cost a lot, making them juicy targets when corporate belts have to tighten.

    So let’s talk about what to do for the next year (or five) while things finish sinking and then start to recover. New projects will be few and far between but improving what an organization already has deployed costs little. And the “Do it now” mentality of Agile and DevOps has helped in a ton of ways, but honestly leaves a trail of “could be better” implementations behind. Implementations that you can use iterations to improve. My “Look at me! I’m a pundit too!” recommendations would be:

    1. Review
    2. Consolidate
    3. Evaluate
    4. Iterate

    Review what you have across the entirety of the org. I’ve made this recommendation before—possibly even as recently as six months or so ago—but it bears repeating. We had a phase where teams and even individual developers made choices for a product or product line. This created a lot of overlap. List tools used at each step of the DevOps chain; for each project/DevOps toolchain.

    Consolidate. In the short term, costs need to go down and be sustainable once they do. Better to eliminate software or SaaS redundancy than that other kind of redundancy. This step also means simplifying the corporate DevOps infrastructure. So compare products doing the same job and choose one. This isn’t as easy as pundits make it sound; each product brings something unique to the table. But you have to make the hard choices and reduce maintenance costs and hours.

    Evaluate. This is not hard to believe for anyone who has worked at multiple organizations, but in case anyone missed it, no environment is perfect. They all have strengths and weaknesses. The best staff in the world, given unlimited time and budget, will do at least one dumb thing. Then after a year, a lot of the smart things they did look dumb too—because the market and/or the technology have moved on. Look at each step of your environment—from which libraries/modules are included to the deployment steps and run-time monitoring. Figure out what could be better and then organize by cost and schedule them out.

    Iterate. Start back at the review stage and do it again, because if you think you got it all the first time, you’re wrong. Or you are a heck of a lot smarter and more thorough than the rest of us, one of the two.

    This will keep the team improving the corporate IT environment while outside factors keep budgeting down. This cycle is different from others because it is unclear how IT staffing will be impacted. While it is looking more and more like IT will have the same impacts as usual, at the start of this downturn that wasn’t at all clear. Either way, improving the environment without million dollar projects is useful in hard economic times. My company is currently reconsidering our choice of frameworks and how we’re still using a test tool as DevOps management tool, for example.

    Keep the lights on, keep rocking the servers and apps running the org and focus improvements around factors you can’t change. But keep making it better, no matter what’s going on around you.

     

  • Build Once, Deploy Anywhere

    Build Once, Deploy Anywhere

    This late in the game, it seems stunning to me that I have to write this, but there are plenty of you who aren’t there yet—so let’s try a different approach.

    A huge part of DevOps is reducing redundancy. We have cut the fat until it hurt, then refocused and cut some more. There is a blog here about burnout, but we’ll save that for another day. The point is, we don’t do the same thing over and over unless there is a quantifiable benefit—like rebuilding and testing new iterations or spinning up base images to start a new project.

    We insist that our applications run in as standardized an environment as possible, with images from a repository and standardized software installed.

    And then we promptly put it into a unique environment where deployment is completely different. Yes, I’m talking about which cloud or which data center; if it’s deployed via a data center, then hardware, virtual machine or container? The work to standardize the important bits and leave the less impactful bits to teams to decide has been astounding on the Dev side of DevOps (for those new here, what I call DevOps), but have been far less effective in DevOps. Even companies that claim to be pure cloud generally have two or more cloud vendors, each of which is doing things differently.

    So, with all of that said, I can say what I should not have to say after all of these years:

    Make containers standard. I don’t care which flavor and management tool you use, but add in that portability layer. It will protect you from a future where your preferred hosting environment (be it cloud, local or hosted) becomes hostile or overly expensive—or simply ceases to exist. And it will standardize the rest of your DevOps processes. Yes, your preferred cloud vendor offers Kubernetes, and no, that’s not sufficient. Kubernetes offerings today are like Microsoft in the 1990s. Embrace and extend to build the lock-in barrier higher. You do get some great functionality from the cloud vendors I pay attention to, but processes become non-standard. Better to limit platform-specific work to “spin up a server” and have everything else be reusable whereever you are deploying. The benefits here are far-reaching; there will be configuration and security issues that are different when you move from cloud to cloud or cloud to data center or data center to cloud, but the core of DevOps will remain standardized. There are tools to help with baseline cloud security and risk management, so those in addition to using Kubernetes as a portability platform will make current work less onerous, and massively increase portability from deployment location to deployment location.

    You’re short of staff. It does not take a rocket scientist to predict that this situation will not change. Adopt the things that will reduce your workload—like DevOps standardization on Kubernetes (or OpenShift or whatever) as a deployment platform—and keep rocking it. You can’t wave your hand and get more staff or hours in the day, but you can implement containers as the sole deployment option to save time. And use those hours to solve other company problems—because that’s what we do, solve problems for the business. And keep smiling, because you’re rocking it. If you weren’t, they’d find someone who could.

  • Everything is DevOps

    Everything is DevOps

    It’s funny how we in IT tend to be absolutists, shoving everything into the current bucket. XML was going to eliminate programming—even though it wasn’t even particularly good at data representation. NoCode was going to eliminate programming (yeah, that’s a theme, I’ll skip the other 50 or so ‘going to end programming’ cycles we’ve been through), AI will eliminate IT, you must have a mobile app, even if no one will ever use it, my OS/browser/language is better than yours and will win—you get the idea.

    We’re currently coming to the end of the “Everything is DevOps” phase. DevOps is a great methodology that helped solve several intractable IT problems but, like everything, marketing and a ton of smart people that want to be “right” have taken it too far. Or have they?

    The thing with DevOps is that contrary to what those who try to make a separate market out of DevSecOps or SRE try to tell you, it was designed to literally encompass all of human IT activity. We can excuse the DevSecOps crowd, since DevOps’ roots had a definite “Security is an unimportant bottleneck” feel to it. The SRE crowd, on the other hand, appears to be another case of “my silo is special”. Or, better yet, a stompy marketer saying “We’re late to the DevOps game! Let’s invent something just like it, claim we do it more broadly and give it our own name!”

    Let’s look at what each does, for those of you who are skeptical. 1) break down silos. Check. 2) Monitor everything. Check. 3) Use monitoring results to improve product performance and stability. Check. 4) Design and implement products and product features. Check. There’s more, but you get the idea.

    Do you know what all of this ignores? You.

    One of the issues that our absolutism brings about is that marketers get to dictate what is important. Sometimes that is what enterprise IT needs. Far too often it is what the marketer is paid to dictate.

    I don’t care if you call it DevOps or SRE, I care that your organization feels like it is getting its money’s worth out of IT—something, I might add, that most organizations did not feel prior to DevOps. At least most of the ones I dealt with. I even had a job where they created a whole new department and hired me to run it to “get around those people in IT.” I spent as much time educating business leaders about not duplicating entire groups like storage services and InfoSec as I did building and deploying new products. So avoiding that kind of issue is near and dear to my heart. And we’re more ‘there’ than we have ever been.

    I do care about whether you call it DevSecOps or not. It should not be different. Shifting security left is the appropriate solution to the double-edged sword of security staffing shortages and increased adversary capabilities. So if your developers are getting secure coding feedback in their IDE or each time a build is kicked off, great! But call it DevOps. Say it is part of the DevOps process. Same with configuration reviews like cloud configuration management—it’s just part of the DevOps process, not a security step to be thought of differently.

    I could go on, but hopefully I’ve spurred your imagination a bit and don’t need to.

    You rock. Streamline your thinking to streamline your processes, ignore absolutism and call it macaroni for all it matters. Just keep doing what needs doing—at the most convenient place to do it. And keep developing, deploying and monitoring the systems that keep the company running.

     

  • Who Controls Your Build Process?

    Who Controls Your Build Process?

    We have come a long way in a short time in IT, and are currently in a ‘digestive’ state, where we take a bit of a break and prepare for the next wave of changes. Okay, not really a break, just a change in what we’re looking at.

    In the old days, CPU, memory speed, network throughput and disk access times would be in constant flux; one would be the server performance bottleneck and, when that was resolved, another would be the bottleneck. We’re in that kind of relationship right now. The majority of organizations are now using DevOps and our focus is currently on DevSecOps along with multi-cloud. Along the way, by unwritten consent, vendors have started calling Kubernetes “cloud.” This is totally amusing to me as a pundit, because Kubernetes is really a “cloud killer.”

    All public clouds are based upon the premise that they will have massive lock-in. And Kubernetes breaks that premise—unless an organization uses a cloud vendor’s customized version of Kubernetes; then they have other issues. No, I’m not blind, I am well aware that there are other reasons that cloud lock-in occurs, but there is a difference between “We chose this bit of custom functionality, well aware it locked us into our chosen vendor,” and “The act of using this vendor locks us in.” Multi-cloud and repatriation are things that would not be popular enough to talk about if not for tools like Kubernetes.

    So the next bit, IMO, is that once cloud portability is an established fact at most organizations, we’ll circle back to another iteration of build management. Many organizations are still on CI/CD iteration 1.0 and using Jenkins to manage their build process. While companies like GitLab and Atlassian have taken a bite out of the Jenkins-as-build-management phenomenon, they are also solid lock-ins through value add, and both of these vendors (plus most others in wave 2.0 of DevOps-centric build management) are heavily invested in the cloud. While the ability to build in the cloud is a very good thing, the requirement that a tool which is central to your toolchain be run from the cloud is not.

    That means we’ll be circling back to visit this issue again. And probably sooner rather than later. There is no denying that budgets are going to decrease for the bulk of us in the coming months, and I have no idea when they’ll rebound. That means that recurring expenses—like mandatory monthly subscription fees—will take a hit in B2B the same as they are in B2C. Don’t shy away from those assessments. Actually look objectively at your monthly payment plans and determine the ROI of each subscription your organization uses. Years ago, organizations centralized and reduced the number of development tool subscriptions they managed; this is just a continuation of that process in modern times. Spend your organization’s money where it will reap the most rewards instead of “where we always have.”

    You also want to make sure that you trust the company in charge of your builds, and that you believe they are stable. Over the years, we’ve seen several examples of companies that appeared stable when they were not; actually making the assessment is important.

    You’re building and maintaining the systems that keep the organization online and productive. Don’t hesitate to cut the fat. And don’t hesitate to maintain control. If the tool and its lock-in are forcing the org to work in ways that don’t fit the culture, look at alternatives.

    In short: Keep your options open. We know that money will be tighter in the near future, so let’s be sure we’re set up for that and retaining control of our build process. And keep rocking it.

  • Lock Down Your Toolchain

    Lock Down Your Toolchain

    We have done amazing things with Agile and DevOps, increasing IT responsiveness to levels that most people would not have believed and our business counterparts only dreamed of even a decade ago. Think about it—we can check in a single source file and kick off a chain of events that involves half a dozen or a dozen software tools that collectively shepherd an entire application through the development process to deployment. Through all of that, only deployment might require human interaction, depending upon corporate policy. In many places, even deployment does not require a person. Just development.

    We didn’t get to this point by moving slowly, and sadly, we did not get to this point by acting in a security-first manner. Thanks to the security-toxic mentality that DevOps was born in, we have had to go back and try to bolt on security where it was ignored initially. But we’ve largely managed that process. We have tools that handle privileges, monitor connections, scan source, look for vulnerabilities in support software, even scan entire container definitions before the build.

    Know what most of you still don’t have? A truly secure toolchain. This is bad policy. A knowledgeable attacker is going to go after one of the aggregate locations. Places like container management servers that give them access to a lot of resources. Or the toolchain, which gives them access to every bit of software the organization develops. If an attacker wants to do code injection without detection, the toolchain in charge of builds and deploys is it.

    And yet, for ease of use and keeping up with iterations, many of you have credentials for the toolchain stored in easily accessible places so that whichever tool is running the show has access to credentials for each piece of the toolchain. I wish I could say “No one keeps that data in flat files anymore,” But alas, I cannot.

    One of the reasons it is not sufficiently secured is that build toolchains fall somewhere between Dev security and Ops security. That means our teams that came from a development background are not focused on security for the build chain, and our teams/team members that came from an ops background are not focused on the toolchain selected and often initially deployed by developers.

    Which brings me to my recommendation. Put security professionals on the toolchain to verify it is locked down at every step. There are a lot of places for the toolchain to be insecure, not just poorly stored credentials. And the toolchain touches all of IT’s crown jewels.  So make sure it isn’t wide open. If your organization doesn’t have dedicated security staff, bring in consultants to review the toolchain and make recommendations. There are some great security teams for hire out there that will be happy to lend a hand.

    And keep rocking it. Your teams are building and deploying at a rate never seen before—even if DevOps is “slow” at your org, it is still likely faster than its predecessors—for doing anything. So keep working miracles, and lock down your castle’s the inner keep.

  • Filter the Firehose

    Filter the Firehose

    We are tired. Information overload is a problem in the modern world. We hear instantly about events we never would have known about otherwise, or that we would have learned about months after the fact. Today, moments after an event, we have thousands of “professionals” analyzing it for us, a millions-strong army of amateurs telling us they know everything about it and a legion of bots telling us what to think about the event. No matter what the event is. Often “the event” isn’t even newsworthy, and yet the process feeds on itself.

    We in IT have it worse. We have a flood of security and application events constantly flooding in and so much data that we call it a ‘data lake.’ More like a data ocean at some organizations. We’re tired. They call it by various names—alert fatigue, data overload, event flood, “the firehose.”

    No matter what it’s called, it’s a big problem. And if it is not effectively managed, we will make mistakes. Particularly in security—but also across IT—we cannot afford to allow the flood to distract us or burn us out.

    Look into AI. The amount of data, alerts, information, metrics, logs, traces, etc. that we are accumulating—and the rate at which we are doing so—is growing faster than humans can reasonably keep up with—even if there were no staffing constraints. Don’t hesitate any longer; don’t assume you or a vendor NOC are managing this well enough with only eyeballs. AI solutions have matured enough to reliably sift through at least the top layer(s) of noise, reducing the burden without increasing risk. In fact, given the increasing possibility of human error as volumes increase, probably with less risk.

    Ask vendors about it; talk to your security vendors about their offerings. A huge number of security vendors have NOCs now that they would love to have you subscribe to. Many of them are highly automated and use AI. Data vendors are behind a bit in this regard (for a lot of reasons) but they are catching up. Not too long ago, the vast majority of work done to normalize disparate datasets was done via trial and error. AI and scripts written from massive experience are lightening that load, at least.

    So check it out. Find ways to make things manageable again. We all know things are slipping through the cracks as the volume of items reported and the number of locations they are reported from increases. Don’t wait until you have another emergency; look into the options and get something running.

    For those of you already using AI/ML, unless you implemented it very recently, you’ll probably want to validate what you’re using, at what level and how much the AI/ML solution is helping, and tweak or replace. This space is moving too fast for “set it and forget it” today. Maybe in a few years, we can.

    With the rare exception of something I use every day and think you might benefit from, I adhere to a policy of not recommending products here, so I can’t point you in the right direction other than to say that your vendors (data, SIEM and security) can offer solutions and/or suggestions. That, and “You shouldn’t wait” are my suggestions.

    And keep rocking it. Part of that data/log/alerting growth is all the apps and infrastructure you have helped put in and that are critical to the organization. So protect it with every tool available—and use that free time to work more miracles.

  • The Other Reasons for Password Management

    The Other Reasons for Password Management

    I try not to write about ongoing work—if it is important enough to blog about then it is important enough to write about in the work product, and blog about something else. But every once in a great while, the need overrides my simple rule. After all, zealous adherence to rules is not really a thing we do well in IT.

    I hit a couple of those while doing work on password management for a client. I mentioned a couple of important issues in the work product, but one was mentioned with the caveat that it was not something profound while the other was evident and necessary but with solutions other than password managers that could handle it. I didn’t belabor either point.

    The wider market, though—those who either haven’t yet addressed password management or those that are just now starting to delve into it as a corporate solution—probably don’t realize that these are two issues that an enterprise requires and that DevOps needs.

    The less obvious is the enterprise requirement. There is a lot of buzz out there about changes in the perimeter driving changes in perimeter security (if perimeter security is going to have any meaning, anyway). But the underlying change factor is not so much our perimeter going out to embrace items on the larger internet, but the growth of that internet. Yes, our perimeter extended to cloud or service providers (or multi-clouds or more than one service provider), but the enabling technologies making the internet and the web also extend our perimeter in less visible ways.

    Right now, in your organization, there are hundreds of unacknowledged accounts at your business partners. These accounts should be known, but they’re hiding in plain sight. Accounts exist for places that provide small things—the type of stuff that used to be paid for with petty cash. The store where marketing buys the one-off items, or the emergency backup account for when office supplies run out in accounting—accounts like that. Password management protects these accounts automatically (assuming it’s being used, anyway). There’s not much your organization can do to protect those accounts from being attacked on the server side—those are not your servers, after all—but password management would protect them from the user side by enforcing things like strength policies and change frequency rules.

    The more obvious but underrated one is secrets management. Most password management vendors do secrets management today, sharing in the password protection technology already implemented in the core product. And if you are a DevOps shop that is even a little bit automated, you need secrets management. Secrets management could be done elsewhere (and often is), but most organizations are using things like hardcoded credentials and API keys instead.

    My rare forceful statement: “If you are hardcoding credentials, API keys, SSL certs or other secrets, stop it. Stop it now. You’re DevOps, not ‘Here are the keys to the kingdom’-Ops.” Password management can do this for you without hardcoding. So can other products, but you get two solutions in one if you use your password manager for secrets management, too. A password management product that offers secrets management with both API and command line access is huge for DevOps. To use the product in any given DevOps scenario, one of those two will address the scenario.

    As always, keep kicking it. You are running the face of the company. Whether sales, marketing, PR or whatever, when people go to interact with the company (be they employees, partners or customers), it is almost always the systems you run that they’re working with. Also, remember to protect your baby. Password management is not a product-centric thing, but push for central implementation with secrets management. Drive faster with password management, deploy more with secrets management and worry less with both. I don’t generally plug myself, but I’ll come close because it might be relevant to you—if you’re interested, there is more about password management out there under my name, including my take on who you should consider.

  • Quick! Define DevSecOps: Let’s Call it Development Security

    Quick! Define DevSecOps: Let’s Call it Development Security

    For a good long while, DevSecOps referred specifically to vendors like Veracode that did static application security scanning, dynamic application security scanning, software composition analysis and some form of runtime monitoring (usually interactive scanning). Then we realized that DevSecOps was potentially a lot more than that and, like DevOps, we drove the word to encompass all things that shift security left.

    That’s where we are now. But that leaves the DevSecOps market confusing to a lot of people. I propose we rename the tools that specifically focus on code quality to Development Security—even though that name is not perfect and all-encompassing. It’s a better fit than terms in other spaces, like Password Management, which includes two-factor authentication (2FA), multifactor authentication (MFA) and, increasingly, passwordless and secrets management, none of which actually use passwords for the focus of their functionality.

    This way, we can all understand that DevSecOps is the mindset and methodology and Development Security is one piece of implementing DevSecOps.

    There are probably people smarter than I am who have come to this (or alternate) conclusions, and I’m happy to run with their solutions, too—this is just an attempt to make it less confusing for those who don’t live and breathe the security and DevOps markets. If a new person says, “I think we need DevSecOps,” they are in for a journey to actually figure out what they meant and if what they need is actually DevSecOps. We should strive to clarify that so, at the beginning of their research, it is clear that DevSecOps is like DevOps—an overarching term for policies, procedures, mindset and products while the tools that can help with DevSecOps, like those that make up the overarching DevOps toolchain, belong to a market segment named X. In my case, I proposed Development Security because it’s a close fit. What we actually call it is irrelevant, as long as new people understand it is a subset of DevSecOps.

    While you continue to keep the systems spinning and the organization online, when you research these products, use phrases like, “I’m looking for Development Security tools that fit with a DevOps environment.” Markets are largely named by vendors and pundits but, in the end, all these people want to talk in terms their customer base recognizes. So, if enough of you ask for Development Security, that’s what they’ll all start calling it. Effective immediately, that’s what I’m calling it. It is a distinctive set of functionality that all orgs who do internal development need, so let’s give it a distinctive name. And yeah, drop me a line if you have a better name. I’m tied to making it easier, not to a specific term.

  • At Some Point, We’ve Shifted Too Far Left

    At Some Point, We’ve Shifted Too Far Left

    Those of us involved in DevOps have a tendency to see the world with blinders on. It is rather easy to fall into the “If all you have is a hammer, everything looks like a nail” trap. We have used the phrase “shift left” with the attitude that this is the solution to every problem in IT.

    Just for clarity, it is not. That is as blunt and unambiguous a statement as I can make. DevOps is great and has helped in many areas, but it is not the only solution, and not every problem is a DevOps toolchain problem.

    In the end, much critical work in IT is not DevOps-related, nor can it be impacted by DevOps. I’ll mention two areas that are obvious and talk in-depth about one of them.

    First off, no matter how far down the Agile/DevOps trail you travel, the wagon you are riding in is separate and distinct from the DevOps process. If you have a data center, all that hardware is still, in the end, hardware that must be configured and managed. For the “But we’re 100% in the cloud!” crowd (who say that loudly and forcefully enough that I suspect they’re trying to convince themselves that this is a good thing)—in the end, you are still using a network and computers to connect. The massive work-from-home migration due to lockdowns made it abundantly clear that IT is needed to set up and support both. Some of the horror stories coming across ops and DevOps social media were … ugly. Employees don’t want to be—and should not need to be—IT specialists to get their job done.

    The other easy one is security. Yes, insert security into development and DevOps workflows. Yes, make operations staff more aware of security. Yes, automate security as much as possible because we’re chronically short of security staff. But, in the end, lots of security is external to DevOps. While Kubernetes security is absolutely DevOps, password management, for example, absolutely is not. And that’s just one of many, many examples.

    So I am going to ask (but not answer) the question: Where is the line? How far left is too far? The answer is yours to decide with regard to your organization and the tools/environment that exist in your organization. In a primarily cloud environment, the answer to this question is far different than in a primarily data center environment. It is rare that any organization be purely one or the other at this point in time, so there are similarities, but few exact solutions we can name.

    Security is akin to a cost center. It is actually insurance, but few orgs view it that way. And even if they did, it would not make much difference. Much as my father once declared that he and Mom were “insurance poor” and canceled half their policies, most orgs want to contain the costs of security whether they admit it or not. When ‘security merged into DevOps’ hits the ears of people trying to stretch budgets just as far as they can, bad things tend to happen.

    DevSecOps implements security where it should have been all along—at the development stage—but it does not replace other security, it just makes our apps more protected. There are cases where security testing hours are seriously reduced, but since testing in all forms is the first thing to go when staffing is tight, again, it is not replacing something that was happening—it is making it possible for us to do what we should have been doing all along.

    It doesn’t help that our toolchains have taken on lives of their own—at many organizations, staff are required to maintain them—because they, too, are apps that DevOps struggles to help. Once you’ve invested time and money in building that edifice, you want to use it as much as possible. So, many orgs want to shift left everything in IT, right into the toolchain octopus.

    You know your org better than anyone. You are the ones making it work day in and day out, in all of your environment’s glorious complexity. Start talking about what is too much; when adding a step in an already complex toolchain needs to be justified. Push back against everything needing to shift left (where appropriate, I’ll say again for clarity, DevOps has done amazing things for us, but too much of a good thing is our living nightmare and, historically, is tomorrow’s technical debt). Keep rocking it, and make sure you are rocking the right things in the right places.