devopsy.kinsta.cloud

Author: Don Macvittie

Security Across the SDLC

We’ve seen some interesting movements in the security space over the last couple of years. My focus is on application security, but one of the movements we’ve seen is the blurring of what is application and what is infrastructure security, so I do a fair amount of that also. And then, with the advent of WAAP, we have network security playing a role in AppSec… It’s a growing snowball.

I will start with the mandatory happy declaration that DevOps no longer treats security as an annoyance. While we’re well past that, I think the shock of the whole “security is a problem, not protection” mentality that DevOps started with is still hanging with us, and some still actively believe that. But at this point, we’re well on the “Security is part of the process” trail we should be on.

If you look at the top-level changes, static application security testing (SAST) vendors have grown to be development security, covering SAST, DAST, IAST, RASP and even a bit of data security. Meanwhile, load balancers and WAF slowly evolved into full-on web applications and API security that covers the production side with active protections for things like DDoS and DLP.

Likewise, we’ve seen this with the growth of cloud workload protection and how the entire digital architecture is being pulled into CSPM.

I’ve said it before – we’re in a consolidation phase. It looks like it will go even further than I thought, though. Now, we are starting to see an interesting development: WAAP vendors are starting to include some or all of DAST, SAST and RASP. Arguably, they always did some IAST because their focus was on runtime protection, but their emphasis is more on protection than test, so it will be interesting to see that bit expand also.

And all of this includes an inconvenient fact. We don’t have any more people to do security than we ever did; in fact, we have less. That’s part of the reason that we haven’t shoved security everywhere – there aren’t people to do it. In true DevOps fashion, we have distributed the work across other teams – Dev is right there where code security and API security start, so we’ve implemented SAST and DAST bits right into the IDE, for example. Or Ops has always done some amount of security, and we’ve just upped the amount. But they don’t magically have more time, either. I’m sure you are feeling it if you are a practitioner; almost all of us are. AI promises to help with that, and in security, it will likely deliver. Another thing I’ve pointed out before is that security is one space that really does stand to gain from generative AI – in helping make more secure code, in helping make all the permutations of tests, in making rules based upon observed data and more. The time-saving benefits will deliver, in my opinion. But it’s still a lot of work, and some of it is straight-up additional work because we weren’t doing things like container security a few years ago.

In the short term, if full SDLC security is applied, there will be more work. In the long term, the mega platform for security may well reduce this work as it limits notifications across tools. Maybe we should name the coming mega platform SecArch. Also in the long term, AI will help alleviate some of the load. Until then, buckle in. You’re not likely to get staff to cover new work. But it will be worth it. Just keep plugging along, and know when to say you just can’t add one more thing to the pile.

January 31, 2024
Agile is no Excuse

Did you ever hop in a car and just start driving with no destination? I know someone who used to do that for relaxation, but while I do enjoy a good road trip, the destination is far more important for me than the journey. My Mom lives nine hours away, and I love to go visit her. I know the routes like the back of my hand and just get to enjoy the ride—precisely because I know where I’m going.

Most people wouldn’t sign up for a cruise that refused to tell them what stops were planned or even where the ship was going. But those same people are currently in a boat built by either IT or vendors and they have no clue where that boat is going. This is an issue with the software industry, not with the people in question. I recall a couple of memorable cases (not naming names here, but you no doubt were a victim of or heard of at least one of the examples I can think of off the top of my head) where a vendor suddenly announced they were changing everything, and customers would have to deal with it (moving platform, changing licensing, suddenly becoming a product in a different market …)

That’s crazy. Honestly. You could have a piece of software central to your DevOps operation suddenly change to be less useful or even unsuitable. A company you have a contract with. The advent of “commodity” virtual hardware and everything-as-a-service has made us lose sight of the fact that customers have a need for stability and are not in the least wrong to demand it.

And that brings us to today’s actual topic. More and more vendors respond to “What’s on your roadmap?” questions with, “We’re agile, so we are constantly improving.” That is a non-answer. I am a human, so I’m constantly improving, too. Weaning from a bottle at 18 or 24 months says nothing about the roadmap from today for the next 18 months. As an analyst, I generally start advising users that this stance is bad for them and they should demand any vendor they do business with have a plan that they can count on for at least the next 12 months. And that they should leave vendors that don’t have that, and not sign with ones that won’t if not already a customer. Agile means you are making constant changes. We didn’t ask how often you update; we need to know where you are steering your product. If you can’t answer that question, no one can, and the product is in jeopardy.

The problem is, because we, too, are agile, a lot of IT shops are doing the same thing to their customers. While the feedback loop is shorter and more direct, the fact is that most of you have surprised business users with some new thing. The fact that they often are thrilled to have it doesn’t change the fact that they should have known it was coming long before implementation. Agile doesn’t excuse bad communications, and DevOps is not short for ‘random change on a whim.’

So, demand your vendors offer you predictability – even in a fast-changing market – and offer predictability to your users. It only takes a second to update (automated or via email, Slack, whatever) customers that a change is in the works but not assigned yet. Or even just got assigned; just offer some warning. Be good to your users, and demand vendors be good to you. Agile, DevOps and SaaS do not require unannounced changes (good or bad), so stop acting like they are acceptable when notification systems are already set up.

In short: Stop saying, “We don’t know where we are going, but we’re Agile, so we are getting there super fast!” And stop letting vendors say it to you. You are knocking it out of the park every day. Systems run and people use them and all works as intended… Because of you. Now consider letting users know when changes are coming – not just the big ones, but the ones hidden in “We’re Agile!” Make your vendors do the same to give you peace of mind, too.

January 24, 2024
No, You Don’t Want to be a Prompt Engineer

Prompt engineering became a thing in 2023 and will stop being a thing in 2024.

We’re already seeing signs of its decline, but let me set the stage for those only vaguely familiar.

A prompt engineer is someone who knows how to get an AI (Normally a chatbot, but honestly, any of the raft of generative AI solutions out there for art, literature, code generation, etc.,) to do what is necessary. While at first, no one was really thinking about this, it quickly became clear that getting a quality result required some massaging in the same sense that getting a quality result from a search engine requires a specific ordering of keywords and limiting of possible results. So, as time went on, those who were better at it started being referred to as ‘prompt engineers.’ I am not certain of the genesis of the phrase, but it honestly doesn’t matter for this discussion. The name became a title, and it got popular; there were some crazy claims about exorbitant salaries for a wordsmith that made it attractive to many.

AI Tools

For some of my work, we use subcontractors to do technical research. Normally, we use technical people and have them turn out results we know will be topically close, even if they didn’t understand the specific technology when they started researching for us. On occasion, we have tried people with less technical backgrounds, and the results are definitively lower quality. With the advent of decent AI tools, we have done some research on using them for this work and have discovered that they allowed less technical people to generate roughly the same quality of results as the more skilled folks.

What this approach doesn’t do is raise either of these researcher types to the level of being able to do something with the results or understand the market well enough to determine results. And we see this across the board. In our development/scripting work, we use generative AI to help, but it doesn’t write finished code complete with the context of our tools/environment. We do that afterward, along with validating the code it does produce. My other business is in publishing and uses a lot of art, so we have contacts with a fair number of artists. Our art director uses generative AI for inspiration but not creation. She asked the other artists we use to follow that example at least until the courts work out IP law about generative AI, and across the board they came to the same conclusion. Some artists are using it for personal/learning purposes, but most AI-generated art is not being generated by professional artists. That has the usual implications for professional artists, but they’re used to being under-appreciated and being underpaid for their work … whether it’s an AI depressing costs versus an amateur doesn’t matter much in context.

Learn it and Use It

Prompt engineering will steadily become less necessary due to simple forces. Just like people learned the ins and outs of search engine prompts, they will learn to do so with AI in their space. As the AI gets more training/specialization, it will get easier to use, and at the same time, it will get more accurate. So, my advice is the same as with all automation tools. Learn it, see how it fits into your organization and use it to your advantage. But it’s not a career; it’s an automation tool. As time goes on, it will be able to do more and more, and maybe one day we’ll be able to say, “Hey Google, write me a CRM that works with our billing system.” But for the foreseeable future, focus on coding and use it to do some of the grunt work.

The only caveat I have also offered before. Go out of your way to hire and train good entry-level people. The weakness of AI doing a lot of the grunt work is that the “learning period” is shorter, so you do need to consciously say, “What is necessary to debug and/or manage our systems?” and make certain your entry-level people get that – because the day-to-day that they have traditionally trained in is far smaller.

And use it to create and maintain even better systems in the new year! I am stoked about the possibilities it opens up, and you should be, too. If I don’t have to write and debug that for loop, just generate, review and test, I’m thrilled. We should all be happy to use that time on more productive things than entry-level code.

January 17, 2024
New Year, New ‘Tude

There are multiple types of debt in an organization. In IT alone, you can have technical, ops and (my favorite) attitude.

We work in a high-stress field with a lot of moving parts and an enterprise depending upon us to keep things running. Even if you are in the product development org at a software company, you literally make the product; people are more dependent on you doing it right most of the time and being super proactive when things go wrong.

All of this creates stress that we tend to either internalize or leave to find a new place to get our stress.

Internalizing makes for jaded IT folks. Or worse, cynical ones. Vacations are part of the salve to relieve that stress, and we just got a couple of weeks off. Or at least most of us did, some of you all are about to, and others will in February. But it is all the same.

To continue improving, we need to leave the baggage behind. “Blameless IT” improved how others judge devs and operators that make mistakes, but we’ve done little to improve when devs and operators see problems that they cannot fix and it festers.

When you return to work, take a deep breath, consider what you can do something about versus what you cannot. Then, take action on the things you can and ignore the things you cannot. Improvement is what matters; improving a specific broken process isn’t. Of course, it will eat at you if you pay a ton of attention to something you cannot change. I have always taught those working for me (and my kids, for that matter) to change what you can and not dwell on what you can’t—precisely because you can’t.

If the thing you can’t change is a show-stopper and makes your life miserable, then the thing you can change is your job. Start looking.

Between Agile, DevOps and new deployment methods, we have come a long way in technology, but we also have created a ton of work for ourselves. This stuff doesn’t implement itself. Focus on that work and on how you can make things better – for the org, for IT, for fellow developers or operators. And do that. Always improving and not dwelling will make life happier for you and generally more productive for your team.

I’m excited for a ton of things that 2024 is bringing, and a couple of things that aren’t necessarily ‘2024 new,’ but that I’m still figuring out. Let’s do new stuff with a smile, because we know we’re making things better/faster/more reliable. And implement that feature your users all want and your team hasn’t done yet, while you’re at it. Users are the reason we do this, in the end, be they internal or external.

We made it through 2023 with some great improvements—no doubt you did, too. Now, it’s off to 2024 to do more. Keep rolling out features, protecting the systems, deploying to new targets, etc. And forget about that stupid annoyance. (Until you can change it—then crush it like a bug). Meanwhile, thank you from all those that don’t realize they add to your stress every day.

January 10, 2024
Shared Responsibility? Yeah.

One of the things that the last decade or two of enhancements has brought about is an increasing reliance on everyone across the organization to be more secure. While users have long since thrown up their hands and looked for ways to minimize the fallout when a vendor they frequent is compromised, enterprises have to lock down their own things. And given the ever-increasing volume and sophistication of attacks, the demands are growing.

Cloud brought us the concept of a ‘shared responsibility‘ model. It is necessary in a world where an org does not control the physical servers and supporting infrastructure that applications ultimately run on. The fact is, the shared responsibility model was really cloud vendors reminding us that we still have to protect our apps, no matter where they’re deployed.

This model and DevOps together have driven us to demand more of developers and DevOps teams with regard to security. It has also driven ops to have to care more, as more is exposed and security demands are up while security staff isn’t.

And I’ll just remind you of the obvious. If you are responsible for security at the organization, you can get help from everyone, but those people’s job and direct responsibility is not security. They’re lending a hand because everyone cares, and that is the current state of most enterprises – DevOps made everyone involved in the SDLC a bit more responsible for all the things that aren’t their jobs (often a very good thing, often … not), but they’re not specially trained for it. That’s what security teams are for. And high-quality development security/policy management tools offer Dev and Ops and everyone in between a bit of guidance to good security practices along the way. I’ve praised putting security feedback right in the IDE half a dozen times here and elsewhere, and as a security-conscious developer, will continue to do so until it is so pervasive that people start to worry I’m losing it by going on about it. That’s a great example of helping everyone be more secure without requiring a ton of security staff hours in training. Developers don’t purposely write insecure code any more than they set out to make buggy spaghetti code. Giving tools to help them make the code more secure is a huge bonus – to them and to the corporate security team. So, if you’re not using them, start. And look into all of the other options to help non-security staff lock down their bit. The number of available security people on the market is perhaps the largest it has ever been (It’s so bad I actually know a security researcher who took months to find a new gig – that is pretty much unheard of or a unique case before recent days), but budgets to grow the security team aren’t there – any more than they are currently there to grow most teams. So tools to help are still imperative because you’re not going to get double the security staff in 2024 unless there’s only one of you … and you threatened to quit if they don’t get you help, which no one recommends as a negotiation tactic, so even if you’re solo, no doubling for you.

Just remember who is responsible for overall infosec. Make use of the specialist and their knowledge in their space by finding tools to help them secure in a manner that matches workflow. Then verify because the organization’s security posture is security’s job.

And keep rocking it. We’ve got so much cool stuff going on at the moment in IT in general that there is plenty of opportunity to improve architectures that are even relatively recent. Check what you have, consider it from a security perspective, improve what you have and keep rolling along.

But first, relax, take a breather, and enjoy the holiday period.

December 20, 2023
Things To Do During Slow Times
We all know it happens in North America; once the U.S. Thanksgiving holiday hits, things start to slow down in IT—from vendor salespeople busy with family to coworkers out of the office in larger numbers. Since all regions and most religions have holidays clustered around the new year, the worldwide slowdown goes on for a good month and a half, sometimes—and in some locales—even longer.

And how we experience that slow time varies pretty widely. For some, it’s time to be completely off, like those coworkers. For others, it is time to finish those tasks that have been simmering on the back burner. For still more, it is a time of frustration as the things they’ve been working on grind to a halt. Back in the network computing days, we would do large, complex technical evaluations of the gear the vendors sold to enterprises. I covered mostly storage and servers with some security at the end, but my lab had more security, some software and some networking. We needed vendors to participate, and participation mostly meant traveling to our labs for any physical product, to set it up and let us test it. Needless to say, this slowed to a (sometimes frustrating) crawl as people reduced travel closer to the new year. I’ve felt that same frustration as an architect—when we couldn’t get vendors to respond and you really can’t blame them for doing the best they can in a time of seriously reduced staffing.

So, what can we do that doesn’t tie us to others who may or may not be available?

You know the list specific to you far better than I do, but here are some general suggestions to ponder. Most of these I have done during slowdowns.
- Learn that new language or framework. We all have one that the org is interested in, but in which we are weaker. Pick it up in those quiet hours; you can slap on headphones and focus with fewer interruptions.
- Same with new tools like test generators, DLP, GitOps or whatever your org is considering implementing. Heads down; figure it out in a test bed, and become at least versed and, at best, an SME.
- Clean up all that cruft you know you left behind over the past year. We’ve all got dead-end branches, stray text files, outdated bash scripts … Dynamic environments spew extraneous artifacts more than historically was the case. Take that time to go clean it up. I had a large script a couple of years ago that spun up a bunch of containers and ran tests. When I went to clean up, there were three different versions of the script and several orphaned containers still running. Good to get taken care of.
- Research, research, research. Whether it is something like reading a product’s site, checking crowdsourcing tools like G2 or looking at what some of the better/smarter analysts out there have to say, pick the tool you think would benefit the org, validate that it will and maybe even start documenting the use case and ROI justification (if you use ROI for consideration of new products; benefit analysis if not).
- Catch up. While generally the least fun use of this extra time on the job, it is often the most practical. We all have a ton of work, and sometimes the peace of mind that cranking out backlog offers is worth not doing the other things on this list.
- Take the time off. Seriously. Most of us (though no longer all of us) got into IT because we loved the technology and the promise. Walk away, take a breath and don’t even think about it for a week or two. Then, come back; I bet a bit of that wonder will return, making the job more fun and you more adaptable. It works if you can manage it. As others have said for many years, walking away and rediscovering a hobby can be invigorating, too.
Don’t sweat it, and don’t totally waste it. Figure out the best use of that time and move on it. There are a few spaces that simply do not slow down for holidays, and for those people, this blog is unlikely to be useful now, but at some point, slowdowns will arrive. I was a team lead at a tax software company. We slammed the holidays prepping for tax season, then got short weeks all summer. The mental checkout of teammates was similar to the end of year but delayed until early May. So we did all of this then.

But no matter what or when slowdowns happen, you’ve knocked out another year of keeping the org running. So, celebrate in some way; you deserve it.
December 13, 2023
2024: The Year of Testing

Caveat: I currently cover DevOps – including test – and security – including the *AST. This does give me a viewpoint that others may not have, and also potentially gives me blinders that others may not have. I am not generally a predictor but a reader of direction. Proclamations like this are rare for me and are shorter-term/more tactical than the grab bag of predictions usually seen this time of year.

I was thinking about this one this morning; the direction of travel of traditional and security testing is very similar at the moment. We are seeing the vast complexity of the overall space and the rapid growth of automation come together to once again make all testing a topic worthy of consideration.

I loved test-driven development (TDD). I can’t speak for others, but RPC/API is just a cool way to implement centralized functionality, and TDD is almost mandatory to develop APIs and the clients that call them. There is no UI to distract or have separate requirements in API development; there is a stated function and correct or incorrect behavior. To a computer scientist, this is predictable beauty. TDD made it more than predictable—it made it a trail through the woods. “We know it needs to do X, Y and Z, with the only side effects being A,” can be detailed with tests to prove that is what it does, and then look to prove the “It doesn’t have other side effects” part. And you can do it all while developing—no need for dev/test iterations. Like any IT solution, it isn’t a great fit for everything, but even if it isn’t called TDD, it is nearly impossible to develop API-based solutions without some level of TDD.

The thing is, TDD is absolutely a developer’s tool. Developers and DevOps pretty much must use it to turn out usable APIs. The test team only has a use for a subset of those tests. Because if the dev team did it right, they passed all the TDD tests, and the test team’s role in that scenario is to look closely at what the developers might have missed. Integration is often beyond the control of an individual dev and/or API, for example, because many different clients may be integrating with it. Test teams, in a true TDD shop, fill the gaps with meaningful tests that evaluate the overall application quality.

Security testing—at least static application security testing (SAST) and increasingly dynamic application security testing (DAST)—are also becoming a core part of the development process. This is as it always should have been. It is far easier to fix a security flaw (or any flaw) at the design/early code stage than in a testing process after deployment. The quality of security testing inputs is also increasing, making it far more viable. And finally, the tools are being effectively integrated. Being a developer who did a lot of security work because we often needed it and had no one to do it, I will say I get a little rush about being told right in my IDE about the security risks in my code. I write, I see, I revise where necessary, and I turn out better code. All without having to pause for security or bang heads over policy.

To date, all this testing is separate, but I suspect, having vaguely the same location in the process of development, that testing will all occur together and the markets may even start to merge a bit—though merger of security testing and functional/performance testing tools is a bit of a stretch, it is more possible than in the past. In most orgs, even though developers have an increasing stake in both sets of testing, the target purchasers are still two different groups—security versus test—with different priorities. That’s why I think they aren’t likely to merge any time soon.

While AI in most technology spaces is currently experiencing the same level of FUD as XML, Java, JavaScript or low-code was in the past, all super-hyped tools in tech do have a sweet spot. Security and testing are in that sweet spot for AI. Both in test generation and in results evaluation, it does it better, faster and with more coverage while giving staff a concrete set of results to pore over and improve security posture and/or code quality. I cannot say loudly enough that in terms of enabling the steps in the SDLC that have suffered from a lack of resources and timeline to write, execute and evaluate tests, AI is a game-changer.

So, all of these things together mean that 2024 is a good time to take a hard look at your testing environment(s) and architecture, both traditional testing and security testing. Now that AI has made advanced automation a fact, it is time to consider implementing the level of testing we always knew we should have but never had the resources. Build it right into the build process and make it part of the organizational DNA—just like Agile is.

If you could hire a specialist who would paw through your code and give you a list of the 25 things you need to fix first, you would do it. That is, essentially, what modern testing tools are. They can do more tests, evaluate more issues and side effects and filter down to the “No—this is really important!” list without costing the team a ton of time.

And keep rocking it. We’re all sitting on bugs and vulnerabilities. I do hope we are way past denying that fact. Apps are big old complex monsters; these days, sometimes built from generations of technologies cobbled together. Of course they have issues. Even yours. So let’s take steps to clean it up, just a little bit. And keep clipping along.

December 6, 2023
The Trouble With SaaS (or, Nothing is a Silver Bullet)

It is intriguing to me that we have to have this conversation about SaaS and software updates yet again. There is supposed to be a balancing act between the desire of investors, efficiency for the org and the needs of the customer. Right now, that balance is (yet again) all out of whack.

I actually got to thinking about this doing work for my non-IT company. One day, I realized that between browser, OS and office solutions, we were updating and restarting something nearly every night. Then, we were forced to upgrade our accounting to a really great SaaS product. Considering that a product that has served both of my businesses so well for a decade will soon be replaced in our environments, apparently, it is excellent for investors’ mindset, not so much for customers. Oh, it’s easier to access, but it comes with a separate set of weaknesses that would cost my less profitable business a ton of money. Since I like to keep the solutions standardized between the companies (for the accountant, if nothing else), the SaaS product will have to go at both orgs. We reverted to their discontinued local install product until evaluations were done to replace it entirely.

The problem is that both SaaS and rapid iteration are great on paper, but like just about every good idea in IT, they are being treated like a panacea. Sure, recurring revenue from SaaS is great. But if you lose customers doing it, that may not work as well. The same is true with rapid iterations—once you get to consumers, updating even weekly is a bit of crazy-town. Most end customers look for a way to turn off updates as soon as they notice the pattern. Business users aren’t much better, we all have stories of systems or hardware that weren’t updated for years. Heck, I worked on teams that hadn’t updated their own systems to get them off of outdated OSes and hardware, let alone kept on top of vendor updates. This is part of the reason for the drive to SaaS – to force upgrades. But if frequent updates introduce errors for the customer, well, it’s a lousy reason.

Make your products good, and make certain you are serving the broadest piece of the market that makes sense. Of course, follow trends and make sure you are still doing what makes sense. But what makes sense is what serves the market and/or the organization’s bottom line. Everything else is input. Some will say, “In good SaaS, the user never knows if there is an issue.” Yeah, and I never wrote a bug. Seriously, don’t force things on customers; give them what they need, where they need it. In one (security) space I cover, my regular advice to vendors is, “Users need this one bit of functionality behind the firewall and stable. Give it to them, SaaS or not, or lose them.” But that sentiment is increasingly being applied outside security. Users want vendors and products to do for them, not to them.

In short, do what you like to see from your vendors. The convenience of a SaaS accessible from anywhere and free of capital expense is huge, but it isn’t always the deciding factor and, in many spaces, limits market appeal. You all know what is useful, and you know for a given toolset what is not. Apply that mentality to your own customers. And keep rocking it. Sooner or later, this deployment octopus we’ve created will come together, and we’ll have different problems to fix.

November 29, 2023
It’s All About Communication

Things get crazy, don’t they? I mean, IT, even with Agile and DevOps, is a very interruptive endeavor. When something in production breaks, whatever you were working on suddenly just got a lot less important. Agile is aimed at making more stable code, and DevOps is focused on shifting troubleshooting left, but inevitably, IT systems – be they ones we wrote or ones someone else did and we manage/maintain – are complex systems, and they break unexpectedly. As time goes on, things like test-driven development (TDD) or continuous testing have steadily improved quality and reduced interruptions, but we still say things like “Oh, the container crashed, spin up a new one and keep moving,” when that crash should be a work-stopping event. If it crashed once, it will again, and it only takes a few minutes to find some of the horror stories out there caused by doing exactly what I just described. There is a line between “resilient application infrastructure” and “We ignore critical failures because we can.” And one side of that line is pretty ugly.

So, with all of our advances, we are still facing unexpected issues cropping up – sometimes caused by coworkers or vendors – that we need to deal with.

One thing to remember is that this, too, is part of your job. Start there. While most of us aspire not to be “the fix-it guy,” if you wrote the code or managed the infrastructure, you are the right person to fix the unexpected problem. So, no stress, but others (like management) might be stressed themselves, so communicate clearly. A five-minute email or thirty-second Slack message can save a ton of frustration and meetings later. A simple “This came up and is a priority, so that thing I was set to deliver tomorrow will be delayed by at least X amount of time,” will do a lot to help others understand the impacts on your daily flow.

I’ve worked a fair amount of my life as a contractor, and I admit that I honed the skills in the above paragraph living that life. While contractors are generally paid by the hour, ownership is a bigger deal, but in the end, contractors are there to do what needs doing, and the client decides exactly what that is. The same applies when not a contractor, though also, and in my experience, it applies double or triple if you are in charge of a team – you have an entire team, and you better communicate if an issue so big it side-tracks deliverables occurs. We had one of seven inputs to a massive data pool that was the source of billing fail on one team I managed. I mailed stakeholders in our other projects with a simple, “The company likes to get paid, so our attention is 100% on restoring that feed; your project is delayed.” This example was one (of two when I led that team) that was pretty extreme, but it’s a good example of effective communication; just let them know. Most business leaders are there for the org while focused on their piece of it, and if they have an issue they can hash out the business’s priorities and come back to you.

And keep nailing it. Errors and issues and vendor relations failures happen. You’re the ones minimizing the disruption and keeping the org rocking. Don’t stop or get discouraged when it sets you off your timelines; just communicate … and keep being the star.

November 22, 2023
It’s Funny How We Forgot About Container Sprawl

…Because it sure as heck did not forget about us.

We are (thankfully) past the days when everything was a container. At least, most of us are. Just like anything else, some orgs will continue that approach for reasons both great and terrible. And all of us still use far more containers than we need, I would say. “One container, one function,” works if you are talking about one whole application, and maybe sometimes for busy parts of an application, but it certainly isn’t needed to host a couple of low-use APIs, for example. My favorite example is separate instances of the same exact application for different teams (not business units – that is a good method to separate business units).

Thankfully, technology is once again saving us from ourselves and our misdeeds. Processing power is no longer an issue, meaning we can spin up a million containers and keep them running until our public cloud bill arrives. So, it’s not about processing power. I cannot remember the last time that an org was bandwidth-limited, so it’s not about the network. The biggest issue is still security. More containers means a bigger footprint, which inevitably creates more vulnerabilities. Right behind that is container inventory management. What is running where, what images are available and what state are they in, which running containers have drifted from their definition, etc.

I’m a developer that has sometimes done support for the apps I develop. That means that I laugh at your “Our images do not drift because we’re agile and build-driven.” Of course they do, because troubleshooting is real. The real question is, can you get them back into alignment when the ticket closes? The DevOps toolchain is pretty darned good at that if you let it. Changing a running image and telling no one is a terrible idea that still happens nearly every day across our industry. But technology has brought us drift monitoring. We can be notified of changes as they happen. For those in highly sensitive spaces like FinTech, changes can be blocked in production, but for most of us, we want to know when it is happening and who made the change. And we can get that today.

Technology has also brought us some wonderful scanning tools from SBOM to SCA to Registry. These tools can give us a list of things for security to worry about in our images. In order to reliably generate these lists, we still have to know what images we are keeping and where—but choose a reliable registry or two, fit them into your DevOps processes and scan on commit. Simple and thorough—as long as everyone uses them.

Thanks to policy and regulation, container management tools can offer you a detailed list of not just what instances they’re hosting, but metadata like what image they came from. That helps manage inventory, and by now we are all building hierarchies of images from base to application specific.

So, we can tell what is running, what images are available to run, how far running images have drifted, what known vulnerabilities exist in both our images and our instances and how compliant we are WRT containers. Dunno about you, but it seems technology has our back yet again.

You are cranking it out, keeping the lights on, and dreaming of more you could be doing. Use the tools we already have to make it easier and the resulting systems more secure and reliable. You won’t regret it. And once again for those of you whose users never say it: Thank you.

November 15, 2023