devopsy.kinsta.cloud

Author: Don Macvittie

  • Improvement Cycles and AI

    Improvement Cycles and AI

    It’s that time of year when pundits start to crank out their predictions for the next twelve months! I’ve been asked to contribute to a private group (and occasionally, Techstrong Group asks me to contribute to their annual list), so I won’t thrill you with my predictions just yet; but I will discuss the cycles and trends that previous predictions have shown.

    Back when I was testing physical servers, we thought of throughput bottlenecks in terms of where the problem was. The circle would go disk->memory-> CPU, and in the case of things like NAS, insert network in there somewhere, too. When one bottleneck was resolved, another would crop up. And users will always seem to be pushed to the next bottleneck.

    We see similar cycles in DevOps. First, Agile made development faster, and that caused us issues throughout the rest of the process. Then, DevOps helped with the build and a little with the testing and deployment steps, but in the early days it was more standardization than speeding up. And since then, we have been in that cycle. We’re seeing GitOps speed up and automate deployment and upgrades—based upon the automation that came before—we’re seeing test and security tools feeding directly into the IDE to speed resolution, we’re seeing container automation cause “cloud-first development” (I hate that name; Kubernetes has cloud-like interfaces, but is not precisely cloud) to be fully automated, and thus faster.

    And we’re looking at that cycle continuing. No doubt you will hear a lot of predictions that start with “AI …” For the most part, AI is a tool and/or technique. Listen for what it is improving because it is doing very little that’s actually new. (Delete list of things that sounded alarmingly like those predictions I said I wouldn’t make.)

    All in all, we are still living in a very cool technological time. We can have a tool generate code snippets of the dull work for us so we can focus on the important bits. Some shy away from this use of “AI,” but honestly, we already use libraries to do the grunt work—I haven’t written to a screen buffer in forever—because we don’t have to. Same story—use the tools that make you productive. Some of the more obscure bits of Linux command lines are far better dealt with by asking an AI to give me examples than searching through tons of docs across several platforms. Never was much of a fan of man output for complex commands, this makes it easier. The nice bit is that bash and UNIX utilities don’t change much over time, so answers can be a bit dated and perfectly accurate. That’s not true for much of IT.

    The environment has consistently gotten more complex, while the building blocks have gotten easier to use. This allows things to keep moving and changing while it offers us better ways to serve the organization’s needs. This does change at some point—infinite change is indeed a thing, but any technology does have its limits—but I honestly can’t see the point at which we are content with our IT tools and stop looking for more. Oh, most organizations go through those cycles—organizationally, once an infrastructure and process works correctly, it is likely to hang around until better is found—but the industry will keep changing.

    And you’ll keep learning and changing. That’s one prediction I will make for next year. Teams and IT staff will grow and learn new things, because at its heart, that is what we do. We learn new things and keep the lights on. In good years, we offer sweet new tools and functionality to the company or our customers. But either way, we keep things online. Year after year. You all rock, so that’s an easy prediction.

  • At Some Level, We’re Doing Security Wrong

    At Some Level, We’re Doing Security Wrong

    Years ago, right here in this blog, I pointed out that the worst of gamification was being applied to information security–play the game until you lose. I wish I could say we were past that, but we are not. And I think it is time for us to have a bit of a rethink (as my UK colleagues would say).

    Consider this. Twenty years ago, breaches were relatively common and we were concerned enough to enact laws about data, security, etc. Today, breaches are relatively common. In the intervening years, we implemented more and more security.

    The problem is that our environments—from mainframes to today—have grown increasingly complex. Our mission to make applications more useful and accessible makes them more vulnerable at the same time. And the changes are constant.

    If we were doing this right, breach notifications would be super rare, and we’d all want to know if it was a broken machine or human at fault—kind of like aircraft crashes are handled and reported in most of the world. Instead, we go out to a site like haveibeenpwned to see if we’ve been impacted, shrug and assume that the free credit monitoring service from the last five breaches should have us covered financially, and we move along.

    Internally, there is a post-mortem that closes whatever hole the attacker got through or exfiltrated data through and carry on, knowing full well that a different attack is coming.

    We are currently staring longingly at the bright lights of AI, hoping it can protect us from unknown vulnerabilities going forward. A bit of critical thinking will make it obvious that this is simply keeping up with attackers who also have access to AI and are using it to refine and launch even more complex attacks.

    The problem is that we all see the non-stop changes to attack vectors and architectures and don’t know how we could protect more broadly—and that is the appeal of trainable AI. We hope that it can rapidly generalize from its training to detect previously unknown attacks and, given enough inputs, can computationally detect attacks humans might miss in the sea of data. But that is still a point solution, in my opinion.

    How attackers approach the problem and what tools (both tooling and vulnerabilities) they have available are constantly changing, but the goals are unchanged. And I think we have to adapt our thinking to address those goals instead. Policy enforcement is a step in that direction—but, again, that means we’re protecting against the known.

    I don’t have the answers. I keep thinking of two approaches—airplanes are routed around a ton of threats to get their cargo to its destination safely, and that seems to be analogous to the problem. And hardened fortresses that literally ignore current mentality of “the perimeter is everywhere” and take the core, important bits of applications and lock them away even more tightly than we do today, with single-API access to a given bit and the API is monitored and analyzed constantly.

    But those are both sweeping, drastic approaches, and that may be our problem. Security spaces are merging slowly, but we still have a largely “That’s ops, that’s dev” mentality in security (as just one example of why sweeping overall protections are problematic). So, maybe with merges, the time will come that this is naturally resolved. But we’re not there yet.

    If you think about it, we’ve solved much more intractable problems, as the phone in your hand or the online game you enjoy spending a bit of time on demonstrate—we in IT and Dev are capable of amazing feats if we set clear goals and commit to both developing and implementing a solution. We just need those clear goals, and fighting today’s attack and hoping against hope another won’t come tomorrow just isn’t it.

  • Technical Debt: Don’t Buy Buzzwords

    Technical Debt: Don’t Buy Buzzwords

    It is absolutely true that the only constant in IT is change. No sooner do a mass of us get the ‘next big new shiny thing’ implemented than people start telling us how much we need the next ‘next big new shiny thing.’ Often, that next big new shiny thing is actually helpful, and we feel (sooner or later) compelled to implement it. Sometimes, the next big new shiny thing is not very useful at all, and yet far too many of us implement it anyway.

    Not naming names here, but you’ve all got technical debt that I would classify as “bling” or “keeping up with the competition” functionality. It’s there; you’re paying for it and/or to maintain it, and yet it provides little value. Sometimes, you’re even forbidden from removing it because it is some exec’s pet project or some corner department is actually using it. You’d just have to replace this version of the next big new shiny thing with a different version.

    And I’m here to tell you: Stop. When considering new technology, the very first thing to do is build a list of the problems you are trying to solve and then describe scenarios where those problems crop up. Then compare the next big new shiny thing–be it features in an existing product or a new space/product completely–to how it solves real needs. People are the absolute worst about this with open source. “It’s free, so let’s try it out.” It. Is. Never. Free. And it is a rare team that tries it out and then removes it if it doesn’t work. Most of the time, they’re off to the next project too quickly to do that level of cleanup.

    SBOMs and SCA are ‘next big new shiny things’ of the recent past that actually help you clean up test/pilot garbage from the latest bout of next big new shiny things. So are containers since you can spin them up, play with the toys and then take them down. I am surprised at the number of teams out there who do not do this, but that’s changing pretty quickly. Use them; it’ll save a ton of cleanup time and a lot of technical debt.

    But most importantly, only buy what you absolutely need. We’ve been dropping an ever-increasing number of tools into the development environment for a decade, and those tools do not (yet) maintain themselves. There is infrastructure cost, possibly licensing cost and staff time. Don’t waste it; focus on what is literally most important.

    And that might mean putting your foot down. We in IT have been loathe to do that since Agile started, but telling a C-level that they don’t need an AI to create new drink recipes (or whatever the current generative AI boardroom craze is) is in the best interests of the company. Focus on systems that help do the work or that protect what does do the work.

    You’re still rocking it, so if rocking the boat a bit in the process causes problems, you might just be at the wrong company or in the wrong role. Achieve, but achieve meaningful results.

  • Generative AI: The 90% Rule

    Generative AI: The 90% Rule

    There is a concept we need to solidify that I will call the 90% Rule—until someone smarter than me comes up with a better name. The 90% Rule is simple: Generative AI will be able to do 90% of the work in areas that it is good at. Right now, that would be writing, but coming up soon will be image generation and coding and, not too long after that, video generation. For those of you buried in the space who are offended, I’m not rating it as advanced as it is for specialists; the point of generative AI is not to let AI specialists do things, but to let average people in the space do so. And that list is representative of the current space.

    This is just based upon the state of things at the moment I wrote this blog, but that seems to be the immediate order. I’m only talking generative AI here—AI for things like filtering test results is a completely different beast (that might just meet that 90% rule … Hmmm). I can tell generative AI to write a story, a script or code for me, and tweak the results. The amount of tweaking certainly depends on the topic and how much time I spent zeroing it in, but 90% seems right. There are those looking far ahead to a time that will herald the arrival of our AI overlords, but for the foreseeable future, we will not be ditching authors, coders and ops people. Our jobs will just change.

    Crikey—given the last few decades, a change to our jobs that means less work is not a terrible thing. Today, writers have to be publishers and marketers, coders have to be ops and ops have to be coders and the entire group has increasingly had to be security and architecture specialists, too.

    Let’s celebrate a tool that says, “Here’s a start, make it useful in your specific situation.” I know I will. I sometimes contract people to do initial research for me and send me links and short descriptions. I encourage them to use generative AI to do so. I do not contract people with my level of knowledge for this work, so they’re going to have things I need to read carefully and follow the links for—they’re not specialists, after all. Generative AI suffers from this syndrome, too, but that means there is no difference for me and they can work smarter, not harder. It’s not like they are writing content; merely providing introductory research through a scan of web properties.

    As time goes on, I expect they will become specialists. But not in the things I pay them to research—in making generative AI do what they want. I would argue the person I contract with the most often is better than me at it, just because she’s doing it regularly. That skill set will inevitably be transferrable, and I may lose her to a full-time position making the I in generative AI actually mean something, and I’m great with that. Meanwhile, I’m getting the background info I need to do my job(s).

    Embrace it. Aim for 90%. Turn out even more rockstar-level apps for the org, and maintain them with less effort. This is where we’re headed, and it’s entirely possible you’ll be less buried in work for some of that time. Win/win. Go out and enjoy. The business will thank you for being even more productive … then waste that extra time you created by implementing new projects. But that’s in the future; use AI in the short term to get some breathing room and still be the force behind the apps.

  • Unused Code: Proving a Negative

    Unused Code: Proving a Negative

    We have a ton of code that is no longer being exercised. We really don’t talk about it much, but it’s out there. From feature flags shutting off code to API versioning making an interface obsolete, it’s just hanging around out there. Like an albatross. Waiting for someone to misuse it. Or, to address the more pedestrian concerns, confusing people new to the codebase. But that code is not contributing a thing.

    Easy example. You are on version 3.2 of your API. Version 2.1 is not called by anything, but the code is still hanging out there. It is not that 2.1 cannot be called, but that it is not currently called. The problem with that is that new devs have to work through what’s current, and since 2.1 and 3.2 probably share 95% of their code bases, what portion of that code base is unused? And the API is sitting there, waiting for someone to penetrate your network and find malicious uses for it.

    Proving a Negative

    But here is the problem: Identifying that a bit of code–be it an API, a method, just a few lines in an if statement, whatever–is never exercised is terribly difficult and, in highly complex environments, borders on the impossible (it is never impossible, assuming IT owns all of the code that could exercise that bit). The old adage about proving a negative absolutely applies. We can–and testing tools in both the functional and security spaces do–generate a list of code that is exercised, or identify code that isn’t being exercised. But not a list of code that is never exercised. That functionality in test tools is super useful, because it helps prioritize issues that are being used a lot and leaves the ones not seeing a ton of traffic for later resolution.

    But I’ll say that in most organizations, that’s no longer far enough. The cruft in your code causes problems, and you need to clean it up. If you’ve worked in IT for any length of time, you’ve been through the “Oh, that bug is because the developer called something no longer used” scenario at least once. And it is a straight-up waste of time.

    So here is my proposal. Test tools generate a list of what isn’t called from systems under test. Pick one–security testing or functional testing–and require it for any bit of code, no matter how small, to go into production. That means that the system knows about all source code and thus knows definitively if a bit of code has been exercised.

    Clean Up as You Go

    Is that a tall order? Yes. Testing isn’t free. But given a scenario where you reduce technical debt by eliminating excess code, I think it will pay for itself. And that is not counting the security posture improvement that reducing the code base implies.

    You’re running the systems that present your organization to the world, so you may as well do some due diligence on yourselves. Keep making cool new stuff and upgrading the old, but clean up as you go with some form of monitoring, this or a more procedural one, so the quality is high and the excess is low.

  • Steady On. We Still Have a Job to Do

    Steady On. We Still Have a Job to Do

    It is all the rage to talk about how significantly large language models (LLMs) and generative AI will change every niche and corner of IT. And change IT it will. But the hype cycle is at its highest right now, and I’d caution against getting too worked up about it. We’re in for a few interesting years, and the spaces I cover (DevOps and security—with a more application-centric perspective but with ops experience) are definitely going to benefit from the various AI bits being worked in. All testing—including AppSec, but all testing—will benefit both in test generation and results analysis/filtering. In fact, I’ll go ahead and say that the question in testing will no longer be, “How much time to we have to devote to it?” and will rapidly become “How many systems’ resources do we have to dedicate to it?” It will still be a long process—AI isn’t speeding up the cumulative time it takes to run thorough testing (be it functional or security). It will make that the bottleneck though, because test generation will at least be LLM-assisted, and will eventually become completely generative AI-created while at the same time, results will get more and more filtering and analysis. A minor result here plus a minor result there will automatically be detected as an extended vulnerability. There are a ton of other possibilities, but these two seem pretty obvious when I look across the testing technology space. More testing and better results analysis are on the horizon—without requiring more staff. Of course, you can do so much more, which would require the organization to add staff to follow up on results, but that’s nothing new; testing finds issues that are interruptive and take man-hours to fix—we all know of bits in the application portfolio that we should probably fix but that aren’t high enough priority to do so, for example. But fixing is different than finding. I have seen demos in security scanners that offer auto-fix options for developers. Run the test during CI, create a ticket and let the developer see suggested code rework right in the IDE. It is early days for this, but if you look, you can find vendors doing it. And that’s really rather cool.

    But we’ve all heard the “this changes everything” storyline over and over. 4GLs, XML, Java and JavaScript were all going to eliminate developers. In fact, they all created more developers. AI has a better chance of actually removing parts of our workload than any of these technologies did, but we’re still in “watch and wait” mode. It seems obvious that grunt coding will fall to LLMs … And aside from keeping staff trained to manage source, that is a good thing. We use open source, modules and libraries to do much the same thing today. I see this as equivalent to entry-level writing jobs. We’ll have to transform from the current model, but the current entry level development model wasn’t ideal anyway, so lets make a better one. Testing capabilities will be greatly enhanced, but testing of all kinds was so far behind new code generation at most orgs that the impact will be all positive as testing catches up. AI-based test generation today can generate thousands of permutations for a given piece of code, and analyze the results to just have analysts look at the truly disconcerting or confusing test results. And the question will become, “How many servers do we want to dedicate to run this?” Which is different, but just means we have more tests running to make a more solid and/or secure product and environment.

    So, in summary, is change coming? Yes. And that is not significantly different than the last couple decades where change has become IT’s constant companion. For most of you, the way you do your job today is not the same as the way you did it even two years ago. ML/GAI/LLMs are just another iteration. The apps still have to be built, deployed and maintained. So until HAL comes along, keep kicking it. You’re keeping it together, and who knows what the future brings.