devopsy.kinsta.cloud

Author: Don Macvittie

  • The Security Pipeline

    The Security Pipeline

    Over the last few years, the ability to secure our applications has grown, and deep integration into the DevOps toolchain has, too. There are more tools doing more security checks protecting more of the infrastructure and source than there have ever been. The key is putting them to use intelligently.

    We now have the ability to secure the image an app is based upon, the images that are derived from that original—complete with supporting software. We can get feeds to tell us which bits of our included open source components are insecure and how to protect them; we can scan the source, the assembled application, the APIs and the network. We can put runtime protections in place in front of web apps and in front of APIs. We can even track the long-tail attacks like slow data extraction that are becoming more common as security tools master more obvious attacks.

    That is a great set of security capabilities. And automation puts every bit of it within reach. Your job is to pick the bits you need to increase the security posture of your environment and get them worked into your specific toolchain. This is not all that difficult, but nothing is free, so you will need at least some implementation time and some staff time to work through results. After all, no matter how automated, current solutions do end with a list of changes that need to be made to increase security. Some tools will put a runtime fix in place immediately, but honestly, that is a patch—it only stays in place until you can fix the underlying issue.

    Integrating all of the great security tooling with each other is another aspect that will require some time. Some vendors cover a massive amount of the security testing needed across markets and technologies. These companies show us that even if an organization would rather use best-of-breed, there is a lot of benefit in increasing integrations. For an easy example, consider this: If the software bill of materials (SBOM) generated during the build steps shows that a given framework is included in an application, then active testing via dynamic or interactive application security testing can include extra tests specific to that library.

    Integrations will take effort, but once they’re in place they will enhance application security. There is no perfectly secure application, but the tooling is available to make a given environment less of a target because it is too much work to attack. And that is what you should be shooting for.

    I’ve said before that making security part and parcel of your DevOps practices is mandatory at this point, but that was in reference to the great DevSecOps toolsets currently out there. I’m suggesting now that we include it all–from design to GitOps runtime drift management. Because we can, and the result is reduced risk to the organization.

    And keep rocking it. Honestly, you’re maintaining something worth protecting–that alone means you are successful, even if you aren’t aware of every issue that crops up. Issues are why you’re there, and the systems keep running because you are.

  • It Is Time for DevOps to Focus on Ops

    It Is Time for DevOps to Focus on Ops

    The advent of both Agile development and DevOps were great steps in improving an IT process that had become ossified, with policies that alternated between resistance to change and trying to do things better/faster. Between them, Agile and DevOps broke the cycle of resistance … but only for development.

    The confusion around CI/CD is a great example. CD is both continuous delivery and continuous deployment. Do you know why? Because up to that point (and even recently), DevOps included Ops in the name but did little with or for operations. I have no doubt that it was originally intended to include everything from build testing to full-on deployment, and we are finally getting there.

    You have no doubt noticed that both operations and security are finally getting DevOps’ attention at critical points. I have been chatting with coworkers about value stream mapping and, if you map the value stream of a software delivery process, you will find that all of the weak points have shifted right. Thankfully, we are also paying attention to them these days.

    From GitOps to infrastructure-as-code (IaC); from cloud-first to policy-as-code (PaC), we are finally approaching the operations side with some level of seriousness. And it is generating results for those involved.

    My two areas of attention these days are DevOps and application security (AppSec). As I’ve written elsewhere, both are benefitting massively from things like policy-as-code. The ability to apply corporate deployment policies, security policies and compliance policies from a centralized location is massive. It is early days, but community contributions to PaC libraries are already massive, making it easier to deploy without recreating that big old list of generally useful policies.

    Given a project, a set of tools and a timeline, you could fully automate the process from the first line of code to running in production, with stops for approval and the specialized steps that still require human interaction (no matter how many public policies are out there, someone with knowledge of your network still has to set up IPs or ranges, for example. Or someone still needs to write the code). Someday, it may be possible to say, “We need a phone app that targets both major platforms and is hosted on Linux that does X, Y, Z,” and it just happens … but we are not there yet, and, IMO, the domain knowledge will be a roadblock for a while. But you can take advantage of what we have to work smarter, not harder. And the organization will be better off for your efforts.

    So keep at it, start to evaluate your processes and consider placing any of the great new DevOps technologies into your path. Let them do some of the grunt work so you can focus on adding value. And for all those coworkers that still don’t get it, thank you. You are keeping them in a job, whether they admit it or not.

  • Coming Soon: AutoOps

    Coming Soon: AutoOps

    I’ve spent a lot of time looking at products in the DevOps toolchain and paying attention to where they are going, what they are doing that is new and how they are differentiated. That has led me to one inescapable conclusion: AutoOps is on the way.

    What is AutoOps?

    AutoOps is full robotic automation; the continuing progression of what has been building for the last couple of years. We have DevOps, GitOps, DevSecOps, IaC, CloudOps, AIOps … and the list is still growing. But they all move toward one thing: AutoOps.

    Now that development has been fully integrated into DevOps, the market is moving to do the same for operations. It is harder with operations because DevOps had Agile, but it is coming. And it is coming to your org.

    That is a good thing, in my opinion. Compliance almost demands it; what we currently call AI enables it and staffing in hard macroeconomic times will make it appealing.

    The questions that you should start asking now are what does the organization need to enable rapid deployment, management and upgrades while leaving humans in control for the majority of decisions?

    Some of you are living this. I have a friend that is in a GitOps shop with automated CloudOps where they are implementing AIOps to handle minor day-to-day things and monitoring for drift … but they’re the exception; most of us are nowhere near there.

    Knowing that is where we are heading means you can plan – and it is glaringly obvious this is where we are heading. Vendors are branching from their current subset of XOps to take us there and eventually, we will have a full end-to-end vendor that can do everything from GitOps to fully automated rollout, green/blue deployment, monitoring and drift correction with human intervention only where the organization wants it. Some vendors are already very close to this environment and I expect that several will get there in the next year or two via acquisitions or partnerships.

    We just don’t think of the overall DevOps/concept to retirement SDLC process in the absolute terms that AutoOps requires, but we’re going to want to because the environment is changing. The changes are for the better, but they will be massive and cause disruptions. However, they’ll make everything better in the end by eliminating error-producing bits while leaving us in control of the flow and process.

    Start Planning for AutoOps

    So start planning, understand what your organization wants and needs from the coming full-robotic automation I call AutoOps, and keep kicking rear. Eventually, we’ll have systems that can do a lot of Dev and Ops work, but that day is a ways off, and will require people to manage the systems that manage the systems… Funny but true.

    *I’ve never seen AutoOps used anywhere with regards to IT, I just thought it describes where we’re headed pretty well. Let me know if you’ve seen a better term coined by a smarter person. Or a not-smarter person that has a cooler name for it.

  • Forget Change, Embrace Stability

    Forget Change, Embrace Stability

    DevOps is about change. It was introduced to accelerate change, and it has opened up a whole new world of tools and possibilities that are all based on the premise of changing IT. Starting with Agile, we now have DevOps variants and subprocesses that run the entire software development life cycle (SDLC) process.

    Want to make more frequent updates that can regularly show demonstrable progress to business owners? Agile and CI! Need to get the app settled in its inevitable home with automation? You need CD! Want testing and zero-day protections? Start with DevSecOps built into the CI/CD process and add in application and API security! Want to fully automate deployment in the whay builds are automated? Grab a GitOps tool and let’s goooo! The list goes on and on. Literally. We’ve almost got more tools than we have names for spaces at this point.

    And all of those tools introduce change. Most of them introduce a massive amount of change. You don’t get to GitOps without breaking some of your existing processes and reworking them. We have fallen in love with change in IT.

    But there is a funny thing that’s happened along the way; I’ve mentioned it before, but never from this perspective. All great changes require time for consolidation. Be it stock market swings or revolutions, afterward, there is a period of consolidation. In IT, containers are a good example. When they became capable of actually running workloads, IT experienced a period of time when making all the various machines run in VMs was the thing. People weren’t (generally) looking for the new big change in the process, they were just glad that those one-off machines that had custom hardware could now be moved to something more manageable and that spinning up servers could be done in days or weeks instead of the months it used to take when you had to order hardware. But that consolidation created organizational standards and stability that the organization enjoyed for years – even after containers rose to challenge VMs and some organizations even today.

    So every once in a while, take a breather, consolidate gains, create standards that have meaning for your organization, then continue on with whatever the shiny new change is. Make sure that once you have identified, for example, the need for code scanners that can work within or that are complementary to the integrated development environment (IDE), that you have one scanner—or one scanner per programming language—to maintain and monitor instead of one tool per project. That saves time and creates stability, both of which will serve the organization moving forward.

    Change is here to stay; it’s the only constant. IT is caught in a cycle of change followed by change, with the next wave driven by generative AI. But in times of change, seek stability and, in times of stability, seek changes that can have a positive impact. So go standardize and organize before the next wave hits.

    Mostly, the changes will serve the organization by freeing up your time and letting you spend more of your time rocking the IT world. Those systems stay up because of you. More time is more innovation, and more stability means less burnout. All of which is good for the organization.

  • Is AI the Answer to Devs’ Documentation Problem?

    Is AI the Answer to Devs’ Documentation Problem?

    I have written this blog half a dozen times over the years, but every once in a while something comes up that makes me realize I should revisit it once more for those in the back. This time, it was a social call with a group of friends, some of whom are in high-tech, all of whom are smart and often insightful. One member of the group, a tech lead for a company that you do business with, was talking (in vague terms, of course; this person is a professional) about work he is doing, and the group started to make fun of documentation.

    Let’s face it—since the very first line of code was written, we have sucked at documentation. It is a rare project that (a) gets just enough documentation and (b) keeps that documentation up-to-date. What we most often get is no documentation or documentation that records the initial design but, by the end, bears little resemblance to the code’s actual workings. The worst of all types of documentation is the “It’s self-documenting!” kind, which has gotten another boost based on the idea that OpenAPI/Swagger will be a tell-all about what is actually happening with the API. Like far too much API testing, it does a good job of telling you about inputs and outputs and, in most cases, absolutely nothing about what is done with the inputs or what magic happens to get you the outputs.

    After our call, I got to thinking. What can we do to adequately document the code? I mean, with AI becoming all the rage and the set of lines of source code available for training being so massive, you would think we could do better. Maybe we can—but right now, the language processors that we currently pass off as AI are too busy being conversational or generating the code itself. Both of those spaces are going to take a while to get “good enough” to rely upon without review (not as long as many think, but certainly not tomorrow). Maybe that point will be when some enterprising engineer will turn it on source. Though I think that it will take a documentation specialist and not a coder—after all, we coders created this problem and haven’t fixed it in decades.

    Many of us truly believe the “self-documenting” part when that is and absolute lie. It may be self-documenting to someone who wrote it or, in the best cases, someone with deep domain knowledge, but certainly not to the person who has to pick it up. We all have horror stories of working on that application—or porting/rewriting an application—that claimed to be documented or self-documented and was not even close. The saving grace in all of the cases that I worked on was that the code actually worked and performed the job required. Given some extra time to step through it, it could be rewritten even without good comments and docs. (Hat tip to the dev that had to pick up one massive porting project from me because I left to go write about storage—the project was huge and was a blast, but having to hop in halfway through probably made him crazy.)

    If you are planning for the future, I recommend a ton of things, but for documentation, only two. First, write design docs, discuss decisions and put them in Git with the source code. Require that they be reviewed and updated after each sprint. Then, make sure that there are comments in the code. Using naming conventions to document code is helpful, to be sure. But more is needed—particularly as code morphs into whatever it is going to be in long-term final form. We have all seen that method or function that has a name … and then doesn’t do what the name implies. Or worse, that does what the name implies but also applies half a dozen side effects because devs needed somewhere to put new functionality. And we haven’t even talked about complexity like feature flags.

    So do both, and require that devs spend a bit of time on each during each sprint—not to be a jerk, but to pave the way for dev #327 that inherits this code and has to make sense of it. Help them get productive quickly.

    And keep rocking it. We have documentation problems because we solved business problems rapidly. That’s a good thing, we just have to think about the next gen of devs that will be on this project, and give them that little boost into productivity that documentation offers. And maybe you’ll help the compliance team while you’re at it, but that’s a different blog.

  • Standardize: It’s Not the Where. Sometimes it’s Not the What

    Standardize: It’s Not the Where. Sometimes it’s Not the What

    In our industry’s attempts to follow best practices that marketers assure us we must have or we are going to lose to our biggest competitor … No, wait, we’ll lose to that startup no one ever heard of … No, hold on, we’ll lose to foreign competition … No, wait … (okay, I’ll stop. You get the picture), many of us have gotten completely unreasonable in our IT approaches.

    Like asking a vendor, “Where can I run your application?” Well, the more “where” options the better, because every organization has different needs. But a vendor has to start somewhere, and you can tell when they get to a certain size because they don’t want to talk about it. “We’re SaaS,” was the en vogue thing to say six months ago; now we’re back to “containers” or even “cloud,” thanks to a whole slew of cloud-first offerings out there. But what all of these vendors really need to be able to say (and a disappointingly small number of them actually can) is, “We run on containers, so we don’t have to care where you deploy us.”

    Those are the types of scenarios that forward-looking enterprises need to start anticipating—the broadest, easiest-to-implement, longest-lasting solutions that can be found. The days of “devs picked this DB no one ever heard of because it was 0.0000001 microseconds faster on a hundred-line query,” need to be gone. Standardize on highly usable tools across the DevOps toolchain from libraries approved for use in apps to deployment technology.

    A big part of this approach right now is my pretty common refrain: Advanced automation. The more a system can do, the more time staff has to do other important things. All forms of testing are finally within our grasp because of advanced automation technology ranging from Kubernetes programmability and controllers to ‘AI’ filtering of results (Side note: What most IT folks currently call AI is not; we need to stop calling it that and give it a more accurate name. Machine learning is better; advanced automation is more to the point, IMO). But whatever you call it, it enables massive pre-release testing. From functional to security, it offers the ability to make these tests useful by speeding up testing and results processing so staff need only look at issues considered critical. No one cares where the test engine runs, just that it can run tests wherever the application resides.

    The same is true for a whole lot of other ‘whats.’ Vulnerabilities are probably the best example. Tools are starting to look for them in an automated fashion. Very few organizations have a comprehensive list of which vulnerability databases a tool must support. Any given organization might have one or two databases in their list, but most just require that the functionality be available.

    So don’t get bogged down in the details, and don’t select five different tools to do the same job on five different teams. Start standardizing again. Oh, standardization won’t last when something evolutionary comes along in the space, but it will leave you less clutter to clean up and more specific expertise in-house.

    And that leaves you more time to keep wowing the business. Keep those systems running smoothly and keep cranking out more useful stuff for the organization. Most orgs show their appreciation these days, but some still do not. In those cases, we thank you for them. We know what you’re going through—we are going through it, too, and yet the lights are still on. Keep rocking it.

  • The Right Tool for the Job: AI Edition

    The Right Tool for the Job: AI Edition

    (Actually, the right job for the tool.)

    We are currently in a really weird space with regard to AI capability. For my AI friends, I will make the disclaimer that what we call AI today is not actually AI, it is more akin to LiSP “AI” than actual intelligence (and, in reality, is firmly somewhere in between). In short, we have great language processing tools that we call AI but that have no cognitive ability—but, man, are they useful.

    And the suddenness with which the rest of the world realized they were useful was nothing short of astounding. One day we are using AI algorithms to process test results and filter security data; next thing you know, our business people are bringing up AI and how advanced it is in every meeting.

    They’re not wrong, but we need to differentiate the technologies. The promise and danger of chatbots is not the same as the promise and dangers of AI in security (where it can be used by attackers as easily as defenders, btw). We’re going to have to work out the concerns and benefits in real-time, but we need to start by pushing back and making clear that it is not a panacea, any more than the last 50 technologies that came along. It has great promise to change the world, but so did VMs and Java and containers and …

    This one seems bigger, but that’s largely because it came on so fast and with seemingly broad applicability. But in the end, it is good at processing large datasets and noticing both patterns and anomalies in the data. This is a great, hugely powerful tool, but for things with sparse data? Totally useless. This is critical to remember moving forward. AI that monitors a certain aspect of security in the organization can detect patterns and anomalies, but unless the dataset is massive, there will be a lot of false positives along the way, and a more traditional tool is preferred. If that same AI (same software, algorithm, etc.) is being trained on data from hundreds of companies? Sign the check, it’s probably going to be better and more forward-looking than your current solution.

    And that’s the key. It took the entirety of the World Wide Web—including, I might add, The Gutenberg Project—to train the language processors, so unless that security AI has a massive dataset, it’s not likely ready for prime time.

    In IT, look to AI today (and this will change relatively rapidly, so check back often) for large dataset processing and for advanced automation. Those are the two real use cases. More will come, but right now, the ability to find patterns in large datasets or to say, in real-time, “A+B+C = block that port!” are the biggest viable use cases.

    Over the years, I have grown ever more skeptical of claims about removing the need for developers. Those claims have been made so many times that my friends and I have a stable of jokes about them. Java, XML, 4GLs, RoR, UI generation tools—the list goes on and on. And each one created even more code. I expect AI will be the same, but there is no denying it can offer initial frameworks, much like mobile UI generation tools do. We’ll have to see how much change that bit introduces.

    And keep rocking it. While I hope my readers don’t need reminding not to upload their source code to any of these AI tools, I will point out that your (Dev and Ops) source is so proprietary and critical that employers freak out when it is uploaded to an AI chatbot. No one cares what is done with chatbot-generated source, but the DevOps team’s source is considered critical. Sleep well knowing that.

  • DevOps Onramp: Security on the Rise

    DevOps Onramp: Security on the Rise

    I don’t normally advertise stuff in my blog, because that’s not what it is here for. It is here to offer you the thoughts and ponderings of someone like you, and advice you didn’t ask for. But today we’ll make an exception since I also blathered for Techstrong Group recently, and soon you’ll be able to hear it.

    If you are not familiar with it, DevOps Onramp is this week, and you can go and register to hear my session and the sessions of other great voices that are involved in DevOps. My session is about DevOps and Security, but there are a wide variety of sessions for you to choose from.

    I am trying to make the time to be involved in the conversation when my session airs, and it looks like my schedule will allow it, so if you are interested in DevOps security, I’ll likely be around to discuss it with you.

    I can’t speak for all of the sessions, but mine starts with the basics for newcomers and then covers more advanced topics as it goes on. The truth is that I was supposed to do two sessions–intro and advanced, but I started talking too fast, and the next thing you know, I crammed it all into one session, saving you time for some of the other great content that is available at the show.

    So (virtually) drop by, watch my session, see what else you’re interested in, and feel free to drop any of us presenters a line telling us what you think, good and bad. I’m sure the show organizers would be happy to receive feedback too, but I’m all about presenters.

    And keep rocking it. We’ll be back next week with our regular content (and no ads).

  • Things We Should Acknowledge, Part Three: Remotely Good

    Things We Should Acknowledge, Part Three: Remotely Good

    This installment of my off-and-on “Things We Should Acknowledge” series is going to focus on an area that most DevOps teams are skirting, but increasingly will not be able to: Remote work.

    For background, I have worked remotely for decades; my experience spans Fortune 100s, publishing companies and startups—and that’s just full-time work. I’ve worked for a ton of vendors and publications as a contractor, too.

    I will start by making clear my stance on remote work—a stance I’ve happily talked with people about since long before COVID-19 measures made remote work a regular thing. You either thrive working remotely or you wither. There is a small group in the center that can make it work with some effort, but inevitably that effort becomes routine and those in the middle become part of one of the two groups.

    Because people drift toward one of two camps, we get the currently popular stance on, well, everything, which is polarization. But we shouldn’t. Just like positions and people that should/should not be in meetings to resolve problems or people who should/should not have direct reports, we should consider positions and people that remote work is best for.

    And post-pandemic, the vast majority of DevOps roles can be performed effectively in a remote position. That leaves us with the people question. Mostly. Note: There are organizations and roles that need to be local; those are not a topic of this blog. The 99% of other roles/orgs are.

    This topic gets really ugly, pretty quick. If a significant portion of every organization goes remote, city centers in several metropolises will die. That is not conjecture; you can go look into commercial real estate in any information worker hub. It is in a massive slump (and calling it a ‘slump’ is being nice), with occupancy circling the drain and property values plummeting. That is largely because formerly high-demand real-estate is not needed in the volume it was.

    That should not be the concern of any given corporation. It is the corporation’s responsibility to see to shareholder value, not the real estate market (unless they are invested in real estate, which many large organizations are). But if orgs are keeping any offices downtown, it becomes an issue about safety. In other words, it’s about if the collapse of the market means the offices become less safe for workers, regardless of investment. This is not a decision-making bullet point, but it does inform actions. I have a ton more to say on this topic, but it’s a side bit, so for now, this is enough.

    For people who thrive working remotely—and a lot more people now know that about themselves than knew that five years ago—being forced into the office is a non-starter. I have been offered astounding positions at organizations on both coasts over the years, but the ones that came with the “…and you have to move here” demand—I immediately turned down. More people are doing that; some of them truly great employees.

    So, like it or not, DevOps teams are going to have to consider whether all being in the same building is more important than hiring the most qualified co-workers. That is a team/management/corporate decision and always has been. I consider any “We all do this one thing” stance—from “We’re an X shop” to “We don’t do remote” to be a negative, but I don’t have to make these decisions for an organization.

    It does need to be part of the conversation, though. Because the work environment is unlikely to return to pre-pandemic standards, no matter how much some want it to. The more organizations are willing to work with both remote and on-site employees, the more options those employees—your potential coworkers—have. That changes the dynamic, even in light of recent layoffs, because the new job hunt is not tied to “local” anymore.

  • Start Automating Now

    Start Automating Now

    A bit of feedback I received from last week’s post is that I didn’t provide any background, I just jumped right in with examples. So let’s take a step back and talk about the bigger picture.

    It is always a surprise to me when I talk to enterprise friends and associates and hear that automation has not taken over their DevOps processes. Yes, we still (and might always) want the ability to have a person review what is being done and make course corrections if the automation has something wrong, but most of the automation that is available for DevOps is no longer new and should be a no-brainer.

    The ability for a build to block based upon any given criteria and notify Jira or ServiceNow—or whatever issue tracking system you use—while alerting the people responsible for the build is simply obvious … and yet, some organizations aren’t even that far yet. The ability to stop a build, create a ticket, assign it to the appropriate person and give it a priority is built into most of these products. It should be looked at to see if it’s suitable to the organization.

    This goes on and on. We’re now to the point where everything in the toolchain can be automated. IaC and GitOps are tackling the broad operations end, and if the application is 100% public cloud-based, CloudOps simply solves the problem. Automation from check-in to production is possible, with very few places where intervention from humans is needed.

    Given that most of this technology is mature—even though some of it has not been around as long but has broad enough adoption that it can be considered mature (Flux and Argo come to mind here)—it is time for enterprises to start another round of automation. Just like AI won’t eliminate most development jobs this year, automation won’t eliminate jobs, either. But we’ve been so strapped for time for so long that this may be a surprise. Automation might reduce the need to scramble for new team members while asking current employees to give a bit more … over and over.

    So get started. The math that made DevOps appealing in the first place—streamlining to reduce time to market and free up IT hours—is the same math that applies to the current round of automation. The more that can be automated, the more time team members will have to perform more productive work.

    Having mentioned AI in passing, I’ll use that as a segue to include AI in the “automation” discussion. At this time, nearly all of the tools we call AI are simply advanced automation tools and should be considered in the process of increasing IT automation. The ability for an app to run, learn your systems and improve them (be that improvement in development, security or ops) is just advanced automation. In the places where it works well in a given environment, it should be adopted. It is highly unlikely that AI will get worse over time, so even marginally acceptable AI results make it worth implementing, counting on technology advancements and continued training to improve that benefit moving forward.

    So, start automating now. Free your time, take a holiday at the beach, do the same job in less time or, to quote a previous generation, “Work smarter, not harder.” You’re doing the job that keeps the servers humming and users coming, now do it with less effort. And keep rocking it.