devopsy.kinsta.cloud

Author: Don Macvittie

  • Security Debt: Speed vs. Common Sense

    Security Debt: Speed vs. Common Sense

    A couple years ago, we had some spectacular security events that involved DevOps and Kubernetes, where the managing team simply redeployed containers whenever one crashed. It turned out that many organizations were doing the same thing, and, what’s worse, they were not talking about it because they knew it was not a long-term solution to crashing issues.

    At the time, I thought these high-profile lessons would teach people to actually fix code issues, even in high-throughput Agile-meets-DevOps shops. I admit I also laughed, yet again, at The Phoenix Project’s expense, thinking, “Yeah, but the CISO isn’t holding things up by demanding that get fixed!” I wish that would ever get old, but it won’t.

    It seems I was wrong. Or at least partially so. Some—perhaps many—shops seem to have missed the message that these issues sent. Anecdotally, we heard this was still happening, so we started asking questions. Yep. There are shops out there (since we started with ‘anecdotally,’ I won’t pretend to put numbers around it) that still take an “If it isn’t causing disruption in core systems, don’t fix it” mentality.

    Ouch. We’ll be revisiting some of those in the future, I’m sure—when it’s listed as a vulnerability or breach somewhere.

    I’m not a security professional. Because the vast majority of enterprises do not have dedicated security and even more don’t have anything close to enough security staff, I have done security as a part of almost every job I’ve taken—because it was what needed doing the most. Many, if not most of you, are in the same boat.

    That said, I can tell when you are blatantly asking for a security incident. And running code known to be faulty or misconfigured to the point that it repeatedly crashes is deliberately asking for a security incident. “But it isn’t exposed to the Internet!” is only a valid rebuttal until an attacker gets inside the perimeter and uses your fault for something like privilege escalation. Then no one cares if it’s connected to the internet or not; they want to know how the attacker did it. And that’s on you.

    So for those organizations in which speed has overridden common sense, let’s be very clear and to the point.

    It is a best practice not to run crash-level faulty software.

    Since you must not have one, write a policy that explains this and lays out a process and/or timeline to fix these problems. Then drive it home. Make certain it is known that this policy will be enforced. For that huge majority of my readers that don’t have the level of authority that “create and promulgate a policy” implies, convince those who do. Point out the issues to them in writing. “Notified in writing of faulty practices that could bite the company in the rear,” tends to get even the most unfriendly management to click into CYA mode and do something about it.

    When we all watched those spectacular failures back in the day, I assumed that this was all about growing pains; you can do so much more with Agile/DevOps/cloud and/or Kubernetes that this type of thing would crop up and, eventually, get addressed. I still believe that. You all are still testing the boundaries and pushing the limits and making all of IT look good. It’s just going to hurt getting there sometimes. Fix the apps, make the policy, keep kicking butt and know you are killing it (as an industry. If you aren’t, stop laughing at my blanket compliment and fix yourself).

  • What We Don’t Know is Dangerous

    What We Don’t Know is Dangerous

    Pop quiz: How many database management systems (be they RDBMS or NoSQL or even flat file) do you have running in your overall organization? Not actual databases, but management systems?

    How many containers do you have that are exposed to the internet, directly or indirectly? Not container management systems, because you’re only running a couple of those and they’re all relatively new, but actual containers?

    The questions could go on and on. Not too long ago, because there was an identifiable cost associated with it, cloud inventory management became a popular thing – finding and removing cloud instances, services and storage that were no longer necessary became a part of normal operations for organizations that had even a mid-sized cloud presence.

    But now, we have sprawl all over the place. So let’s extend inventory management to all software, and get rid of what isn’t needed on a regular basis … because it does indeed build up quickly.

    Yes, there are a few companies out there trying to do just this, and I encourage you to check them out. I have not, because our inventory at Ingrained Tech is manageable without that level of overhead. Our development and ops time is limited, so we keep a tight rein on what is out there and the maintenance it requires.

    But for larger organizations, the more you have out there, the larger attack surface and potential data sieves you have in place. So make sure if it isn’t needed, it’s gone. In the age of containers, this actually requires a fair amount of investigation; first, a list of all containers part of an ongoing project (and in even a mid-sized company, “They’re all together for each app,” won’t create this list because random containers are out there, too). And then a list of all containers not associated with a known project that can be gone through and usage determined.

    It’s not glamorous. It’s a bit of a slog. It’s in the organization’s best interests, though, and the slog should motivate your teams to put a policy in place to manage container life cycles. Like notes in Docker files that say when the container is no longer needed, or what it is needed for, so future you can easily get rid of it without stress.

    If you are in a containerized environment, this problem is growing almost by the day. Seriously. On our last project, we had 12 active containers before we pared it down to the five we needed long-term. The others were R&D along the way or tools we were using before focusing on a different method of achieving the same goals. Since we’re a small team, we just cleaned up when the project was coming into focus, but a larger org needs to be more focused on cleaning things up as they go.

    Meanwhile, those assets are letting customers get to your org, and allowing employees to manage the systems. Keep rocking it, and just remember to pick up, particularly after cutting-edge new projects, but also occasionally as just a normal cleanup step.

  • Turnover, Documentation and Missing Links

    Turnover, Documentation and Missing Links

    When employee turnover spikes, IT in general and DevOps teams in particular need to be prepared.

    At the time of this writing, the stock market is roiling, and DevOps compensation is bouncing up. This has an impact on turnover because many in DevOps (and nearly all senior positions) have compensation that includes stock—stock that may be down significantly or that is currently at high risk of going down. This causes two types of movement—those leaving an organization to avoid further loss in their compensation and those who are leaving their current gig to join an organization whose stock is down. Combined with those who leave for a role with a larger base salary, there are those who are leaving to earn more and those who are leaving to avoid more loss. IT budgets ballooned during the pandemic but those, too, are deflating, so while we have a worker, not a position shortage, there will be pressure from both sides.

    So, why is this geek going on about employment, compensation and available positions?

    Simple: Because DevOps is not any better at documentation than any other flavor of IT. In tightly-run shops, documentation is available and maintained. In most shops, it is spotty at best. In some, it is downright nonexistent. This has forever been true in IT and, if anything, it is more true because the early stages of DevOps implementation result in a massive amount of change.

    People leaving in an environment with less-than-stellar documentation can be a problem. What tools are used? Where are they located? What’s the login for these SaaS tools? Etc., etc., etc. If turnover means you’re losing entire DevOps teams in relatively short order, this can become a nightmare relatively quickly. And that nightmare is very real for everyone left behind and those that are coming in. Someone has to go figure it out; IT management has to let the rest of the business know that retention issues are causing backlogs. Everyone not tasked with figuring it out has to pick up slack for those who are … you get the picture.

    Take the hour or four and document what you have. It doesn’t have to be comprehensive, it just has to list what tools, where they are, who at the organization is responsible for credentials at any SaaS vendors and under what circumstances the tools are used. If each team documents their own stuff, even in a shared toolchain environment, comparing the documents will uncover little details one missed that another did not.

    Let’s face it, we have—for better or worse—moved largely to a world where loyalty is based on compensation. Much like mercenaries, most IT employees are always listening for a better offer. So the organization, while acknowledging this fact, needs to take steps to minimize the negative impacts of turnover. New ideas and suggestions to change things are all good and you don’t want to interfere with that aspect of turnover. But anything that impedes the organization’s ability to deliver should be addressed beforehand. And that means documentation so remaining and new employees can get up to speed quickly and take ownership rapidly and with confidence.

    And keep rocking it. Whether you are looking for a new gig or not, keep the apps you are responsible for running—and document them. You’ve contributed to the success of your organization; don’t slack off while moving along. And don’t hate those that are moving along. Not all of us base our loyalty on compensation, but we do all allow compensation to influence our decisions (unless you are working for free and I missed it?) Keep rocking, no matter your status. And don’t hesitate to tell interviewers that you helped your last organization achieve success.

  • Sometimes, You Should Fix What Isn’t Broken

    Sometimes, You Should Fix What Isn’t Broken

    DevOps has been with us long enough now that there is a lot of legacy DevOps stuff running in all of the places I have eyes into. Ignoring items that are not actually working; those pieces that pose a serious security threat because they house vulnerabilities that have been fixed—there is a lot of “It just works,” piping in place and even a bit of “We’ve always done it that way.”

    Considering DevOps is still often pushed as “the inevitable new hotness” by marketers, I find this interesting; the fact that DevOps is now the de facto standard across enterprises is a topic for another day. I want to talk about the stuff running quietly, relieving you of the need or desire to think about it while focusing on issues that do need you to think about them.

    But that type of thing is exactly what created the need for DevOps in the first place. DevOps came in like a wrecking ball and forced you to look at the unnecessary steps that had built up around the items in the organization/culture/infrastructure, often around these very “This is just how we do it,” or “That ain’t broke, let’s look at this other thing, instead,” attitudes.

    At Ingrained Tech, we have scripts that run our backups to the cloud. They are part of a fully automated process; we designed them to make sure our system of what gets backed up and when highly adaptable. The system runs like a champ every night unless we tell it not to.

    Our cloud storage provider has come a long way since we first used their API to get the data up there and wrote those scripts. In fact, customers like us are the reason the provider keeps backward compatibility with those APIs. Which means it is time for us to revisit, use the newest APIs and see which of the many, many new features we want to take advantage of. “Good enough” is just that. The system isn’t broken—we get viable backups out of it (which we needed not too many months ago, so we know we get good backups), but we’re not getting maximum value out of it; we had to do a bit of hoop-jumping to get the data set back up the way we wanted.

    Our simple backup routines are nothing compared to similar “good enough” arrangements I’ve seen in larger companies. “Good enough” arrangements that are creating larger and larger speed bumps that sometimes resemble the exact environment that made the huge gains of early DevOps implementations inevitable.

    Take a look at your infrastructure and toolchains with a critical eye. Look for low-hanging fruit, and knock out some of the baggage that has built up. Schedule some process renewal days, where a team member or two specifically looks at everything in the environment with a critical eye, and see what they propose.

    And keep rocking it. Speed bumps or not, you’re making the apps and infrastructure that keeps the entire company running. Remember that, and keep a smile on your face while you’re solving recurring problems.

     

  • DevOps and Containers

    DevOps and Containers

    “Works on my machine!”

    We’ve all had that co-worker. I have had one at pretty much every company I’ve worked for. At several jobs, it devolved into a running joke.

    There is a belief that containers alleviate this issue. And they may—or they may simply mask the issue. The choice is up to IT.

    Inevitably, something that works on one machine and not another can be traced back to on-the-fly configuration changes. If “we use containers for DevOps” means that developers have full control of the container environment, then it must also mean that any configuration changes be codified into a documented container image that can be spun up repeatedly and even recreated in a Dockerfile if needed.

    If, instead, it means that devs have a playground in containers and, once they get it going, they can chuck it into production—even if you don’t say it that bluntly—then you are making things worse with containers, not better. At least in the days of “works on my machine,” the issues had to be resolved because the only time you heard that phrase was if something didn’t work on the machines it needed to be deployed to!

    As a developer, it is tempting in both container and virtual environments to upgrade a package or tweak a setting without documenting it. But a clean DevOps environment needs to be able to build the product from Dockerfiles, not from a pre-existing image. Yes, you can just clone the existing image, but then there is no documentation of what has actually been changed and it takes a lot more work for those following you and debugging future problems.

    So take the time; keep an up-to-date Dockerfile (or whatever equivalent script is used to build your containers). It is not that significant an investment to stop for a minute, open a file and type a few lines. And it pays off in spades if, in the future, someone is trying to figure out what the heck is going on in there. Consider it self-documenting configuration. I’m not a fan of “self-documenting code,” but for config, it is pretty clear if you go, “Oh, we needed version 10.2, not version 10.1; let me update the init script.”

    There are a fair number of shops out there that modify things running in production, and the world hasn’t ended yet—though that may well be how it happens. So, a good practice that will help those trying to follow behind you is clearly not a necessity. But each of us has gone back after a year and said, “Why did we do that this way?” which was (or wasn’t) then answered with, “Oh, here is exactly how the environment was configured.” But it might help answer the ‘why.’

    Your apps are powering the world—or your specific apps are powering your little piece of it. And you’re kicking it just keeping them running. Now, go make them better! Consolidate changes, manage configuration through Dockerfiles, Ansible, init-scripts, whatever combination you use … And keep rocking.

  • Where Has All the Testing Gone?

    Where Has All the Testing Gone?

    Testing has long been a problem child of IT in general, AppDev in particular, and now it is DevOps’ problem. There are things that DevOps can do to improve the chances of tests actually occurring for your application(s). Interestingly, there are some that think we no longer need testing; the argument is that because new releases can be cranked out so quickly if something needs to be fixed that it is irrelevant. Combining feature flags with the ability to roll out a new version and, well, there may be some truth to that—at least for a limited subset of testing.

    If an organization implemented feature flags for user acceptance and new feature testing and then standardized on test-driven development for DevOps development teams, that would leave systems and security testing. Systems testing could, theoretically, be replaced by the ability to roll out a new version. Although systemic issues can be complex and difficult to fix, this is a riskier proposition than extending test-driven development to the systems level. For the sprint, assign a person responsible for the overall integrated system and have the rest of the team support them through the process of making certain all works as expected.

    Testing—particularly systems testing—slows releases. I get that. The rate at which most of us can iterate once DevOps systems and processes are in place is fast enough that you can afford a bit of slowdown. This leaves security testing as the last standalone bit. But some security validation can happen during the development process if DevSecOps tools are being utilized. That still leaves some security issues to test for but does remove a large chunk.

    The key to these last bits is to get them integrated into the process/toolchain. DevOps runs straight through, over and over. Any bit worked into the “runs straight through” will be executed. If today’s iteration falls for the “We ran over, we’ll make the deadline with less testing” mentality; not an issue—it’s one iteration. Just don’t let that become the default answer to someone not completing an iteration’s work on time. Development can have unexpected speed bumps that delay delivery and acknowledging that and documenting what it was for future reference is important. Covering it up by reducing testing time risks a less stable product and does not help overcome the issue in the future.

    So start thinking in terms of DevOpsTest. Make sure the product you are delivering represents the massive time investments you’ve made producing and maintaining it. You’re rocking it—and something like integrated testing ensures that users see how great you are rocking it and not an error page.

  • Watch Your Language(s)

    Watch Your Language(s)

    Just about every organization I am familiar with today is using more languages, frameworks, libraries, etc. than they were a decade ago. Back then, most organizations were doing a lot of Java, and maybe a little bit of PHP or .NET (almost always VB.NET). Now we have Java, Python, Ruby, Go, Swift, etc. And etc., etc.

    The problem is that employees don’t come pre-instructed with your selection of languages. Most developers can bring Java or Python and most know at least one other language well. All of them can learn (more on that in a moment), but when hiring, an organization needs to focus on the one or two languages the hiring team is using, not all the languages an organization might ask a developer to learn over the course of their employment. After all, when hiring 20 years ago, the tools of the mainframe team weren’t generally listed as requirements for a desktop developer. While there is a difference (desktop developers were only asked to work on mainframes when the shortage of experienced big iron devs got really bad), it is a similar situation today.

    You can hire with an eye to training, and most developers will be more than willing to pick up a new language and learn its ins and outs. But ask any manager that pays close attention, and they’ll let you know that there is a huge difference between time-to-productive and time-to-proficient. One manager told me that any programmer with experience could start contributing relatively quickly in Ruby, but to do the important stuff—things like knowing which array access methods were faster and the most efficient communication between Ruby and other languages—took far longer. His numbers seemed too high, in my opinion and based on my experience, so I asked around. All the managers I know that worked with Ruby (because to a great extent, this is language-dependent, so I didn’t want to make hasty generalizations though, in hindsight, the same data point in Python would have given me more responses) agreed there is a curve, and the range was somewhere between six months to two years. This is strictly anecdotal, but it seems the more technical the team in question, the longer the manager thought proficiency would take.

    But even if the lower number is accurate and applies more generally, it is an argument for keeping your language count down. Yes, some languages are better at X or Y or Z than others, but most modern languages can do just about everything an average enterprise team will ask of them. So, keeping languages within a smaller pre-defined set still allows teams to choose the best tool for the job while keeping the number of skills developers at the organization might need down to a manageable level.

    And keep wowing them. No matter what languages you are using, your user base is logging in and using the results daily—or perhaps your devices are reporting the results daily or processing them. No matter how you are delivering, you are, as our digital world shows. Keep rocking it, and do what you can to keep the number of languages under your roof down.

  • No Matter How Things Change, Have a Plan

    No Matter How Things Change, Have a Plan

    Had enough change yet? It is almost amusing that I and some other tech people had that conversation a decade ago, and since then, nothing has slowed down. Our conclusion was that, for the near future, change would be constant. That conclusion was right; I’d have to ping everyone else to see what they think, but “near future” turned out to be longer than I expected.

    Inevitably, though, change requires taking a breather at times. It appeared that high-tech was finally ready to admit that burning out your staff was not a viable long-term business model, and was slowing down … and then, COVID-19 hit. Now I’m hearing rumblings about going back to non-stop change as if the last couple of years were a vacation and staff is all rested up and ready to replace massive chunks of the infrastructure.

    The reason that change requires intermissions, though, is not 100% about staff and burnout. That only shows up when you’re making a ton of change non-stop. Before staff burnout becomes an issue, recovery and business continuity plans become outdated. At the rate we’ve been running, I’d go so far as to say they are likely useless.

    Yes, all this change has brought us some cool stuff. While we are busier, some of the busywork is far easier, or even non-existent now—but that doesn’t change the fact that you need to breathe. This is a perfect time to go over plans to keep the business on its feet in case of IT catastrophe—or at least some major systems catastrophe. Saying “We’ll just deploy a new version” is cool, but it only works in some scenarios. I touched on this and discussed one case where it doesn’t work in October 2021. But there are a ton more. Every environment is different, but hardware failures have not disappeared and destructive writes still occur, as just two more examples. Your infosec team can give you a few more examples.

    Knowing what you’re going to do when the worst-case scenario happens is huge. And for those who’ve never been through it, knowing before people are stressed and asking you for a timeline will save you a lot of headaches.

    For those who’ve never had a DR or business continuity plan, consider it. At least have a basic DR plan that lists critical systems, things most likely to go wrong and what the steps are to recover it and get things moving again. The process of making such a list will often uncover things that should be done but aren’t—like backups or backup verifications.

    For those with a DR or business continuity plan, I’ll take a wild shot and say, “It’s out of date.” Organizations tend to create them and then forget about them. These plans should involve a bit of each project, so it’s taken care of in bite-sized chunks, but—probably because the first implementation of DR or BC is normally a massive project—it tends to be set aside and revisited every few years. Don’t do that.

    As I’ve said before, you and yours have built the heart of the modern enterprise. Don’t forget to protect it. It doesn’t take much to use some of these sweet new technology advances to put in DR procedures. Tack DR review/updates onto each major initiative, be it a new product or a big change to existing software. And keep rocking it. If you have a say in your organization, consider slowing the pace down for a bit and even updating DR. If you don’t have a say, may your employer be wise enough to do so.

  • It is Time to Secure Git

    It is Time to Secure Git

    At this point, we’ve got a ton of experience with the bits of Git that we use. And locking down Git is well-documented. Okay, it is documented; we can say that, at least. Interestingly, much of the ‘how to secure Git’ information out there is actually ‘how to keep critical information out of Git,’ and that is the problem. ‘Critical information’ always includes warnings to “not put configuration information into Git. Ever.” And that has been the default way to lock down Git for nearly its entire existence.

    Then along comes GitOps. What is its purpose, in a nutshell? To put that critical information into Git. So we have been waltzing along; only a fraction of us actually locking down our implementation because we were making sure the data in Git, while valuable, was not full of bits of information that would be useful in compromising systems. But now it will be. In fact, in GitOps, nearly everything is one of those bits of information useful to attackers.

    All of which means that it is time to get serious about Git security. Not content, but actual Git security. I won’t go crazy with links to lockdown documents here—it is a vastly different process to lock down a local Git repository versus a remote one and it is drastically different when you’re locking down a service like GitHub or GitLab (actually, some of the work is done for you with a SaaS vendor) … so, you’ll have to look it up yourself, based on your Git implementation. But there are documents out there that walk you through it; you’ll just have to make sure you find the ones that say, “How to lock down Git repositories” and not “How to keep privileged data out of Git repositories.”

    But whatever you do, stop putting it off. Now, if a neer-do-well gets into your repository, it won’t just be source code for various apps that are leaked—and that is certainly bad enough, even if your organization is not a software organization—it will be infrastructure information that can be used and abused. So make certain that information is protected in every possible way. Point security at Git and tell them, “We know you are insanely busy, but this is the most important thing to protect if we are going to implement GitOps.” Because it quite literally is. If database configuration, network configs and possibly even login info are to be stored in Git, locking it down (both local and remote copies) is imperative. You wouldn’t leave that info in plain text on a centralized data store and count on existing protections. So don’t do the equivalent with Git.

    If GitOps isn’t in your immediate future, that’s still your source code full of vulnerability advertisements out there, so at least ask your security team to give your Git install/policies/config a review. Even as a developer, I am less worked up about source code repositories than configuration repositories. There are several reasons for that, but the big one is you all. The bulk of organizations just don’t seem that worked up about source code repository security, but that changes when operations is asked about the list of holes in the firewall being moved into Git—and being configurable from Git (as just one example).

    So, keep kicking it! You all are making, delivering and securing a metric ton of software, and the complexity of our environments did not decrease after the remote work wave hit. Yet, you are still rocking it. May you have enough staff to do the job and applicants that come looking to work with you over competitors. And keep showing the world that software is manageable. But secure your Git.

  • AI: It’s What Staff Crave!

    AI: It’s What Staff Crave!

    If you’ve seen Idiocracy (Side note: I am not recommending the movie; I’m using a phrase from it we’ve all heard. The movie itself was a comedy not at all meant to be family-friendly when it was made, and changes in social mores since its 2006 release make it far less acceptable today), you are familiar with the title phrase I’ve used here. That’s what is repeated almost religiously to justify why they buy/use/drink Brawndo, an energy drink.

    One of the scenes involves people telling the star that plants need Brawndo—“It’s got electrolytes … That’s what plants crave!” This is repeated over and over until he asks, “What are electrolytes?” After some stuttering and failures to answer, someone says, “It’s what they make Brawndo out of.” Someone else inevitably tacks on, “It’s what plants crave.”

    Now, substitute “AI” for “Electrolytes”. That’s where we seem to be. Ask a vendor what algorithms they’re using, where their training data comes from, the difference in results between the test reservations and the real world. Most will end up saying a lot, not answering your questions and ending with, “It’s what staff crave!” or some such.

    A few companies will lay it all out there, assuming you can parse through the information and are familiar with the AI training algorithms they are using or the algorithms they’re using to classify inputs against the dataset. And these companies are grand because they (a) know what they’re actually doing beyond “It’s got AI/ML, it’s what staff crave,” and (b) Expect that you are knowledgeable about the data you’ve asked for or are capable of looking it up in this age of wonder that has boundless information at your fingertips.

    Don’t let your vendors get away with just repeating “It’s got electrolytes” over and over. AI and ML are a massive help in some areas, but they’re not yet a panacea and success rates vary wildly whether we’re talking problem domains, datasets, algorithms, training style, etc. Make them tell you the details. Where the data comes from and how big that dataset is. How much was reserved for testing and whether it was reserved randomly. Ask them what they hope AI will achieve, then ask them how that correlates to human interaction. Most AI today is still evaluating or replicating human interactions with something—most AI in IT particularly so. What algorithms were used to validate? And finally, make sure to ask for real-world results. I know people in data science and one has confidently said to me, “My models are 95% accurate.” When I finally dug into that claim (because my experience is that real-world use of AI/ML is almost never that accurate) it turned out that claim was true … for the reserved test data. So, the same dataset used to train got the same results on the reserved subset. That’s a good thing, but it is no indication of how said algorithm will do in the wild with unfiltered data. The big problem really is the switch from test data to real-world data.

    In fact, in IT, we are a little better off than most spaces wrestling with AI/ML, because very often there is no difference between test and training data in IT. A network security vendor will test against actual traffic—because they have access to massive volumes of it and it shortens the iterations to use what’s at hand and what the system is being trained for.

    And keep rocking it. The current iteration of IT is the opposite of the dark comedy future depicted in Idiocracy. You all are putting up systems and keeping them running at increasingly fast cadences; sometimes remotely because of COVID-19 or other work-related issues. Don’t stop turning out success for your coworkers and customers and take a moment to force vendors to explain what they’re doing with AI. If you’re a generalist like me, ask over email, though, so you can go look up the answers and make sure you understand the details.