devopsy.kinsta.cloud

Author: Don Macvittie

  • The New(ish) Complexity

    The New(ish) Complexity

    The API economy and microservices architectures, led by Agile and DevOps, have introduced a massive amount of hidden complexity. I call it hidden complexity because we have a tendency in IT to focus on the problems in front of us, resolve them, then focus on the new problem in front of us. This leaves technical debt in little piles behind us, like the bits that fall off the ends of a snowplow blade (or the asphalt that squishes out the end of the steamroller for our southern friends).

    I was thinking about this when I was looking into how a majority of you handle scalability. The layers of complexity for scalability are surprising. In the past, we invested upfront to be prepared to scale to the levels we need. Today we allow expenses to grow as users grow. That is a great model if users are tied to revenue. If there is any fungibility at all in the correlation of users to revenue (and there almost always is), this is a potential disaster waiting to happen. “Our traffic spiked 200%, but revenue only grew 20%,” is pretty much the last thing your business leaders want to hear. Because in the “Pay as you grow” world, a 200% spike in users means a significant spike in overhead. How much overhead varies pretty wildly, of course, so I’ll just say “Guaranteed and significant” and leave it at that.

    This is a known bit of complexity, but those of you I’ve spoken with have done little to hedge against it. That’s … interesting. How many other hidden land mines are out there, being ignored by us?

    What can you do? That greatly depends upon your business. with internal systems, you have a lot of say in usage. External systems are where this is potentially an issue. You can pay your provider to block DDoS, so at least you’re not paying for that bandwidth if that happens—and most of the people I spoke with really do have some form of DDoS protection that stops traffic before it becomes a fee-generating hit. But for actual growth, the best you can do is get with business leaders and determine what influences the correlation between users and revenue. What makes those people upping your traffic actually spend money while they’re upping your traffic? What can you do to convince more of them to spend money? What services are you not offering that might be useful to those customers and not significantly increase overhead. Basically, this is the modern equivalent of the conversion problem. They’re on your site, how do you convert them to paying users of your site.

    We really can’t get more detailed than that—you know your business, and a media site is a different model than a retail site, for example—so you’ll have to develop beyond that. But have a plan that does what you can to make sure that when expenses go up, revenue goes with it.

    And keep rocking it. We’re outside the historical IT realm here—because you’re all rocking it so well that we have new things to worry about. Give those customers rockstar apps to work with and figure out how to make them paying customers. Develop those bonds with business leaders; you do indeed need each other.

  • …And all Those Other DevOps Tools

    …And all Those Other DevOps Tools

    When we started going through the DevOps toolchain for our new project, we found a ton of bits that normally we (and most likely you) wouldn’t pay too much attention to, because they don’t matter—until they do. Then we all pay attention. That’s why we added them to the toolchain list of options before they became a problem.

    It is time to stop, take a look around, make sure you have an inventory of what tools are going into the process of making your secret sauce. The easy one is IDE. For the last little while—say, a decade—we’ve let this largely be a team or an individual decision, but the steps required to do things and the add-ons/libraries used make this less and less about individual choice and more of at least a team, often an organizational decision. We’re doing it at the project level, so functionally equivalent to team. Our Java/Spring app is done in Eclipse because most of you are doing the same for that architecture. But our new Godot project will be in the built-in Godot editor (for Godot Script) and VS Code for C#. Again, because that’s all the rage in several languages, including C#.

    Here is the thing though – if you are using VS Code, you should keep a detailed and current list of plug-ins. I’m not a fan of their whole extension installing methodology through the command palette and want to make sure everyone has the tools needed at hand. In an enterprise setting, the ROI is there to just set it up and distribute to developers what they need in a package. But this is only a real possibility if most developers are on the same platform.

    What Git Desktop client is in use? While all of them support Git command line in the same format, helping each other use the UI for different operations kind of begs for standardization. Our team doesn’t change much, so I’m not terribly worried about this particular issue. We’re officially using GitHub Desktop, but as long as I’m not doing tech support (because you have a tool that you can’t use well), I don’t care if people stick to it. You might care more about that than we do—in this regard, we’re more like a startup than an enterprise. Since they share a command line, we can afford not to care. If you, as an enterprise, are providing ready-to-code environments, that is a different story.

    A couple of years ago we would have had to worry about containers, but these days we’re okay saying Docker locally and Kubernetes for production. While the Godot app doesn’t have a server component (hence the reason we’re keeping the Java/Spring app around), the build environment will be containers because we want to be able to build wherever we need to.

    Keep rocking it. I just thought I’d drop a list of what we’re going over—there is more, of course; this bit was focused on the developer tools side of things. I’ll eventually update with the final full stack, but for now, this bit is being put on the back burner while I look at some other areas. ::waves at security peeps::

  • Meet the New(ish) Pipeline

    Meet the New(ish) Pipeline

    My company keeps a sample project running pretty much all of the time. Our current project (multi-target web/Android app using Java and Spring built with Gradle and Ant that we’ll call “Finder” here) is getting a bit long in the tooth, so we have been casting about for a new one over the last several months. From non-work sources, we came upon the idea of multi-targeted Godot with C# as a viable side project. Moving to a completely different set of tools is kind of the point; it makes us stretch, and that is the idea of our ongoing projects. The stated goals for these projects are:

    1. Stretch knowledge about different languages, DevOps tools and platforms.
    2. Keep doing what you all are doing. Too many pundits have “grown out of” hands-on work, and it shows.
    3. Try out new ideas/architectures.
    4. Avoid “kitchen sink.” We can’t have a current test app that includes all available technologies, so keep it streamlined. Approach this as a production app that we use for testing, not as a test app.

    That’s why we’re looking to replace our Java/Spring project. We’ve got that pipeline down, we’ve tweaked the app until we are happy with the core and it hasn’t changed in a good while. But change is a part of DevOps, so we’re off.

    The Obvious

    Let’s take care of the obvious elements in our toolchain right away:

    1. Git. We’ll use GitHub, but it was a given that some form of Git would be holding our code.
    2. Visual Studio Code. We’re doing C#, so this is going to be the most useful option for code editing. The built-in Godot editor is good and will be used, but VS-Code offers C# functionality add-ons.
    3. Antsle is the deployment tool of choice, though this has changed over the years. We’ll use it to set up install sets for clients.
    4. Jenkins or Hudson. We’re going to use this only for test management.

    The Less Obvious

    There are several constraints we want to (or have to) meet. First, we want to avoid using Jenkins or Ant to control the entire build process, mostly because we’ve done both in previous projects, and most DevOps teams are no longer using these methods for new projects. Second, this is a sample we keep, not an investment, so we want to use open source technology if at all possible, to keep expenses down. Third, we’re using a game engine to do app development. This has some impacts–like most game engines, it is in control at runtime and we need to bundle the engine in the final application.

    1. The Godot Engine. Godot is written as a game platform, but its graphics/cross-platform abilities make it useful for graphics-heavy apps like ours. But we have to include the engine and its config files in the pipeline.
    2. Code Scanning. Static analysis was a late addition to Finder, and it’s in Java. This project is in C# and Godot Script. We have not yet found a tool that supports Godot Script, but C# we can get; it just adds restrictions. So we will use a static analysis tool, but we’re still deciding which.
    3. Build Automation. We’ve used Jenkins, Gradle, Maven and Ant to handle this step at different times in the past, but something newer would be a good idea. We are still evaluating this step. We will likely end up back on one of those tools because they are still used today. Godot is a new element to us, so we are also evaluating what is required to hook into it and make functioning apps in a repeatable process.

    Fueled Innovation

    We’re having fun—first couple sprints have us running a working prototype, so we’re happy with that. There’s still a lot to implement in the app and in the pipeline, so we’ll no doubt have more to say here soon. We’re talking about putting a server component in—not because the original concept needs it, but because our outgoing app—Finder—had server and client sides, and our need for things like Postman go away without this architecture. And yet, API security, testing, etc. is an important part of the DevOps architecture. As you can see, we’re still iterating.

    As to Finder, we normally take down the pipeline for a previous playground project when the new one goes up, but Finder costs us little. There are five containers (three permanent for the app hosting; two for build/test) and some processing power, so we’ll probably keep it since it is using a different set of pipeline tools (and languages—it is mostly Java/JavaScript).

    Meanwhile, you all keep rocking it. We’re building an(other) app that no one but us will ever see—but you are keeping the enterprise humming. Kudos to you!

  • How Does That Stack Up?

    How Does That Stack Up?

    There’s an entire class of successful innovations made up of things that people didn’t realize they needed. Things that were an obstacle they found a way around and didn’t think about until the innovation showed them an easier way. Think of things like virtualization (hardware was the bottleneck, but “That’s just the way it is,” until it wasn’t) or web monster shopping sites (“They don’t have that in the stores; I guess I’ll live without,” until you could order the weirdest or smallest things from the likes of eBay and Amazon).

    My rule here is that I only talk about products and services if I use them and find value in them. And even then, I try to do it sparingly. But in the spirit of the innovation story you just read, I’m going to point you at one of the coolest new websites on the net. I don’t know the company; have not done business with them, couldn’t even name a principal. But I have spent a surprising amount of time on their site, both in the Feed section and Browsing Stacks.

    The site is Stackshare.io, and the reason I think it is so innovative is what and who it is aimed at: Listing companies’ DevOps stack and offering advice on different parts of the DevOps stack.

    The Browse Stacks section is for presenting what other organizations are doing, so you can get good ideas for your own stack. But the developers of the site appear to have recognized the other useful aspect of this information: Matching organizations to prospective employees based on technologies. You can go look and see that Company X is using GitLab, for example, and that can then influence your job search.

    The Feed section is just a running discussion that uses categories to ask questions or offer insight and advice about decisions a given organization came to—so, “Chose X over Y” is actually a standard subheading for posts, as is “Need advice on Z” or “Recommends A”. It makes for great reading, and no doubt while reading through, you’ll have some advice to offer one of the “Looking for advice” posts.

    The Stackshare.io participants seem to range in size from startup to massive, and that is one of the things I hope they change as they grow—the options for looking at DevOps stacks are “Popular” and “Trending”. We all want more filters than that—organization size, number of apps, stuff like that—to help us zero in on environments that are similar to our own. My little company does not have the needs that Facebook does. And while it’s true that some startups use more tools in their toolchain than Facebook, you get the idea.

    Another issue I had with Stackshare.io’s site (that I fully understand, but still wish could be different) is forced registration/login to see some data. It’s not even particularly sensitive data—I know what ELK and Azure Data Services are, so putting the definition/description behind a registration page/login is kind of silly. But every free site needs data to pay the bills, and I suppose I’m glad it is info that’s also available elsewhere that is behind a reg wall.

    The company offers APIs for integration to your site/tool and private company instances of StackShare to help you manage one of my other gripes—the five (or fifty!) different DevOps stacks many organizations have allowed to develop in-house.

    So if you are job hunting, Stackshare.io can help you find companies doing what you would like to do or know you’re good at. If you are an organization trying to improve or set up a new toolchain (which is how I found the site), it has a lot of information to digest, though not all of it is easily accessible at this point. Either way, keep kicking it and keep an eye out for tools that do the job, no matter where you are looking.

  • That Other Continuous: Data Protection

    That Other Continuous: Data Protection

    Looking around at things to add to our current project’s toolchain (that is one of the reasons we do these side projects; to check out all sorts of new ideas/tools/etc), and researching the company where a former coworker is the CMO, I was drawn into the resurgent world of continuous data protection (CDP). It is interesting that we could end up adding CDP to the list of ‘continuous’ activities, like CI/CD/… CDP, even though CDP was named long before any of our DevOps or Agile terms were born.

    The whole idea of CDP is that it is always backed up, so you are continuously protected. It largely failed when first introduced because it massively increased both the cost and complexity of your storage environment. Essentially, it required a hot copy of all of your storage. Not something most enterprises were willing to invest in. I mean, it was friggin’ expensive. And it used virtualized storage, which no one wanted to mess with. Long story short, at the time, virtualized storage introduced issues no one wanted to deal with.

    But several things have changed over the years, and it is interesting to see where CDP concepts have cropped up. You see, today it is being used to protect companies from ransomware. It’s still a bit of complexity, but we’ve become experts at hiding complexity. It still requires a lot of storage (I would argue that the current versions I’m seeing require more storage than the original CDP solutions did), but when your environment is taken over by ransomware, a clean machine can be used to restore data to a point before the ransomware took over. From a “continuous protection” standpoint—giving it a modern meaning along the lines of “continuous testing”—that is an astounding feat.

    The way it works—at least the way that the current implementations I’ve looked at work—is pretty simple at heart, though as my former coworker likes to say, “Easy in concept, difficult in implementation.” It keeps every write ever made to storage, with time and currency markers. So it knows what the disk currently looks like, but it can also get to what the disk looked like at 12:33 am yesterday. Or January first, or a year ago… You get the idea.

    It uses a ton of space because there is an implied truth here—nothing is destroyed. In traditional storage, if you change a document and hit save, the document is written over. In the new iteration of CDP, if you change a document and hit save, the changes are recorded, but the original is not altered. Call it “non-destructive writes,” if you will.

    It is worth checking out. The concepts are huge, and offer protection that is otherwise in short supply. If you have a dedicated DevSecOps team, they’re the ones I’d point at it, since the big reason for using this is a security reason at heart (though there are other uses; I’m currently gushing about security because this is a big deal).

    I’ll get back on track and report back about things directly in the toolchain for our newest DevOps project next week. I thought this was worth writing about, since most of you all could use the concept, even if only for peace of mind. You’re rocking it! Protecting your work against the newest range of attackers is just common sense. It’s worth seeing if the modern iteration of CDP is for you.

  • So Many Options, So Little Time

    So Many Options, So Little Time

    So many language/framework options, so little testing time, that is.

    Recently, two projects I was working on drove me to check out a breadth of languages and/or frameworks to handle cross-platform development. While Unity and Unreal have traditionally been my game platforms of choice and I typically use Xamarin for cross-platform (though native development is more common for us on both Apple and Android), I decided I needed to do the occasional sweep to find out what is new out there.

    The two projects are a game on which I’m helping a new game developer learn and a graphics-heavy application that is unsuited to web development. Both would work best in a cross-platform environment; both would benefit from a game engine’s graphics manipulations. But both could be done in any language that supports graphics, as there isn’t a heavy physics requirement to the game.

    So I went looking at what’s available now and how it fits. From React Native to Godot to Flutter and even Kivy, I kicked the tires on each, got them running, ran and traced samples, wrote some code of my own, even talked with friends that were using them to get their opinion.

    Alternate frameworks languages

    The one thing that came through loud and clear? There is no way that security and testing are keeping up with this language/framework proliferation. Kivy has a language (more of a specification than an actual language, but still—source code) that can run on top of Python (it’s not required to use the framework, but it’s kind of assumed you will if the app gets beyond a certain size); Godot has a language that is integral to the framework and that you use pretty much everywhere. React has a ton of add-on code and both Flutter and Dart are new languages.

    I like stuff from each of them. I don’t like that we’re going to use them in ever-increasing amounts and we’re going to expect testing to stay on top of those implementations. Each of these platforms/languages/environments has a large following that is doing some of the testing, but since they’re all relatively new, “How much testing?” is a valid question. And tests have to be done in those languages, implying that those writing need to know them and so do testing teams.

    Where did we end up? We haven’t made a final decision yet, but taking input from the person I’m mentoring, coworkers and those who are using each of the above, we’ll probably end up using Godot. For 2D apps, it is the bomb. Our team is small, so I can offer my knowledge of Godot to shore up testing and everyone agrees that its Python-like language is acceptable, while the graphic design element is nicer than the competition. Both applications will benefit from Godot’s graphics processing capabilities and the game will benefit from game elements built into the Godot Engine.

    Here is an interesting side note: All of the products we looked at are open source. Both of the default products that we (and just about everyone else) would choose for the game are commercial. That’s interesting, but we don’t need the weight of those commercial solutions for a simple 2D interactive game/app.

    For you all, the questions might be harder to answer. With larger projects come larger requirements for source size, test, you name it. Choose carefully, with an eye on the future. Personally, I think React Native, Godot and Flutter would all be good choices for cross-platform applications, with Godot the best solution for cross-platform 2D games. And all three are likely to be around with a good size following for a long time, making them safer choices. That is less clear with Kivy, though the popularity of Python appears to be driving growth in Kivy use.

    Whatever you choose, plan for all phases of development, not just writing code. That’s true of everything, but new cross-platform tools are complex beasts, so it is worth reminding you all.

    And hey, if you have a cool app you developed with one of these tools, point me to it. Always good to see what you all are doing, and nice to see the breadth of solutions these languages can create.

  • Yeah, You’re Doing DevOps

    Yeah, You’re Doing DevOps

    The Stack Overflow annual survey of developers came out this month, and it has a lot of interesting data points in it. There is a lot that I wish was different about their questions/methodology, but one thing caught my attention. Docker, Kubernetes and similar tools (Ansible and Terraform are in there at lower percentages, for example) make up more of the responses than can be accounted for by people with “DevOps” in their title.

    Sometimes by a lot.

    That means we have a fair number of developers that are doing DevOps, but who aren’t DevOps team members. Yeah, there are a few systems admins in the ranks of the respondents also, but not enough to cover the discrepancy. If every respondent that was in DevOps and sysadmin roles used Kubernetes, there would be just enough to cover the number of respondents that said they used Kubernetes “extensively” over the past year. It’s unlikely that all of them are Kubernetes specialists. And that doesn’t touch on Docker, with a whopping 40% of respondents saying they use Docker extensively. Any developer with recent experience can explain away Docker—though “extensive” is a bit strong for your average developer’s usage, it is becoming part and parcel of development, using containers to host dev environments, test, even build tools. And, of course, some deployments. But 40% is still a lot.

    So, if your title is developer, and you’re using deployment tools on a regular basis, go ahead and call yourself DevOps. That’s not the only criteria, but if you were going to pick one, the breadth that using deployment tools represents combined with the implication that there isn’t an ops person worrying about that type of thing makes “uses deployment tools” a good stand-in test.

    To the nearly 18% (20% of pro developers) of you using Yarn, I tip my hat. I used it for a couple of years early on in its life, and am happy to say I do not need to use it any longer. I assume your usage is not tied to Hadoop and that it has greatly improved over my 2015-2016 experience. Still, y’all rock, just by definition of using it to get resource scheduling down. It ain’t easy.

    Across the board, the number of DevOps/Ops tools that make these lists is amazing for a developer survey – nearly 9% of respondents use Ansible, and nearly 9% use Terraform. There is undoubtedly overlap, so the actual number that uses one or the other is less than 18%, but still, nearly one in 10? That’s huge. And that’s DevOps or Ops, not purely developers. Regardless of title.

    And that’s been a hallmark of development since the field took off—doing what needs to be done to deliver software. This is just another iteration, and you’re rockstars, regardless of title. You keep delivering apps that run the business—let them call you whatever they like. Money talks; titles are pretty much meaningless. Not like you’ll retire on a title, but bonuses or stock grants? Those you might retire on.

  • The State of DevOps Is Just Fine

    The State of DevOps Is Just Fine

    There is a growing amount of hand-wringing about the state of DevOps; that adoption is not progressing apace, or that the full benefits of the methodology are not being realized by the majority of adherents. That we can and must do more and do it better, etc.

    That is just stupid. Or, my first reaction: “Bunk!”

    Yeah, I called an idea stupid. No, I didn’t call you stupid. Don’t give me a reason to. The world is full of stupid ideas, and this is one of them. Here’s why.

    DevOps has seen a decade of unprecedented adoption – yes, unprecedented. PCs were not adopted by businesses at the rate that DevOps was adopted, once it caught on. Virtualization took the business world by storm and, frankly, wasn’t even close to the adoption rate of DevOps. Agile greased the wheels and got people thinking about better ways; we all knew the weaknesses of how we were doing things, and DevOps gave us an option that addressed those weaknesses. We took it. We ran with it.

    So talking about how it is stalled, when adoption is darned near-ubiquitous? Silly.

    Those concerned are further zeroed in on one aspect of DevOps – not enough culture change. Let’s face it—automation is easier, the ROI is more obvious and even when ROI is vague, the elimination of busywork from the schedule of developers and operations alike is huge. Some organizations were never going to move beyond that point; others eventually will, but aren’t in a rush. Organizational changes do not define DevOps any more than automation does—DevOps is the process of streamlining software creation and delivery—and both automation and culture change contribute.

    The “full benefits” of DevOps are … what, exactly? That is another silly part of the proposition that a slowing of DevOps adoption is negative. Automation improves the creation and delivery of software, silo elimination improves the creation and delivery of software… The full benefits of DevOps are improvement. That has always been the stated goal—to make it better, smoother, more repeatable. Wherever an organization is on implementation, it is almost guaranteed that their creation and delivery is “better, smoother, more repeatable.” So, benefits are accruing.

    And each organization has different needs. DevOps is a methodology. It has never been the perfect solution to every IT problem—though there are still pundits that will try to tell you it is. Automation is always a positive; Agile and DevOps—well, that is less true. You know your organization and your apps, so the “right methodology for the job” is absolutely your call, not some vendor’s or pundit’s. So, “adoption slowing” partially represents orgs/projects/apps that have opted not to pursue some of the more radical changes DevOps implies (yes, radical. Automation is just a logical extension of scripting; redesigning team organization and reporting relationships is absolutely radical by comparison).

    Which is a good place to leave off. Your apps, your org, your environment. You know what is going to do the most good. So keep doing it. Push for it if the org isn’t there yet, be the change that is needed. But be aware of cost/benefit. Sometimes, the changes being asked for just aren’t worth it. So make sure they are before you push too hard, and if they’re not, make sure you’re comfortable with why and can defend that decision.

    And keep rocking it. Those apps are delivering to users, no matter where you are in DevOps adoption. And that says it all about you and the team. Thanks, for all of your users that never think to say it.

  • Swinging the Pendulum of IT Back to QA

    Swinging the Pendulum of IT Back to QA

    I was reading Nuria Manuel’s excellent article about quality assurance (QA), and it got me thinking about the wide variety of ‘pendulums’ that impact IT. I call them pendulums because they follow the rule of “every action has an equal and opposite reaction,” and they work on fundamental truths of IT. For example, the one that has probably been going on the longest is centralization/decentralization of IT and its assets. No sooner is everything consolidated and cost savings realized than adaptability concerns and needs of individual business units start the process of decentralization. Then, once decentralization is realized, everyone is painfully aware of the chaos that fifteen miniature IT departments, each going their own way, can cause … and centralization begins again.

    Quality assurance is another such pendulum. Much like soldiers in peacetime, most people don’t recognize the value of QA efforts when things are going well. And when they’re not going well, then it is considered a QA failure—assuming there is a true QA function. Otherwise, it becomes the impetus to launch a formalized QA function … until things are going well and budgets are tight. The problem with QA is that for years, timelines, budgets and Agile management styles have all conspired to swing that pendulum away from a formal QA function. Oh, companies pay the price, but they quite often don’t see it as a cry for a valid QA function, and just treat this particular issue as a one-off that they should make sure doesn’t happen again. Unfortunately, issues in production have become both commonplace and accepted because of this. The only issues in production that seem to actually generate angst are ones that keep users from accessing the product. Anything short of that and workarounds or delayed fixes are seen as okay.

    But the apps you are rush-building today will still be around in 10 years—or, at least, some of them will. Some of them won’t for a variety of reasons–including quality issues. Those that are around in 10 years are going to have to be maintained. QA now can save you over the years, so why all the resistance to it? QA might delay the release of new app. It is understandable that delays are to be avoided if possible, but so are bugs, so maybe constraining how long QA can delay the release and building that number right into the schedule would be a decent compromise.

    That way, QA is looking at the most important/used parts of the app, but not getting buried in the weeds (where there might be bugs, but the ROI for addressing them probably isn’t there). The app will be delivered on time because QA’s time was built into the schedule. This does require that we not allow the traditional and abhorrent practice of shortening the QA window to make up for developers not meeting deadlines. Let us be very clear about this: when developers are rushed because they missed deadlines, that is precisely the time that you want QA to look very closely at their code before release. The last thing you should do is less testing in this scenario. The absolute last thing.

    This is completely doable in your environment. The only really difficult part is holding the line on QA. Business owners are going to want to meet their original timeline, but cutting any phase of testing to get there is just a bad idea, so the struggle will be to hold the line. And figure out why devs missed the deadline in the first place so that the issue can be avoided for the next project.

    You all are rocking it. Make sure all that sweet code-and-deploy work is not undermined by quality issues! Go ahead, implement a QA policy and make it an actual policy. Keep rocking it, get accolades from the rest of us. Maybe even from your employer. Maybe. But definitely from us. Because you’re killing it, month after month.

  • Don’t Look at This! IT’S A SECRET!

    Don’t Look at This! IT’S A SECRET!

    To continue the discussion about secrets after perusing this excellent report by GitGuardian—last time I went a little nuts about the number of secrets exposed in IT folks’ personal repositories. And it is a lot. I mean a lot of secrets. But you know what is scarier than, “A lot of secrets are leaked in personal repositories of people rushing to get this week’s sprint in”? The fact that 30,000 secrets were leaked via corporate public repositories in 2020. That’s thirty thousand.

    This is not some poor DevOps person in a rush to meet deadlines and with no real security support for personal repositories who dropped some secrets into a public personal repository.

    This is a system set up by your organization that should have security awareness, should be getting scanned regularly and that should never be a source of information for hackers. Instead, it is a source of information, perhaps even critical information, that hackers can easily snag. Because there are secrets in this public repository.

    The first step to avoiding this problem is to acknowledge that IT can’t serve every need, and ask business units (BUs) to come forward with any apps they’ve developed behind your back so that (non-invasive) security can be applied to those apps.

    You’ll either get some apps, or people won’t trust you and you won’t get any. In the latter case, truthfully explain about secrets hiding in public repositories and ask them to validate their own. It’s the best you can do if people won’t share what they’ve done, and there are still people out there that think IT involvement is a huge impediment to their software. ::Shrug:: As long as you can get them to check their apps for vulnerabilities, it shouldn’t matter too much. But I’m more concerned about getting the secrets, so I won’t debate the centralized-versus-distributed-app philosophies in this blog beyond saying, For secrets, if you can get them to scan/clean them up, the question of who is managing the app and the process isn’t very important.

    I’m still shaking my head—30,000! That is more than 100 secrets checked into public, corporate repositories each day (on average, of course). Please, go through your org, make sure you are not contributing to this total. That is wild; even if you account for the fact that if an app has one secret out there, it probably has multiple; even assuming each vulnerable app had four, that’s still 7,500 apps exposing secrets in the most preventable way possible.

    So, if you’re not certain about your public repos, go get certain. The ROI on “a few minutes for a scan, or a couple hours for a person to search for them” versus “our AWS secret was compromised,” or, “our API key was discovered and abused,” is pretty high.

    And keep rocking it. The sheer volume of apps out there is part of the shared-secrets problem, as is the growth in use of secrets as automation and DevOps have taken over. But you’ve got the solution, just make sure they’re not hanging out there in a public repo. And then go back to doing what keeps the business alive.