devopsy.kinsta.cloud

Tag: Secrets

  • It is Time to Secure Git

    It is Time to Secure Git

    At this point, we’ve got a ton of experience with the bits of Git that we use. And locking down Git is well-documented. Okay, it is documented; we can say that, at least. Interestingly, much of the ‘how to secure Git’ information out there is actually ‘how to keep critical information out of Git,’ and that is the problem. ‘Critical information’ always includes warnings to “not put configuration information into Git. Ever.” And that has been the default way to lock down Git for nearly its entire existence.

    Then along comes GitOps. What is its purpose, in a nutshell? To put that critical information into Git. So we have been waltzing along; only a fraction of us actually locking down our implementation because we were making sure the data in Git, while valuable, was not full of bits of information that would be useful in compromising systems. But now it will be. In fact, in GitOps, nearly everything is one of those bits of information useful to attackers.

    All of which means that it is time to get serious about Git security. Not content, but actual Git security. I won’t go crazy with links to lockdown documents here—it is a vastly different process to lock down a local Git repository versus a remote one and it is drastically different when you’re locking down a service like GitHub or GitLab (actually, some of the work is done for you with a SaaS vendor) … so, you’ll have to look it up yourself, based on your Git implementation. But there are documents out there that walk you through it; you’ll just have to make sure you find the ones that say, “How to lock down Git repositories” and not “How to keep privileged data out of Git repositories.”

    But whatever you do, stop putting it off. Now, if a neer-do-well gets into your repository, it won’t just be source code for various apps that are leaked—and that is certainly bad enough, even if your organization is not a software organization—it will be infrastructure information that can be used and abused. So make certain that information is protected in every possible way. Point security at Git and tell them, “We know you are insanely busy, but this is the most important thing to protect if we are going to implement GitOps.” Because it quite literally is. If database configuration, network configs and possibly even login info are to be stored in Git, locking it down (both local and remote copies) is imperative. You wouldn’t leave that info in plain text on a centralized data store and count on existing protections. So don’t do the equivalent with Git.

    If GitOps isn’t in your immediate future, that’s still your source code full of vulnerability advertisements out there, so at least ask your security team to give your Git install/policies/config a review. Even as a developer, I am less worked up about source code repositories than configuration repositories. There are several reasons for that, but the big one is you all. The bulk of organizations just don’t seem that worked up about source code repository security, but that changes when operations is asked about the list of holes in the firewall being moved into Git—and being configurable from Git (as just one example).

    So, keep kicking it! You all are making, delivering and securing a metric ton of software, and the complexity of our environments did not decrease after the remote work wave hit. Yet, you are still rocking it. May you have enough staff to do the job and applicants that come looking to work with you over competitors. And keep showing the world that software is manageable. But secure your Git.

  • DevOps Teams Struggling to Keep Secrets

    DevOps Teams Struggling to Keep Secrets

    A growing number of organizations are suffering security incidents related to exposed secrets in DevOps CI/CD pipelines, according to a recent ThycoticCentrify report.

    The study paints a troubling picture: Only 5% of survey respondents said most of their development teams use the same secrets management processes and tools.

    The incidents run the gamut, from secrets published in the clear in public cloud code repositories to insecure third-party code to vulnerabilities in the organization’s own code or configurations.

    The problem is often made worse by security teams not knowing what to protect because development teams have their own cloud infrastructure or the ability to create internet-facing applications directly without security teams’ knowledge.

    “The reason why we see so many security incidents related to exposed secrets in DevOps is that using hardcoded passwords and keys is the easy path to getting things to work,” said Joseph Carson, chief security scientist and Advisory CISO at ThycoticCentrify. “We need to make it just as easy for DevOps to use secrets vaults so even when secrets get compromised they are no longer usable, valid or expired.”

    Carson said moving away from persistent privileges to just-in-time, on-demand privileges will reduce these types of misconfigurations and attacks.

    “Unfortunately, many developers continue to use hardcoded passwords or static keys that, when compromised, allow an attacker to gain control of the application resulting in either data exposure, ability to change the application configuration or move laterally around the network,” he added.

    John Bambenek, principal threat hunter at Netenrich, said the fundamental problem is that when organizations go to agile and DevOps, they remove all friction to moving code quickly—and that almost always includes removing security.

    “That means security is inherently reactive instead of being embedded into engineering, and that’s assuming security even knows what to protect in the first place,” he said. “It’s the natural consequence of removing security from the equation and expecting that it’ll just work on the backend. That approach hasn’t worked for decades; a JIRA board isn’t going to suddenly fix that problem.”

    From Carson’s perspective, secrets management is the ability to move away from hardcoded passwords or static keys to just-in-time privileges or one-time-use passwords so even when comprised they cannot be used.

    “Many privileged access management solutions that protected privileged access for years have extended functionality to developers to help move the value into DevOps so they can manage credentials for applications, databases, CI/CD tools and services without causing friction in the development process,” he said.

    Approaches like privileged access security helps enable API-as-a-service and provides instant availability of secrets, SSH keys, certificates, API keys and tokens.

    Secrets Management

    Bambenek added the problem isn’t choosing a secrets management process or tool, but rather that they aren’t in place at all.

    “Pick something that will keep keys and secrets out of public cloud repositories that developers will use that allows for quick and easy rotation of keys as the need arises,” he said. “The most important thing is that it’s the same tool everywhere that operates in a near-invisible process for developers. Four decades of software vulnerabilities have taught me that if you rely on the developers to do the secure thing, you’ll consistently be disappointed.”

    One Wrong Step

    Davis McCarthy, principal security researcher at Valtix, pointed out developers are not security practitioners by trade.
    “Developers are building new ideas, and no good idea comes without experimentation,” he said. “Experimenting with a new technology stack may lead to misconfigurations that unintentionally expose data.”

    McCarthy said shared SSH keys between users create a larger attack surface, and users will store secrets differently creating multiple attack vectors.

    If one user’s endpoint becomes compromised and their SSH keys are exfiltrated, it’s difficult to know what the first mitigation steps are. Between insider threats and malware, the wrong step could lead to a breach.

    “Whereas a software engineer may hold great knowledge of secure coding methodologies, web application attacks and encryption, they are still focused on executing their craft,” he said. “Organizations that do not build security into the development life cycle create a systemic risk that leads to failures in best practices.”

    McCarthy added organizations can centralize and embed secrets management into the tools already in use in CI/CD pipelines by creating an inventory, auditing it and frequently rotating secrets to sensitive environments.

    Bambenek said the short answer is to make sure the infrastructure of CI/CD pipelines is centrally managed and controlled without the ability of engineering teams to ‘roll their own.’

    “No security control will remain intact if someone can pick their own tools without any real governance,” he said. “CI/CD processes don’t require engineering teams to manage the backend of the deployment pipeline which means someone can ensure those functions are in place to begin with.”

  • Don’t Look at This! IT’S A SECRET!

    Don’t Look at This! IT’S A SECRET!

    To continue the discussion about secrets after perusing this excellent report by GitGuardian—last time I went a little nuts about the number of secrets exposed in IT folks’ personal repositories. And it is a lot. I mean a lot of secrets. But you know what is scarier than, “A lot of secrets are leaked in personal repositories of people rushing to get this week’s sprint in”? The fact that 30,000 secrets were leaked via corporate public repositories in 2020. That’s thirty thousand.

    This is not some poor DevOps person in a rush to meet deadlines and with no real security support for personal repositories who dropped some secrets into a public personal repository.

    This is a system set up by your organization that should have security awareness, should be getting scanned regularly and that should never be a source of information for hackers. Instead, it is a source of information, perhaps even critical information, that hackers can easily snag. Because there are secrets in this public repository.

    The first step to avoiding this problem is to acknowledge that IT can’t serve every need, and ask business units (BUs) to come forward with any apps they’ve developed behind your back so that (non-invasive) security can be applied to those apps.

    You’ll either get some apps, or people won’t trust you and you won’t get any. In the latter case, truthfully explain about secrets hiding in public repositories and ask them to validate their own. It’s the best you can do if people won’t share what they’ve done, and there are still people out there that think IT involvement is a huge impediment to their software. ::Shrug:: As long as you can get them to check their apps for vulnerabilities, it shouldn’t matter too much. But I’m more concerned about getting the secrets, so I won’t debate the centralized-versus-distributed-app philosophies in this blog beyond saying, For secrets, if you can get them to scan/clean them up, the question of who is managing the app and the process isn’t very important.

    I’m still shaking my head—30,000! That is more than 100 secrets checked into public, corporate repositories each day (on average, of course). Please, go through your org, make sure you are not contributing to this total. That is wild; even if you account for the fact that if an app has one secret out there, it probably has multiple; even assuming each vulnerable app had four, that’s still 7,500 apps exposing secrets in the most preventable way possible.

    So, if you’re not certain about your public repos, go get certain. The ROI on “a few minutes for a scan, or a couple hours for a person to search for them” versus “our AWS secret was compromised,” or, “our API key was discovered and abused,” is pretty high.

    And keep rocking it. The sheer volume of apps out there is part of the shared-secrets problem, as is the growth in use of secrets as automation and DevOps have taken over. But you’ve got the solution, just make sure they’re not hanging out there in a public repo. And then go back to doing what keeps the business alive.