devopsy.kinsta.cloud

Tag: secrets management

Merging Secrets Management and Configuration

Fresh from picking up $5.25 million in funding, CloudTruth CEO Greg Arnette talks with Mike Vizard on this episode of the View With Vizard about why configuration and secrets management need to converge. The video is below, followed by a transcript of the conversation.

Announcer: This is Digital Anarchist.

Mike Vizard: Thanks for the throw. We’re here with Greg Arnette, who’s the CEO for CloudTruth, they just raised $5.25 million to help us with these configuration issues that seem to plague us. Greg, welcome to the show.

Greg Arnette: Thank you. It’s great to be here, and thanks for the opportunity to tell your audience about CloudTruth.

Mike Vizard: So, what is the configuration problem, and why does it not seem to wanna go away? It’s been around for a while, things appear to be getting worse, and what’s to be done about it?

Greg Arnette: Exactly. So, that ties into the history of founding this company. Both myself and my Co-founder were prior CTOs and DevOps leaders for several cloud-based SaaS companies, and we saw that our DevOps teams were continuing to build internal tools to address issues around configuration sprawl and complexity. And these tools tend to be the weak links in terms of solid deployments, good up time, good security, and basically, slows teams down.

So, the history of kind configuration sprawl and what we see as the challenges is sort of driven by a number of macro trends—first, the fast rise of Kubernetes; everyone wants to get on the Kubernetes bandwagon. That introduces new complexities around configuration management, new sets of tools that haven’t been sort of well adopted inside of DevOps shops. Another trend is the breakup of monoliths to microservices adds lots of new dimensions on configuration complexity. And then there are the traditional needs around provisioning infrastructure, configuring your apps, and managing secrets are the sort of legacy challenges around configuration management. And as the cloud gets more mature and with more capabilities, configuration is not getting easier, but getting more difficult.

So, CloudTruth is here to address those concerns. We heard, during our product market fit research where I interviewed over 500 cloud and DevOps leaders on their challenges managing configuration, that this over reliance upon tribal knowledge in terms of how a system is configured and why it’s configured a certain way is just held in key team members’ heads, and that’s not a very durable strategy for going forward or for on boarding new folks. So, there’s an element of CloudTruth that provides collaboration and sharing around configuration settings and context to help reduce that need of, or that sort of burden of the cognitive load to keep track of how are things configured, where is the data, how can I change it, what changed recently—that type of thing.

Mike Vizard: You guys also acquired a company in this process in the secrets management space, so where does configuration and secrets management converge and how do we think about that?

Greg Arnette: That’s a great question. So, we acquired Tuono, we’re thrilled to bring in the Tuono team, including Jesse St. Laurent, who was the CTO, who joins us as our Chief Product Officer. Jim King, Mike Healey, Mike Stack all joined us, they were the founding engineers at Tuono.

And what we did was, we took the Tuono secret management engine, which was cloud native and we’ve now included that in the CloudTruth platform which rounds out our offering. So, CloudTruth, while it is also a configuration data hub to synchronize your configuration settings from all these multiple tools is also itself a parameter store and secret store that customers can use if they don’t already have existing parameter stores or secret stores in their environment.

Mike Vizard: And in that context, what does cloud native mean? Because some will people say, you know, it runs in the cloud; other people will say that it only runs on Kubernetes, it’s optimized for microservices and containers. Or is there some middle ground there and what do we think about cloud native—what context are we using that phrase?

Greg Arnette: Yeah, when we use cloud native, we mean or refer to the capabilities that are provided as part of core services at Amazon, Azure, and Google. So, an example is, Amazon themselves provide a parameter store, they call it AWS Parameter Store. Google and Azure provide the same kind of internal capabilities. Customers can also bring their own parameter store, like run something like Console on top of the cloud or using Vault sometimes as a general-purpose parameter store.

So, the cloud native offerings like AWS Parameter Store, Secret Manager, and the equivalent on the other clouds are things that customers can use, they have different price points. We provide a layer of management and visibility and observability to all that config data stored inside of all these different systems.

Mike Vizard: And when you talk about unifying configuration, are we replacing Terraform and CloudFormation, or are we just finding some way to layer some form of abstraction around them that makes the whole process easier to manage? Walk me through that a little bit.

Greg Arnette: Yeah, you know, we’re not replacing Terraform or CloudFormation. We’re designed to coexist with the customers’ existing investments and their configuration tooling. So, whether it’s provisioning tools like Terraform, CloudFormation, Pulumi, or app config tools like Ansible, Puppet, Chef, Saltstack, or secret tools like Vault or the cloud native versions of those capabilities. We coexist and become a metadata wrapper around those different tools.

Now, if customers don’t have some of those capabilities, we have functionality in our platform that can fill the gap, but more and more customers already have these systems in place. They’re not looking to rip everything out and replace it, but they’re looking to address these issues around configuration sprawl that leads to a misconfiguration causing unplanned down time or a security incident.

Mike Vizard: We have been talking about DevOps and DevSecOps forever, and the idea has always been that we’re shifting more responsibility for the IT environment to the left of the developer. Sometimes they like that, sometimes they don’t.

But I wonder, are we not approaching some sort of concern or maybe a backlash where folks are saying, “Hey, you know what? This isn’t really working. We have all this misconfiguration out there, we’re maybe running faster, but things are less secure, and we have more risk than ever.” So, is there a balance to be struck, or what’s your sense of where are we on this journey?

Greg Arnette: That’s a very interesting question. This is something that we hear all the time in our outreach to our customer audience is what their journey looks like for adopting shift left or GitOps types of operating principles. We’ve seen a number of patterns emerge over our hundreds of interviews. Specifically, most teams do want to give developers more capabilities to do things themselves and reduce that burden on the Ops team, but they still want Ops or DevOps teams to have the overall sort of judgment and enforce the guardrails that were put in place by the security and compliance teams and by the other teams that are sort of deciding what are the best practices.

So, with CloudTruth, we’re kinda serving both audiences in a purposeful way. You have centralized control and visibility that DevOps wants. They wanna build and enforce lockdowns, enforce default settings that can only be overridden with a permission or an override that’s allowed as part of a workflow. But you also provide capabilities for core developers to manage the config for their own environments and to suggest configuration changes that then can be adopted more system wide.

So, we’re serving that duality that exists today. We’re gonna let the customer decide on where they ultimately land about developing shift left principles and moving that direction. But we see a lot of interest in that, and it helps to reduce that—you know, again, that cognitive load and that context switching on the DevOps professionals. Because so many times, they’re being bothered by other folks and other developer teams, like, “What’s the setting for this Redis cluster? Why is our Mongo configured this way? How do I tune my elastic search test cluster to match production?”

That information will all be self-serviceable and findable inside the CloudTruth platform. And our role based access control permissioning system allows view only, read only, read/write access to the appropriate areas, and then DevOps teams can sort of wall off the production systems from other people having chances of changing those settings.

Mike Vizard: Mm-hmm. What is the role of the security team as we go forward? It feels like, if we shift everything left to the developers, what are those guys doing?

Greg Arnette: They still need to be involved in terms of ensuring that no bad information gets in and no good information gets out. So, we provide a number of views into configuration data that are helpful to security teams. First is, what changed most recently in terms of your compliance management reports for SOC 2 and other requirements and keeping track of changes. That’s really hard in a dynamic environment, especially cloud-based where you have microservices firing left and right, coming up and down. The old traditional CMDBs that were popular in the on prem world don’t really fit well in the new way that we’re using the cloud.

And so, CloudTruth is a self-documenting change management control system that is, behind the scenes, keeping track of all the version settings of every configuration value that we’re monitoring, and then we provide views into that data that can be consumed by the security or compliance teams to get better insights for their needs without having to bother the DevOps teams.

Mike Vizard: Do you feel like the bad people, otherwise known as cyber criminals, are getting better at scanning for misconfigurations and looking for these vulnerabilities to exploit? And we hear a lot about software supply chains these days—it seems like, in a lot of cases, the root cause of the problem is just misconfigurations.

Greg Arnette: It’s true. We’ve found, in our own research, that 75 percent of unplanned down time or security incident was caused by a misconfiguration when you ultimately read the root cause analysis and do all the internal tracing.

So, what we wanna provide are the best practices as early as possible in the software development life cycle process to implement good config hygiene. And then secondary to that is, once config is out there in the wild, provide monitoring and insight into how it could affect security vulnerabilities or security posture. Since everything these days is defined as infrastructure as code, it’s all about analyzing all the different locations where config values are stored, making sure those are secure and locked down, and then being able to use any kind of pattern recognition to understand where certain areas can be vulnerable to being taken advantage of by the nefarious factors out there in the world.

Mike Vizard: Do you think we’ll get to the point where we can use automation to kinda get ourselves out of this mess? We have APIs and CLIs and everything else that’s out there—shouldn’t we, at some point, have enough intelligence in the work flows to say, “Hey, this thing you’re about to deploy is misconfigured, so don’t do it”?

Greg Arnette: That’s part of our goals down the road is, we have a very thoughtful strategy how we wanna launch CloudTruth as a full configuration data management platform. So, the first goal is to get inserted into a company’s work flows where we have access to the data and can provide value with reports and work flows. The second goal is then, once we’re now managing this data on behalf of the customer, to apply AI and ML, looking for patterns, recommending optimizations, best practices, vetting what we see from a customer perspective against best practices being published by other standards bodies that are looking at this from different perspectives.

One goal of CloudTruth is to be that configuration data layer management system that we think most companies using the cloud need these days.

Mike Vizard: We also hear a lot about artificial intelligence—will that get applied to this space? Is there any scenario where we’ll just have a bunch of machine learning algorithms running around, figuring out what’s going on?

Greg Arnette: We firmly believe that that is definitely gonna be happening a few years down the road. I think the maturity model is just getting started. The first act in getting that up and running is getting access to the data for trending the models and analyzing the trending results, optimizing the model. So, a full observe-orientate-decide-act OODA loop is involved here in terms of making sure those models are accurate, but we think that’s gonna be a big benefit in what we can provide down the road.

Mike Vizard: So, what’s your best advice to folks? What should they be focusing on today to get to that nirvana tomorrow? Is there same way I should be organized, is there some way I should be talking to my developers? What do you think?

Greg Arnette: There’s a couple themes that come to mind. The first is—which sounds almost obvious these days—is make sure you are able to do infrastructure as code ideas at all levels that you can, and that’s tied into automation. So, it’s a big portion to continuous deployment, CI/CD pipelines. Once you start the development of those types of workflows that accelerate getting what developers are creating out in front of the customers, whether it’s an internal app or external SaaS or mobile app, accelerating that team velocity is really a key element in terms of making that possible.

The other area is this notion that you wanna implement systems that reduce the amount of copy and paste. So, that’s what we call the Do Not Repeat Yourself, the DRY approach to infrastructure as code management. So, that’s a key theme behind CloudTruth is, we provide capabilities that let you manage your multiple environments with the faults, inheritances, and overrides from a centralized capability, but still allow flexibility in your environment tree to manage certain sub-environments in different ways as needed.

So, different developers can have their own custom environment, but they’re gonna inherit default values that are sanctioned by the architecture team or the security team. And then the overrides have to be allowed by a permission basis.

So, there’s a couple sort of themes that are coming together as we go through 2021 around automation, infrastructure as code, shift left, and the DRY approach to managing input values to all the IAC tools.

Mike Vizard: Hey, Greg, thanks for being on the show.

Greg Arnette: Thank you very much. Have a great day.

Mike Vizard: Alright, awesome. Hey, guys, we learned a lot about security there and things we didn’t know and—hey, there’s hope for the future. Back to you guys in the studio.

[End of Audio]

February 21, 2022
DevOps Teams Struggling to Keep Secrets

A growing number of organizations are suffering security incidents related to exposed secrets in DevOps CI/CD pipelines, according to a recent ThycoticCentrify report.

The study paints a troubling picture: Only 5% of survey respondents said most of their development teams use the same secrets management processes and tools.

The incidents run the gamut, from secrets published in the clear in public cloud code repositories to insecure third-party code to vulnerabilities in the organization’s own code or configurations.

The problem is often made worse by security teams not knowing what to protect because development teams have their own cloud infrastructure or the ability to create internet-facing applications directly without security teams’ knowledge.

“The reason why we see so many security incidents related to exposed secrets in DevOps is that using hardcoded passwords and keys is the easy path to getting things to work,” said Joseph Carson, chief security scientist and Advisory CISO at ThycoticCentrify. “We need to make it just as easy for DevOps to use secrets vaults so even when secrets get compromised they are no longer usable, valid or expired.”

Carson said moving away from persistent privileges to just-in-time, on-demand privileges will reduce these types of misconfigurations and attacks.

“Unfortunately, many developers continue to use hardcoded passwords or static keys that, when compromised, allow an attacker to gain control of the application resulting in either data exposure, ability to change the application configuration or move laterally around the network,” he added.

John Bambenek, principal threat hunter at Netenrich, said the fundamental problem is that when organizations go to agile and DevOps, they remove all friction to moving code quickly—and that almost always includes removing security.

“That means security is inherently reactive instead of being embedded into engineering, and that’s assuming security even knows what to protect in the first place,” he said. “It’s the natural consequence of removing security from the equation and expecting that it’ll just work on the backend. That approach hasn’t worked for decades; a JIRA board isn’t going to suddenly fix that problem.”

From Carson’s perspective, secrets management is the ability to move away from hardcoded passwords or static keys to just-in-time privileges or one-time-use passwords so even when comprised they cannot be used.

“Many privileged access management solutions that protected privileged access for years have extended functionality to developers to help move the value into DevOps so they can manage credentials for applications, databases, CI/CD tools and services without causing friction in the development process,” he said.

Approaches like privileged access security helps enable API-as-a-service and provides instant availability of secrets, SSH keys, certificates, API keys and tokens.

Secrets Management

Bambenek added the problem isn’t choosing a secrets management process or tool, but rather that they aren’t in place at all.

“Pick something that will keep keys and secrets out of public cloud repositories that developers will use that allows for quick and easy rotation of keys as the need arises,” he said. “The most important thing is that it’s the same tool everywhere that operates in a near-invisible process for developers. Four decades of software vulnerabilities have taught me that if you rely on the developers to do the secure thing, you’ll consistently be disappointed.”

One Wrong Step

Davis McCarthy, principal security researcher at Valtix, pointed out developers are not security practitioners by trade.
“Developers are building new ideas, and no good idea comes without experimentation,” he said. “Experimenting with a new technology stack may lead to misconfigurations that unintentionally expose data.”

McCarthy said shared SSH keys between users create a larger attack surface, and users will store secrets differently creating multiple attack vectors.

If one user’s endpoint becomes compromised and their SSH keys are exfiltrated, it’s difficult to know what the first mitigation steps are. Between insider threats and malware, the wrong step could lead to a breach.

“Whereas a software engineer may hold great knowledge of secure coding methodologies, web application attacks and encryption, they are still focused on executing their craft,” he said. “Organizations that do not build security into the development life cycle create a systemic risk that leads to failures in best practices.”

McCarthy added organizations can centralize and embed secrets management into the tools already in use in CI/CD pipelines by creating an inventory, auditing it and frequently rotating secrets to sensitive environments.

Bambenek said the short answer is to make sure the infrastructure of CI/CD pipelines is centrally managed and controlled without the ability of engineering teams to ‘roll their own.’

“No security control will remain intact if someone can pick their own tools without any real governance,” he said. “CI/CD processes don’t require engineering teams to manage the backend of the deployment pipeline which means someone can ensure those functions are in place to begin with.”

November 23, 2021
Shhhh! It’s a Secret

In one of our regular back-and-forth breaks, Lori and I were talking shop a week or two ago, and she shared an excellent report by GitGuardian with me. Frankly, none of the information in the report was a huge surprise for me–Git is filled with information that companies probably don’t want out there, but that is nearly impossible to track proactively. That’s what GitGuardian tries to do for you, but honestly, I’ve not yet dug into them, so caveat emptor. Or, read someone else’s opinion to help you decide if they’re a good idea – I’m just going to talk about the report.

The key takeaway is that you are asking your DevOps teams to move faster and faster. Putting a section of code they are working on into a public repository is an easy way to “take work home” or share it, especially during lockdowns. It is also a good way to hand over critical information to anyone searching for it—about your infrastructure, or even critical bits of data like SSL keys or credentials. And with the current level of cyberattacks against nearly every industry, I’d say there are people looking for that information, no matter what industry you are in.

This is not really a DevOps-specific issue–credentials and such in code has been a problem for IT for as long as I’ve been working in the field. At some point, any automated system needs to perform AAA, and that means the data required is stored somewhere. A multitude of approaches have been attempted to resolve the problem technologically, but none have been a silver bullet. Yet. Eventually, someone will figure out a way to make it work; then a whole new attack vector will arrive, probably aimed at the tool that is settled on to protect this very data.

This is a personnel/policy issue that is exacerbated by the speed of Agile and DevOps. Your best solution, at this time, is to write a clear policy that contains consequences for putting secret data into a public repository. We’re short on “consequences” right now, preferring to say, “Right, let’s make sure that can’t happen again,” but in the case of putting secrets out in public, it is a different level of ‘bad’ than a coding error getting checked in. Those secrets can actively be used against the organization, and if the not-to-be-shared data is clearly delineated, any violation can be viewed as intentional, be it because of convenience or malfeasance.

According to the report, there are millions of bits of privileged information out there. Not all of them are from enterprise IT shops—but some are. And some might be from your organization. Set expectations, get feedback. Make it clear you do not want to stop progress on delivery of systems, but you also don’t want to implement a system with a built-in back door for attackers to use. It is easy enough to use alternative delivery methods for those few bits of data, and put the bulk of the source out where it is needed.

And keep rocking it. Toward Data Science says there are 128 million public repositories out there (Q1, 2021). Think how many lines of code are in those repositories, and how many secrets potentially are, too. Just make certain yours are not included in the list. No sense keeping your next great solution from running because your critical data leaked.

July 8, 2021
Report: The State of Cloud-Native Application Security

The cloud brings tremendous capabilities in terms of increased deployment fluidity and automation. Along with cloud adoption has come the use of cloud-native tools built specifically for developing applications for this domain. However, cloud-native tools carry some nuanced security concerns, such as misconfigurations, known vulnerabilities and leaked secrets. As such, 83% of organizations recognize security as very important to their cloud-native strategy, according to a recent report from Snyk.

Snyk’s State of Cloud Native Application Security report surveyed hundreds of IT professionals on their cloud-native security concerns. Below, we’ll pick out the top takeaways from the report, identify the most common trouble areas for cloud-native application security and see if there’s a correlation between deployment automation and higher degrees of application security.

The Move to Cloud-Native

Teams are moving to cloud-native tech, empowering DevOps with software-driven architecture and infrastructure-as-code (IaC). In this new cloud-native paradigm, 58% of production workloads are deployed as containers, 21% now are serverless and 50% use some form of IaC within the deployment process.

Organizations are moving to cloud-native technology for a variety of reasons. For one, containerized software affords increased speed — 68% of organizations moved to containers for increased deployment velocity, the study found. This is followed by ease of management (at 67%) and reduced cost (43%). There is also the security imperative for adopting cloud-native tools; 36% of respondents cited security as a primary reason for moving production applications to containers.

Top Cloud-Native Security Concerns

While cloud-native technologies such as containers, Kubernetes, serverless and IaC enable more rapid development, they also bring unique security issues. The report found that misconfigurations and known unpatched vulnerabilities are by far the most common type of incidents among cloud-native environments. In fact, 45% of organizations have experienced an incident arising from a misconfiguration, followed by 38% due to known unpatched vulnerabilities.

Other common cloud-native security incidents include secrets leaks, failed audits and malware. Interestingly, the study also found that data leaks by insiders are more common within organizations with higher degrees of cloud-native adoption. According to the report, 38% of organizations with high cloud adoption have suffered data leaks by an insider, whereas the figure is cut in half, at 17%, within organizations with low cloud adoption. Secrets like API keys must be managed carefully, especially as cloud-native tools force more dependencies. “The need for effective management of these kinds of artifacts is a key differentiator from the more centralized pre-cloud era,” the report noted.

Automating Security Testing

End-to-end deployment automation shows promise, yet is still at an early stage of maturity for most development teams. Though 95% of organizations use some kind of deployment automation, only about one in three organizations have an entirely automated deployment pipeline.

Organizations with increased deployment automation tend to embrace a higher degree of security testing, as well. Highly automated pipelines are twice as likely to use security testing throughout their development lifecycle, found the report. Since incidents most commonly arise from misconfigurations and known vulnerabilities, automated scanning could help identify many cloud-native issues, comparing production code against databases of known exploits.

So, when does this security testing occur? More often than not, it happens within the CI/CD pipeline. Over 60% of organizations perform security testing within the CI system. This is opposed to testing source code repositories or local IDEs and CLI tools early on in the development process. In terms of testing frequency, there is a wide range of executions. For those groups with high deployment automation, 70% test security daily or more frequently. Automated security testing seems to be functioning well, as 72% of fully automated teams find and fix critical vulnerabilities in under one week.

The report also exposed an interesting disconnect with regard to security testing ownership. Only a small percentage of security engineers (less than 10%) feel developers are responsible for the security of the cloud-native environment and applications, whereas 36% of developers stated they are responsible for security. These figures may demonstrate the security onus shifting left toward the development side. Or it could underline changing attitudes as full-cycle development becomes more realistic. Regardless, teams will have to find clarity on roles to avoid some difficult conversations!

Cloud-Native Security

With the shift toward cloud-native strategies, security standards are rising to respond to new issues like misconfigurations. To address mounting application threats, increased automation could help shift security left—this and other reports clearly demonstrate a correlation between fully automated deployment pipelines and an increased security testing regimen.

As a result of cloud-native adoption, 58% of organizations have increased their security concerns since adopting cloud-native tools. Part of this involves moving to a zero-trust approach across all infrastructure, whether internal or external facing. The report revealed that 58% of organizations are increasingly concerned with misconfiguration, 52% are increasingly concerned with insecure APIs and 43% are increasingly concerned about known unpatched vulnerabilities, while 41% are concerned about secrets leaks. Hopefully, these benchmarks will help you see how your organization stacks up in comparison to others.

Snyk’s State of Cloud Native Application Security report questioned 600 experts in development, security and operations on cloud-native adoption and security practices. For more insights, view the study in its entirety here.

June 22, 2021
1Password Extends Encryption to Automate Secrets Management

1Password this week added a Secrets Automation platform to its offerings that enables IT organizations to more easily encrypt, manage and orchestrate credentials, application programming interface (API) tokens, keys and certificates.

In addition, the company revealed it has acquired SecretHub, a provider of a separate secrets management tool. Terms of the acquisition were not disclosed.

Carson Brown, senior product manager for 1Password, said the SecretHub team will now focus on the development of the 1Password Secrets Automation platform based on encryption technology previously created by 1Password.

Finally, 1Password this week also announced an alliance with GitHub under which it will, at some future date, enable DevOps teams to employ SecretsHub to encrypt secrets used to access the source code repository. 1Password already provides integrations with HashiCorp Vault, Terraform, Kubernetes and Ansible, in addition to client libraries written in Go, Node and Python.

The 1Password platform for managing and securing passwords is already in use by more than 80,000 businesses worldwide. The Secrets Automation platform now extends the reach of the company’s core encryption technology into the realm of DevSecOps best practices, said Brown.

It’s still early in the cycle of DevSecOps best practices adoption, but it’s clear an initial focus will be secrets management. In the wake of recent high-profile breaches involving software supply chains, there’s now increased scrutiny of secrets management. Cybercriminals have become more adept at scanning for secrets left exposed as plain text. The challenge organizations face is that developers, while building applications, tend to copy secrets for the sake of convenience, which they often forget to delete after an application is deployed in a production environment.

In response, many IT teams vacillate between overly restrictive and overly permissive access controls that are rarely implemented consistently.

Of course, it’s now also only a matter of time before auditors start citing all those unencrypted secrets as compliance violations, Brown noted. A set of DevSecOps best practices based on automatic encryption of secrets eliminates those compliance concerns in a way that is transparent to application developers, noted Brown.

It’s not immediately clear whether it will be DevOps teams that lead the charge to automate the encryption of secrets, or whether cybersecurity teams will take the lead. Regardless of which team assumes responsibility, the number of secrets that need to be tracked keeps expanding as the number of platforms employed increases. Manually keeping track of all those secrets is no longer feasible. Centralizing secrets management is the first step toward regaining control over an IT environment, said Brown.

Secrets management may not always be the first thing that comes to mind when organizations begin their DevSecOps journey. As it turns out, however, securing secrets may very well turn out to be the first order of business as senior business and IT leaders review their entire software supply chain. After all, once it becomes apparent just how dependent organizations are on software, they quickly realize how much of their intellectual property is one compromised credential away from being stolen.

April 14, 2021
Why Secrets Management is Critical to DevOps Pipeline Security
While valuable, secrets management also can be difficult for DevOps teams to employ. Here’s what you need to know

Business is all about speed. Companies want to innovate and deliver functionality faster to remain competitive. This explains the increasing popularity of DevOps as a go-to model for rapid application delivery. A recent Gartner report indicated that DevOps adoption is a growing alternative to traditional waterfall and agile development methodologies, but also pointed to key challenges with security and compliance.

For example, the report noted that DevOps practices encourage automation to achieve scale, but that security is traditionally manual, gate-driven and heavy on processes. In addition, the majority of developers have zero knowledge of secure coding, even many who are well-versed in agile and DevOps.

But there is another critical component of data security for DevOps teams: secrets management. In DevOps, “secrets” are typically authentication credentials used in applications and services. This may include usernames and passwords, either system-to-system or database; API tokens; SSH keys; encryption keys; and so on. Secrets are the keys to your data and, when handled casually, can open the doors to a serious data breach or intellectual property theft.

Secrets Management Presents Challenges

Developers often inadvertently leave behind sensitive pieces of code embedded in applications, files and code repositories. One real-world example is the way in which some developers use GitHub, a hugely popular source-code management system and the largest source-code repository in the world. Most developers seem to use GitHub casually, leading to increased risk of insider threats and hacking incidents. In one such incident in 2016, hackers penetrated Uber’s source code repository via GitHub, accessing intellectual property, AWS credentials and the personal data of more than 7 million Uber drivers and 50 million customers.

There are three main reasons why secrets management is challenging for DevOps teams:

1. Secret sprawl. Increased adoption of the cloud-native development model and the emergence of multi-cloud infrastructures has led to a proliferation and decentralization of secrets. If left to their own devices, these secrets can sprawl over time, leading to data breaches and privacy concerns.

2. A disparate set of tools. Many DevOps teams use different sets of tools for different phases of the development process and often struggle to have a centralized system that can integrate with all these tools and systems. Cloud-native secrets management tools are limited to the specific cloud provider and may not be effective in a multi-cloud scenario. An effective secrets management strategy should be able to integrate and work with every tool in the DevOps workflow and across cloud providers.

3. DevOps blind spots. Most enterprises are unaware of where their secrets are located, who has access to them or whether the secrets have been changed. This lack of understanding of the functioning of the DevOps pipeline often increases the risk of an internal data breach.

Today, most enterprises use containers to package and deploy applications. Gartner estimates that by 2022 nearly 75% of organizations will be running containerized applications in production. Oftentimes, in a container-driven environment, the groups responsible for developing applications are distinct from the groups in charge of deploying applications. Furthermore, database administrators by and large have (or should have) access to the keys, tokens and secrets.

In this setting, a major challenge is how to enable application development and deployment in the presence of this segregation without compromising secrets. If a developer has a piece of code inside a container that requires access to a secret key, then this secret value can be hard-coded into the code itself or included as part of the container image. But both of these options have detrimental security implications because they expose secret values to all those who can view the source code or deployment parameters, thereby violating the principle of least privilege.

Instead, secrets can be securely passed to the containerized code during actual runtime. Using a utility based on a container-orchestration system such as Kubernetes allows the secure transmission of secrets at runtime, meaning the secrets don’t need to be exposed while building or deploying the application. Rather, the utility can monitor the environment in real-time and inject secrets at runtime only when they are required.

3 Things to Look for

Regardless of the tools used in a development pipeline, protecting credentials and API keys from compromise is critical. Simply put, secrets must exist outside of the source code. When considering a secrets management solution, there are three things to look for:
- A flexible deployment model, whether on-premises, natively in the cloud or any hybrid/multi-cloud combination.
- A solution secured in a way that goes beyond a simple key-value store.
- The ability to connect to other applications and services through open standards such as OAuth, OpenID (SAML), LDAP, Trustworthy JWT and PKI.
A secrets management strategy that takes these factors into account maximizes an organization’s opportunities for success. And with the DevOps model as prevalent as it now is, it’s absolutely vital for teams to do so to significantly reduce the likelihood of breaches and ultimately improve business agility.
December 1, 2020
Safeguarding Secrets Within the Pipeline

How can an organization write source code to use secrets without directly referencing them and risking them being leaked to the public?

2020 closes out a decade full of countless code leaks, a topic we talk about quite often. In fact, just recently, Intel became the latest victim after 20GB worth of internal documents were leaked and uploaded to a public sharing site.

Intel finds itself in good company as it joins a long list of 2020 victims whose confidential source code has involuntarily gone public.

Many, if not most, of these victims share one common fatal flaw: They hard-coded secrets (logins, tokens, etc.), then published their source code to a trusted cloud computing service platform such as an online Git repository.

It would be naive to lay blame solely on the platform, however. It has long been a reliable rule of thumb that anything you put online—gated or not—is always at some level of risk. With this in mind, one should never, under any circumstances, include secrets in their source code. Stealing source code is damaging enough; the keys to your kingdom should not be included.

That’s easier said than done, of course. How do you write source code to use secrets without directly referencing them? In this blog, we’ll show you an overview of how you can utilize your continuous integration and delivery tools to inject secrets into your code at build/runtime without actually including them in the code itself, thus placing a veil between your code and your secrets.

Injecting Secrets

One method for publishing source code reliant on secrets without actually including the secrets in the code is to use placeholders within the source code which, at build time, are replaced with the actual values needed.

Microsoft Partner Technology Strategist Primoz Kocevar details such a method in his Medium post. The general idea is that secret variables are provided to a Docker image so that no secrets are found within the source code itself.

An environment variable, set up beforehand, stores the secret. This variable is defined in a Docker file, which is used to create an image. During the build phase, secrets are then passed via the Docker image through the variables specified in the source code. Once complete, the application and image can be deployed.

If you happen to run your images on Kubernetes, you can also leverage its own feature for securely storing secrets. This method allows you some additional options, such as using a Kubernetes environment variable or ConfigMap within a Kubernetes pod to store secrets for injection at build time.

By the way, this approach to secret injection is by no means exclusive to Docker or Kubernetes. Though we highlight them here as specific examples, the same method can be employed via virtually any similar solution you rely on for deploying pods or containers.

Injecting secrets via clever use of environment variables and configuration files in your microservices platform of choice is certainly a viable option, but we don’t think it’s the best option. Consider that this approach requires one image per each secret. This can get cumbersome fast if you have a development environment with one secret and a production environment with another secret, each requiring different images.

Leveraging Your CI/CD Tools

First, a quick primer for the uninitiated: Continuous integration (CI) and continuous delivery (CD) tools and practices are an essential part of the larger software supply chain. Continuous processes are those that enable developers to publish code changes easily and frequently without degrading the product’s reliability.

Continuous integration tools and practices establish uniform processes for creating, testing and building source code, integrating with any required external systems along the way. Continuous delivery covers the activities that follow, streamlining and automating the delivery of source code to infrastructure, especially when multiple multiple environments are involved.

Many CI/CD tools provide their own built-in solutions for securing sensitive data. You can leverage these features when storing secrets until it’s time to inject them during the build.

For example, Jenkins provides its own credentials store and its own automatically encrypted Secret field type. (A word of caution: This alone doesn’t prevent a user from accidentally printing a secret in the console log.)

Another CI tool, CircleCI, offers a number of options for protecting secrets, such as secret masking to prevent secrets from appearing in terminal output and contexts, which limit access to certain variables to only specific user groups.

There are, of course, as many additional options as there are tools available.

We can leverage such features to enhance our previous example: secrets stored in your CI/CD tool can then be injected into our Docker run command (or your platform’s equivalent command) during deployment. In other words, your CI/CD tool of choice injects secrets into the image, which in turn injects the secrets into the corresponding source code’s variables. This method ensures our secrets never appear in the actual source code.

A word of caution: Even utilizing this approach, your secrets are still only as safe as the tool you use to secure them. The Bleeping Computer article we linked earlier in this post points out how weakly configured tools themselves pose a threat: “The leaks have been collected by Tillie Kottmann, a developer and reverse engineer, from various sources and from their own hunting for misconfigured DevOps tools that offer access to source code” (emphasis ours).

It’s important to stress the universal rule that there is no such thing as perfect security. Consider that encrypting your secrets within one of these tools offers little protection if access to those tools is not sufficiently restrictive. We recommend cautiously limiting who is granted what level of user access to each tool, applying the least privilege principle.

Protection Post-Deployment

Now that we have built and deployed our secret-free code, what’s next?

If we were to stop here, we would have overlooked the fact that any developer who has access to our containers can see our secrets. The downside to utilizing a microservice infrastructure is that there are many opportunities for security gaps to appear between all the moving parts, from the moment the first line of code is written to when it all goes live, which can easily go unnoticed.

Containers and pods need to communicate with each other and other endpoints, thus an attacker can leverage this by compromising one container or pod then moving laterally to others, rendering any secrets stored within ripe for the taking. The bigger your distributed infrastructure, the more difficult this is to prevent, because of the inevitable complexity of security policies required to do so.

Thankfully, many tools offer options such as built-in network/firewall policies between containers/pods and endpoints, an option to encrypt data at rest or a mechanism for enforcing the least privilege principle, such as via restricted security groups.

Some tools may also allow you to configure a Key Management Service (KMS) provider to encrypt your secrets. A KMS provider employs an envelope encryption scheme for data stored within etcd. The added benefit to this approach is that the decryption key is stored remotely on the KMS as opposed to on the same host.

Conclusion

Regardless of what tools you have in play along your development pipeline, secrets must always exist outside of the source code and should be (securely) injected into the code as it passes through the pipeline.

Wherever secrets ultimately live, be it in a remote storage location, a container or pod or within a CI/CD tool, they must always be encrypted and never stored in plaintext.

Finally, all the obfuscation and encryption in the world will not protect secrets from a lackadaisical user access policy. The least privilege principle should always be in play and user security groups should be strictly managed to control who has access to the systems responsible for securing your secrets.

All that said, mistakes still happen, and none of the tools discussed here perfectly ensure that no developer will nonetheless accidentally hard-code secrets into your source code. It is with that in mind that we advise adding another layer of security by automatically monitoring your source code repositories for secrets.

September 15, 2020
HashiCorp Strengthens Vault Secrets Management

HashiCorp has released an update to its HashiCorp Vault platform for managing secrets and encrypting data that now includes an internal storage option that eliminates the need to manage a separate storage backend.

Version 1.4 of Vault also adds an OpenLDAP Secrets Engine, which automates the management of static entities within OpenLDAP directories, and the ability to authenticate users and applications using the Kerberos authentication protocol.

HashiCorp has also updated the enterprise edition of Vault to add support for a Transform Secrets Engine that can be used to secure data that reside outside of Vault, such as credit card numbers, by applying tokenization techniques to mask data, as well as a Vault Helm Chart tool that makes it easier to deploy Vault on Kubernetes clusters to the enterprise edition of the tool.

Finally, the enterprise edition of Vault adds improved disaster recovery capabilities and support for NetApp enterprise key management to more easily manage encrypted data.

Chris Kent, director of product marketing for HashiCorp, said thanks to the increased focus on data privacy there is a greater appreciation for the need to maintain application secrets in a platform that resides outside of the core application. Regulations such as the General Data Protection Rule (GDPR) enacted by The European Union and the California Consumer Privacy Act (CCPA) penalize companies for losing control over credentials and data. As a result, more organizations are looking for a way to better manage application secrets, noted Kent.

At the same time, cybercriminals are becoming more adept at discovering the credentials of end users that developers sometimes inadvertently leave accessible within their applications, he noted. Vault provides a way to manage and maintain application secrets outside of the application in a way that serves to significantly enhance overall application security.

As developers and cybersecurity teams collaborate more closely as part of any effort to embrace best DevSecOps processes, it’s becoming apparent that a lot more attention must be paid to how application secrets are managed. In fact, arguably one of the biggest impediments to improving the overall security of applications is the way developers have historically managed application secrets. It’s not uncommon for cybercriminals to compromise an application in a way that gives them access to thousands of individual user names and their associated passwords, many of which unfortunately are reused both inside and outside of the extended enterprise. The challenge at this point is not so much making the tools available to manage application secrets as much as it is changing the processes and culture within organizations that are just now starting to embrace DevSecOps.

In general, as organizations become more proficient at managing application secrets the more secure application environments should become. Of course, secrets management is only one aspect of application security. However, the absence of an effective strategy for managing application secrets can prove catastrophic. By the time most IT organizations realize their application secrets have been compromised, it’s usually far too late to undo the damage.

— Mike Vizard

April 8, 2020
DevOps Chats: Plugins Extend Spinnaker for the Enterprise

Spinnaker Summit 2019 Preview: Secrets management is vital to any vibrant DevOps and GitOps toolchain. As part of its open source maturation and thanks to the contribution of our DevOps Chat guest, Spinnaker now has a secure secrets management capability enabling GitOps, taking non-application secrets out of plaintext in hal config files.

Cameron Motevasselani, software engineer at Armory, is giving his talk “Spinnaker Plugins: Extending Spinnaker for the Enterprise” on Sunday, Nov. 17, at 10:45 AM PT.

In addition to his contributions to Spinnaker’s new secrets management facility, Cameron shares the work in progress to create an extensible Spinnaker plugin system. The current work is an early MVP to be followed by the implementation of PF4J and concrete extension points as part of Spinnaker stages.

As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

Transcript

Mitch Ashley: Hi, everyone. This is Mitch Ashley with devopsy.kinsta.cloud and you’re listening to another DevOps Chat podcast. Today, I’m joined by Cameron Motevasselani—Motevasselani, thank you [Laughter]—a software engineer. He’s also a speaker at Armory. He has a couple of talks that are at the upcoming Spinnaker Summit 2019 in San Diego. First one—I guess this first one is on the plugin system and the second one is on secrets management.

So, Cameron—welcome to DevOps Chat.

Cameron Motevasselani: Hi. Thanks for having me.

Ashley: Glad to have you on. Well, tell us a little bit about yourself. Introduce yourself and tell us a little bit about what you do at Armory and what Armory does.

Motevasselani: Oh, totally. Yeah. So, I’m Cameron Motevasselani. Thanks for the quick introduction. Let’s see, I’ve been at Armory for about a year now. I joined and hopped feet first and started working on this secrets management project. So, that was pretty cool. And then, now, recently, I’ve been working on this plug-ins system. Yeah, that, I think, about sums it up.

Ashley: Great. What does Armory do? Yeah.

Motevasselani: Yeah, Armory, we basically have an enterprise offering for Spinnaker. So, we have some extensions that we’ve written on top of Spinnaker, so we don’t fork and have to maintain all that, we just—we extend Spinnaker and provide some extra features that way.

Ashley: Very nice. Now, the way you phrased that, you’ve been working on secret management and the plugin. Does that mean you’re actually contributing, writing code and contributing it to the open source, or are you primarily an implementer of what’s already been built?

Motevasselani: Oh, no, that’s a great question. Yeah, for the secrets management as well as plugins, we both, we’re contributing directly to open source for both of those projects.

Ashley: Awesome. Excellent. Well, let’s start with secrets management. I mean, I think we all kinda have the idea of where we’re gonna keep our keys, our passwords, certificates, all that good kinda stuff. We don’t want that heading into the source code management system or somewhere in the cloud or in someone’s software. Tell us a little bit about the secrets management part of Spinnaker and what you’ve been doing, what you’re gonna talk about.

Motevasselani: Yeah, totally. So, secrets management really came up as we started talking with customers about their Spinnaker setups and how they deploy and manage the configurations for Spinnaker.

As you know, Spinnaker has the keys to the kingdom, essentially. It has your cluster keys and your credentials in order to deploy cloud resources that cost you money, right? And those are running—it runs the code that your organization arrives on. You can’t have that in source control. That is not, not good to have.

Ashley: No-no—a big no-no, though a common mistake. [Laughter]

Motevasselani: That’s right. Right. [Laughter] I think GitHub even yells at you now, if you have keys—

Ashley: Oh, does it? [Laughter] Nice.

Motevasselani: – in there. So, they’ve been adding nice little security features here and there. People would like to move towards this idea of GitOps so that you can manage your configuration via pull requests. It’s a great idea, and a lot of people can’t do that, because the keys are quite literally in the Spinnaker configurations.

So, we pull those keys out and are essentially referring to them with our secrets management solution. So, you can use different backing stores such as S3 buckets or Vault and store your secrets there and then refer to them in the Spinnaker configuration that way.

Ashley: Mm-hmm. Excellent, great. So, in your talk, you’re gonna be walking through what you’ve been doing, the functionality of it, how to implement it via GitOps?

Motevasselani: Yeah, so—that’s right. The functionality, how it differs from, there are other solutions coming into open source now that tackle a slightly different use case. So, discussing the differences there as well as the future, essentially, of secrets management.

Ashley: Okay. Very good. Great. Anything else to tell us about that before we move into the plug-in system?

Motevasselani: Yeah. You know, secrets management, it is about the secrets for Spinnaker in particular. It’s not actually tackling the application secrets themselves.

Ashley: Oh, okay.

Motevasselani: So, I just wanted to make that clarification. And that’s at the current time. It’s possible that we might have a story for that in the future, but the talk that I’ll be giving is around Spinnaker secrets, not application secrets.

Ashley: Okay. So, this is just what’s required to run, operate, manage the Spinnaker environment, not your application secrets?

Motevasselani: That’s correct.

Ashley: Okay, great. Well, that’s an important clarification. Great. Well, let’s talk about the new plug-in system. What is new about it, other than being new code?

Motevasselani: Yeah. So, the plug-in system is, we’ve been developing it as a very—it’s been a very alpha, MVP type feature. So, right now in open source, we do have a method for users to create custom stages in Orca. There is quite a bit of knowledge and expertise that you have to have in order to do that, so I don’t quite recommend doing it right now. But we’re building on this plug-in system and we’ll be testing it out with users hopefully in the near future.

So, right now, there’s currently no UI, but we’re working on a UI and then that should be getting open sourced as well in the future.

Ashley: Now, that sounds like kind of a tough assignment. How do you create a plug-in system that anybody can do anything they want with, because that’s probably one of the requirements is, you want a lot of flexibility. How do you go about architecting and designing that so what you’re building can be extensible and you’re not gonna have to re-do it in a year?

Motevasselani: Yeah, that’s such a great question. Because we are going with this MVP model of just getting something working and then getting feedback on it to iterate, we actually are already planning on deprecating this first, let’s say, method of loading plugins into Spinnaker. We’ve already been doing some internal revs on it and we’ve been working with the community, Google and Netflix in particular, on this project. So, we are going to be moving away from our current implementation into another full-fledged plug-in framework, and we’ve chosen PF4J for that.

Ashley: Okay. Now, I’m not familiar with that framework. Can you tell us a little bit about that?

Motevasselani: Yeah, so the way we currently did it—and hopefully, this isn’t giving too much of my talk away [Laughter]—but the way we’re doing it right now is essentially shoving JARs into a ClassLoader for Orca. Well, via Quark. So, these are different Spinnaker services and libraries that make up Spinnaker.

Ashley: Mm-hmm.

Motevasselani: Right now, it’s a pretty dumb way of doing it, just adding JARs onto a Classpath in order to load it onto the Spinnaker spring application context.

Ashley: Makes sense. Pretty easy way to do it.

Motevasselani: Yeah, exactly. But like you’re saying, there’s no—it’s very open ended right now. We are going to, as we move to PF4J, define concrete extension points within Spinnaker so that there’s actually not a sprawl of extension points, there’s clearly defined extension points throughout the code.

Ashley: Mm-hmm.

Motevasselani: That will help us iterate on this feature and make sure it’s a good solution as we continue developing it.

Ashley: Okay, and this is the PF4J that’s on GitHub? I think it’s an Apache license, right?

Motevasselani: Yeah, that’s correct. I believe it’s Apache. That’s correct. And so, moving into PF4J will give us a lot of really nice features that our current implementation doesn’t have, such as using a different ClassLoader for every plug-in so that it can have its own dependencies, et cetera.

Ashley: That makes a lot of sense. Are there any unique challenges for implementing a plug-in system in an environment like Spinnaker? You know, this is software to create software, software to deploy software. You know, it’s not an application in and of itself, it’s something that you’re gonna make a key part of your kinda tool chain, your workflow pipeline. I wonder if there’s any unique challenges that come about in trying to build this.

Motevasselani: Yeah, it’s interesting, because we have been looking at, you know, as you develop something new, you look at prior art in order to determine how to build what you’re building, I guess. [Laughter]

Ashley: Mm-hmm.

Motevasselani: For lack of a better word.

Ashley: Technical speak, I know. [Laughter]

Motevasselani: That’s correct. [Laughter] So, looking at something like Jenkins or Eclipse that both have plug-in systems, same with WordPress, those are all kind of monoliths, right?

Ashley: Mm-hmm.

Motevasselani: Spinnaker is definitely many different microservices composed into one application. Essentially, we’re just kind of turning it into a platform, kind of, by making it extensible and pluggable.

But that is, to answer your question, a major challenge is, you have all these different microservices. You know, depending on how you deploy and update your Spinnaker installation, they might be on different—I don’t wanna say different versions, but if you’re deploying master all the time, you might have a different set of microservices in production compared to if you’re all running the same release version.

So, deploying plug-ins into that environment is not an easy task and how you get the versioning done is important, as well as rollbacks.

Ashley: You kinda have a recursive problem here, right? Because you have to version control the plug-in manager, which then version controls of what plug-ins can work with it, c?

Motevasselani: Yeah, exactly. Plugins might rely on some functionality that is in one version of a microservice that’s not in another version, and if you need to do a rollback, how does that work, et cetera. That’s correct.

Ashley: Well, you kinda took my breath away for a moment. You scared me when you said, you mentioned the WordPress plug-in system.

Motevasselani: [Laughter]

Ashley: Please, don’t do that! [Laughter]

Motevasselani: I do think, though, that WordPress is an example of a project that has done very well—

Ashley: They have.

Motevasselani:—because it has such as good ecosystem of plugins.

Ashley: It’s come a long way. It has come a long way. It’s much more, I don’t know, not self-sustaining, but kinda auto updating much better than it used to be. [Laughter]

Motevasselani: Right, right. And we do hope to take learnings, right, from past projects, let’s say, as well. [Laughter]

Ashley: Absolutely. I don’t mean to dis WordPress, you know? You gotta every once in a while. Well, tell us a little bit about more about your talk. What are you gonna cover, then? Is this more how to use it, or you’re gonna lay out kinda where you’re going in this next version and you’re looking for feedback? What are you trying to accomplish or get out of it? What would you like to have happen in the talk?

Motevasselani: Yeah, you know, that’s a really good question. I think there’s a lot of stuff to cover, all the way from our journey running and creating this project, working with the community and those efforts, to showing users, “Hey, this is actually a nice and easy way of modifying or extending Spinnaker’s functionality.”

The idea here is to—there’s kind of a few ideas here with this project, because we really want to lower the barrier of entry for users, particularly in enterprises, to be able to create extensions for Spinnaker. The method we chose to do this at first was a way that most users interact with Spinnaker, and that’s creating pipelines and defining stages.

So, let’s say an enterprise has some custom infrastructure at their business running their data centers, running their cloud. If they want to, let’s say, interact with that in a native—in a Spinnaker native way, you would want to have a stage that their users can just choose. So, rather than having to learn how Spinnaker works on a very deep level, you can implement this nice, simple interface and get it going.

So, at the end of the talk, it would be great for the audience to have an understanding of how to create and implement their plugin on their own.

Ashley: Oh, excellent. Well, then, hopefully, you can, you’re creating an audience and a community to go out and try your new plug-in system and the variations of it.

Now, you talked about using plug-ins as extensions to Spinnaker. Do you also see this plug-in system being kind of an integration interface to third party tools, maybe other open source, other commercial tools, or is it more things that just extend the functionality of Spinnaker?

Motevasselani: Oh, that’s a really good question. So, there is a push to move Spinnaker to be a lean core and an expansive ecosystem, I believe, is the term being used, and this goes directly towards that goal. There’s certain functionality in Spinnaker, the core, Spinnaker core, that doesn’t quite make sense.

For instance, there’s some stages that are specific to companies. Don’t wanna call Gremlin out, I think they do great work. There’s a Gremlin stage in Spinnaker that should probably be a plug-in. So, hey, Gremlin guys, hit me up. [Laughter] But not quite yet, maybe for the end plug-in system.

Ashley: It’s that evolution of software, right?

Motevasselani: Exactly.

Ashley: I mean, that’s how you get to this point where you need to do that, you need to spin things out, so it’s part of its growth.

Motevasselani: That’s right, that’s right. Yes. It would be really cool if you could have a Spinnaker installation where you can essentially pick and choose which cloud providers you want to have installed and running. If you only—you know, if you interact with, let’s say, both Google and Amazon’s cloud, maybe you don’t need Oracle’s cloud software to be on your cloud driver. Or maybe you want to have a cloud driver in each of those data centers—sorry, in each of those cloud accounts that only has the code necessary to deploy to that cloud environment.

Ashley: Mm-hmm.

Motevasselani: So, then you have a smaller attack surface, just leaner and smaller binaries, et cetera, et cetera.

Ashley: Great. Well, I think, unless there’s anything you wanna add, I think you’ve given a really comprehensive look to kinda what that talk, both talks are gonna be about.

Motevasselani: Yeah, that’s—you know, that’s hopefully, just getting the community excited and get some ideas flowing, I think, would be great with those talks.

Ashley: Well, and I also wanted to just throw in my little thanks just to appreciate what you do and all the other contributors to the projects in software. And I always encourage people who are just using open source, and there are folks at the Spinnaker Summit who are implementing it, not writing code for Spinnaker, but talking about it at a summit like this is another way to contribute to the community. So, great to talk with you who are actually writing code as well.

Motevasselani: Yeah, thanks. Nice talking with you, too, Mitch.

Ashley: Great. Well, it’s been my pleasure. You’ve listened to another DevOps Chat podcast. I’d like to thank today’s guest, it’s Cameron Motevasselani, software engineer at Armory, and he is speaking at Spinnaker Summit 2019. The dates for the conference are November 15th through 19th in San Diego, and Cameron has two talks. His talk on the plugins is Saturday, November 16th, at 2:30 p.m. And then he’s also talking bright and early about secrets management—boy, you’re really gonna wake ‘em up here—Sunday morning at 9 a.m., Cameron, so. [Laughter]

Motevasselani: [Laughter]

Ashley: So, you know, maybe you serve mimosas at the morning talk. [Laughter] So, it’s been my pleasure to have Cameron on, and thank you to all of our listeners today for joining us. This is Mitch Ashley with devopsy.kinsta.cloud. You’ve listened to another DevOps Chat podcast. Be careful out there.

— Mitchell Ashley

November 13, 2019
Applying Secrets Management to DevOps

DevOps has changed the way enterprises operate, allowing for quick and efficient product development. In order for companies to keep up with market competition, many companies that are not structurally prepared to implement DevOps will still put together a team just to meet demand, which in turn creates vulnerabilities and gaps within their security posture.

Today, only 46% of IT professionals are confronting security risks during the initial phases of development. With security as an afterthought, DevOps teams may overlook even the most basic security practices. What’s more, in an environment that relies heavily on code, we’ve seen time and time again careless developers leaking confidential information through APIs or cryptographic keys on sites such as GitHub. As a mecca for code, it’s the primary place cybercriminals turn to gain access to the credentials developers have stored in its platform and forgotten to remove before production.

While GitHub breaches continue to put a spotlight on DevOps security, most organizations are still missing the big picture. The reality is security hygiene remains critical, and DevOps can put their organizations at risk by neglecting basic security practices such as monitoring access and embedded secrets, such as passwords, keys and APIs.

Where Should DevOps Teams Begin?

The solution is simple: separate highly valuable secrets from the developer. Luckily, the key to streamlining secrets doesn’t have to be time-consuming or costly. In fact, secrets management is, at its core, an extension of privileged access management (PAM), which many enterprises already have in place.

As an essential approach to safeguarding an organization against bad actors, one of the fastest ways to mitigate the risk of DevOps-related data is to remediate weak security practices related to PAM.

The following are some of the most important security protocols DevOps team should apply to their delivery pipelines:

Inventorying Privileged Accounts and Access

Large enterprises running networks with thousands of servers and network devices often lack an accurate inventory of assets. Most organizations create a security gap by not understanding the credentials within their systems due to inadequate deprovisioning processes. A DevOps environment’s use of automation and scripts makes it even more complicated to manage privileged accounts and access.

To ensure DevOps practices are secure beyond the initial provisioning phase, security professionals must understand where automation is stored and what embedded credentials are stored within that automation. By understanding the entire embedded privileged credential process, adamantly keeping track of each DevOps team member’s access and leveraging automation, enterprises can instantly update a user’s access when their role changes to ensure they have the right access to do their job and nothing more.

Secrets Management Integration

A significant challenge for DevOps teams in managing privileged credentials is finding a solution that can be seamlessly integrated into existing development toolsets.

Many DevOps teams rely on several different tools for different phases of the development process, and simply can’t afford to use secrets management solutions and processes that don’t integrate with and address all of them. For example, if an enterprise has invested in AWS and has an Azure-specific secrets management solution, tedious steps could be added to the process. In an environment that’s focused on rapid development, developers are likely to find ways to work around these inconveniences which can put the enterprise at risk.

To ensure an effective secrets management strategy is put in place, enterprises need to understand that it must work with every tool in the DevOps workflow. By integrating security systems with tools developers know and use, enterprises can ensure no shortcuts are putting its sensitive information at risk.

Understanding the DevOps Pipeline

Traditionally, enterprises think the only way to obtain a secret is to log into a server, but in DevOps that’s not the case. In DevOps, when a user with privilege inputs a piece of code, it goes through a pipeline and many enterprises are oblivious to what happens within that pipeline. This blind spot could pose serious cyber risk to the organization. For example, if three developers are working on a project and they check in code that requires privileged access, such as copying credit card information, a developer could inadvertently interject dangerous code into that project and the security team is none the wiser.

For an environment that’s focused on individualized actions, applying secrets management and thinking twice about who has access to build within this pipeline environment is critical. By not understanding how the pipeline functions and not knowing who has access to specific data, enterprises increase their risk of an internal breach or data leak.

The reality is managing DevOps security has nothing to do with the actual DevOps process. By integrating basic privileged management protocols, enterprises can ensure security is at the forefront–and not the backend–of the development process.

— Tyler Reese

October 15, 2019