Tag: devops security

  • Security Debt: Speed vs. Common Sense

    Security Debt: Speed vs. Common Sense

    A couple years ago, we had some spectacular security events that involved DevOps and Kubernetes, where the managing team simply redeployed containers whenever one crashed. It turned out that many organizations were doing the same thing, and, what’s worse, they were not talking about it because they knew it was not a long-term solution to crashing issues.

    At the time, I thought these high-profile lessons would teach people to actually fix code issues, even in high-throughput Agile-meets-DevOps shops. I admit I also laughed, yet again, at The Phoenix Project’s expense, thinking, “Yeah, but the CISO isn’t holding things up by demanding that get fixed!” I wish that would ever get old, but it won’t.

    It seems I was wrong. Or at least partially so. Some—perhaps many—shops seem to have missed the message that these issues sent. Anecdotally, we heard this was still happening, so we started asking questions. Yep. There are shops out there (since we started with ‘anecdotally,’ I won’t pretend to put numbers around it) that still take an “If it isn’t causing disruption in core systems, don’t fix it” mentality.

    Ouch. We’ll be revisiting some of those in the future, I’m sure—when it’s listed as a vulnerability or breach somewhere.

    I’m not a security professional. Because the vast majority of enterprises do not have dedicated security and even more don’t have anything close to enough security staff, I have done security as a part of almost every job I’ve taken—because it was what needed doing the most. Many, if not most of you, are in the same boat.

    That said, I can tell when you are blatantly asking for a security incident. And running code known to be faulty or misconfigured to the point that it repeatedly crashes is deliberately asking for a security incident. “But it isn’t exposed to the Internet!” is only a valid rebuttal until an attacker gets inside the perimeter and uses your fault for something like privilege escalation. Then no one cares if it’s connected to the internet or not; they want to know how the attacker did it. And that’s on you.

    So for those organizations in which speed has overridden common sense, let’s be very clear and to the point.

    It is a best practice not to run crash-level faulty software.

    Since you must not have one, write a policy that explains this and lays out a process and/or timeline to fix these problems. Then drive it home. Make certain it is known that this policy will be enforced. For that huge majority of my readers that don’t have the level of authority that “create and promulgate a policy” implies, convince those who do. Point out the issues to them in writing. “Notified in writing of faulty practices that could bite the company in the rear,” tends to get even the most unfriendly management to click into CYA mode and do something about it.

    When we all watched those spectacular failures back in the day, I assumed that this was all about growing pains; you can do so much more with Agile/DevOps/cloud and/or Kubernetes that this type of thing would crop up and, eventually, get addressed. I still believe that. You all are still testing the boundaries and pushing the limits and making all of IT look good. It’s just going to hurt getting there sometimes. Fix the apps, make the policy, keep kicking butt and know you are killing it (as an industry. If you aren’t, stop laughing at my blanket compliment and fix yourself).

  • Securing APIs at the Speed of DevOps

    Securing APIs at the Speed of DevOps

    In the 2021 State of DevOps Report, 83% of IT decision-makers told Puppet that their organizations were in the process of implementing DevOps practices to improve the quality of their software, the speed of their delivery and the security of their systems. Those DevOps organizations varied in their stages of evolution, however. For example, respondents in the middle stages of their journey reported that culture blockers such as unclear responsibilities and insufficient feedback loops were making it difficult for them to continue their DevOps journeys and join the ranks of high-evolution organizations. Such obstacles help to explain why 97% of high-evolution organizations had already applied automation to their competitive tasks compared to just two-thirds of mid-level and a quarter of low-evolution teams.

    The Importance of APIs to DevOps

    The prevalence of automation ties into the importance of application programming interfaces (APIs) for any organization’s DevOps journey. In the words of IBM, an API “enables companies to open up their applications’ data and functionality to external third-party developers, business partners and internal departments with their companies.” APIs, therefore, enable the services and products of different companies to communicate with one another in a way that streamlines and/or augments functionality for the user.

    That said, users aren’t the only ones who benefit from APIs. DevOps teams do, too. Many of these team members deal with increasingly automated processes as part of their daily tasks, so they need APIs that can help them to deploy and configure their infrastructure. APIs can also help DevOps teams gain detailed information about an app during runtime. DevOps staff can use these insights to proactively address issues before they lead to disruption.

    Security Obstacles of APIs and DevOps

    Despite their benefits, APIs and DevOps aren’t without their security challenges. Contemporary Computer Services, Inc. (CCSI) identified several security hurdles that organizations may encounter along their DevOps journeys. These include the following:

    A fast development process: DevOps introduces a faster speed of software delivery. In their efforts to keep up with this demand, developers might make coding mistakes that leave applications vulnerable to attack. Absent a timely security review, those vulnerable products and services might make it into production, granting attackers an opportunity to exploit the flaws and to try to access affected customers’ digital assets.

    Poor collaboration: No DevOps journey will succeed without collaboration between development and operations. When there’s a lack of joint processes, these teams may stick to their silos. This way of working may leave security gaps if development and operations aren’t communicating about the management of their credentials, tokens, SSH keys and other secrets, for example.

    Those security challenges can then spill over into APIs. The speed of development along with poor team communications may give rise to multiple versions of the same API. This redundancy contributes to a sense of what’s known as API sprawl, a phenomenon where the number and reach of organizations’ APIs grow exponentially. API sprawl comes with its own obstacles including maintenance difficulties and broken clients, issues that provide attackers with more opportunities for infiltrating the organization’s systems. This reality helps to explain why 91% of enterprises experienced an API security incident in 2020 and why malicious API traffic increased by 300% in the middle of 2021.

    How Can Organizations Address These Security Challenges?

    For an answer to this question, we look back to what highly evolved DevOps organizations are doing. Most organizations are focusing on DevSecOps. In Puppet’s report, 51% of respondents said that they are integrating security into their DevOps requirements, with even more saying the same about design (61%) and build (53%). An additional 52% of survey participants indicated that they’re bringing security into their testing. By contrast, mid- and low-evolution organizations mainly engaged in security when they were in the process of preparing for an audit of their production systems or when they learned of an issue affecting production.

    Beyond those areas of focus, organizations need to emphasize runtime protection. Salt Security agrees with this statement, noting how organizations need to “security test your APIs, but know that you will also need runtime protection to catch changes that don’t go through standard build process and abuses that testing tools aren’t designed to find.”

    Finally, organizations need to treat DevSecOps as the cultural change that it is. They can do this by assigning security responsibility to one person from their DevOps team. Doing so will help to ensure that teams won’t overlook security during their jobs. Simultaneously, organizations can conduct security awareness training for their developers and regularly invite both development and operations to share their stories, priorities and successes with one another. This will help to foster collaboration, thereby minimizing instances of API redundancies and API sprawl across their systems.

  • DevOps Chat: Maximizing the Benefits of DevSecOps

    DevOps Chat: Maximizing the Benefits of DevSecOps

    When discussing security in DevOps, we often focus on the security tools instead of the DevSecOps process itself. In this DevOps Chat, ZeroNorth CEO John Worrall takes us to the root of “why” DevSecOps, focusing on the business benefit, gain and measurement of what we seek to accomplish through DevSecOps. John advocates we concentrate on the process and data enabling us to assess risk, prioritize the most beneficial security work and for decision making to creating business value.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Transcript

    Mitch Ashley: I have the pleasure of being joined by John Worrell, who is CEO of ZeroNorth. Welcome, John, good to be talking with you today.

    John Worrell: Thank you, Mitch. Good to be here as well.

    Ashley: And we’re talking about security and DevOps, DevSecOps. But before we get to that topic, would you tell us a little bit about yourself, let our audience know a little bit about ZeroNorth, too?

    Worrell: Sure, I’d be happy to, Mitch. I’m John Worrell, CEO of ZeroNorth. We’re in the DevSecOps space, maybe with a different approach that, a lot of times people focus on tooling. We believe that you need to kinda build a process out first for DevSecOps to actually have business value, and I’d like to maybe talk through that with you, Mitch.

    Ashley: That’d be excellent. I’m all about the business value, too. [Laughter] So, let’s talk about that. So, usually, when you say DevSecOps, it’s about shift left, how you need to get to the security people working with the software folks, how do you help developers write more secure code? You get to the tooling conversation.

    So, I think you’re talking about—well, let’s stop for a minute and ask why are we doing all that stuff? Yes, let’s create better app security, but why do we need to make this a process that produces, maybe, some business value, measurable results, those kinda things?

    Worrell: Yeah, you bet. That’s exactly right. We’re all for the tooling, we’re all for building the bridges between security and the Dev teams. But you have to have an objective in mind, and you have to have some help along the way. You have to have some data along the way that’s gonna actually get you there to pull all that together.

    One of the best quotes I ever heard that got applied to this was channeling your inner Deming, which is, you know, if you can’t describe what you’re doing as a process, you don’t know what you’re doing, right, and there is a real lot of truth to that. If you step back and look at AppSec over the years, it’s been very tool-centric, very sporadic, anything but a process. When you look at DevOps, it’s a process, it’s repeatable, it’s scalable. It’s kind of a self-learning or positive flow of information with telemetry so you can get better and better at it over time just by learning from your past experiences.

    And when you think about integrating security into Dev to drive DevSecOps, you have to think about it the same way, and I think that’s where it all starts. This is a process, this is not a tool selection, this is not just a scan or an integration, it’s that process that you need to build first, and then you can start building out your tooling, your technology underneath that.

    Ashley: You have a really good point, and if I’m being over general here and generalizing this, as security professionals, we kind of think of, “Let’s buy boxes, let’s buy software, let’s put functions into the network, into software, and not into the software development process that will solve our security problem.”

    But security is not just people attacking you, it’s also how you do your work and making sure you create a secure environment, you operate securely, write secure code. So, I think you’re spot on about the process and how you can do that consistently, referring back to Deming, right?

    Worrell: Yep.

    Ashley: You can’t manage what you can’t measure, your adaptation of that phrase. So, talk to us a little bit about how do you make that real? When you talk with people, how do you help them sort of elevate the conversation to that and then get on the right track?

    Worrell: So, I think it starts out with that process conversation and whichever analogy you’d like to us for that. Think about it this way—the process, in our view, works like this. You’re gonna set some kind of a governance standard for your organization. You can’t have each Dev team setting their own or each business unit setting their own, you’re gonna wanna try to get global visibility so you can actually have effective global risk management. And we’ll start that as a kinda catch-all for the business objective—but in a sense, that really captures what we’re trying to do with DevSecOps. You need global risk management across your organization. That means you have to have some visibility into it, but you have to have a way to impact your current state to make it better, and you need a process to drive that in a continuous way.

    It starts with setting some governance standards and saying, “Okay, what is our ideal objective for our top tier applications? What is our ideal objective or governance model we wanna put around our mid-tier?” or whatever the application categorization might be. Then you wanna make sure you’ve got a continuous process to build the data that’s gonna inform you, whether or not you’re actually meeting those objectives. You have to look at performance reporting so you can measure how you’re doing against that. 

    And then you have to be able to action that intelligence to say, yeah, that data is helpful to me for a number of reasons. Number one, I get visualization of risk, I can get prioritized remediation if I do this right so my developers can make intelligent business decisions about which vulnerabilities get fixed now before we progress the build and through the build. The third piece of that is, which teams need what kind of training on security coding, good security coding practices? Instead of trying to treat everybody the same, use this telemetry to say, “You know what? Team A is struggling with cross-site scripting. Let’s go do a lunch and learn with those guys and watch the immediate impact we’re gonna get from that” and then let the process continue and go all through. There’s a lot of technology in there, there’s a lot of things in there, but it starts with the process.

    The next piece that’s really critical is the data that comes out of that, right?

    Ashley: I was just gonna ask you about the data part of it. [Laughter] 

    Worrell: [Laughter] That data drives everything. It drives so much. You know, we had talked about this idea that you’ve got different tooling—well, sure you do. You want the best tool that you can get, you want best of breed. But trying to pull those tools or the data from those tools into some meaningful, actionable intelligence is a key step in the process, right? You wanna take different data points and put them together so you get a complete view of what the risk posture is for that particular target or entity or application you’re working with.

    We talk about the analogy of the blind man and the elephant. You have four blind men all experiencing an elephant from different places, and they all can come up with a different understanding of what they’re really touching and trying to investigate. And it’s a perfect analogy for a multi-tool environment when you’re scanning across the pipeline that, if you just look at one experience or one tool, you’re not gonna get a complete view of what’s going on.

    Imagine the difference of being able to just do static scanning and realize that you’ve got all these highs and criticals, but not compare that with your dynamic. So, a lot of those highs and criticals may not be exposed in my environment, right? I wanna pull those together.

    I’m trying to understand, number one, where the risk is, but I’m also trying to optimize my developer productivity and making sure they’re not fixing a bunch of stuff that doesn’t need to get fixed, if I’ve got a compensated control in place or it’s just not exposed already. That’s not high on my list. I want to really focus in on what’s valuable.

    So, that data can be used for making good business decisions on risk. It can be used for prioritizing remediation. That data can be used for, you know, call it the wall of fame, wall of shame. It’s how do you really measure and incent and reward your business units for meeting their risk management targets, right? Again, how do you help those that are lagging behind, how do you help them get better?

    And that, to me, built into this process, is where you get real business value. You’re getting the faster delivery of software, you’re getting the better developer productivity out of this, you’re getting that risk management layer and visibility you need, but you’ve got it as a process, and you can measure week to week, month to month, quarter to quarter, how much better you’re getting at it. That, to me, is a real business solution.

    Ashley: You know, you described a lot, so let’s unpack some of those things. The third piece of it usually is, you know, you talk about people process and technology people. What are the kinds of roles of people who are gonna be using that kind of information, that kind of data? Because we’re often talking about getting security information into the developer’s hands so they can fix things, but you’re also talking about, “Well, what should we fix? Let’s not waste our time on things that don’t matter, right? The environment secures against cross-site scripting, so let’s not worry a much about maybe that as something that can be exploited.” Who uses that information?

    And then I guess the third part of it is, it’s about continuous improvement, that’s a cyclical process of, “Are we getting better? Are we getting value? Is there benefit to the work we’re putting in on this?”

    Worrell: You bet. So, you’ve actually said a lot there, too, so, I’ll try to—

    Ashley: I did. I’m just following your lead, John. [Laughter]  

    Worrell: Yeah. [Laughter] Good, we can confuse each other in this whole process.

    So, you know, if we start with this idea of the continuous improvement, just kinda going backwards here, the whole goal is to have that process, which is a continual healing process, if you will, you’re always gonna get better. DevOps does that with telemetry about, “What is the productivity of my developers? What are my key metrics on how many story points I’m taking down, how many am I delivering, and am I getting better, sprint after sprint after sprint?” All great stuff—the same model plays out in security.

    That ties back to one of your comments about the role of the CISO. You mentioned, you know, typically, security guys, via technology, they deploy it. And that works or a proactive control, which is what security’s been doing so much for. My days of running the product for RSA SecurID and working at CyberArk and any kind of blocking technology or control—it is. You buy it, you deploy it, you set it and forget it, for the most part, and it works, right?

    By the same token, this isn’t really responding on the SOC side, either, right? This is not responding to an in process attack, this is a different approach to security. And I think the reason that’s important is, number one, businesses are demanding more business metrics, business telemetry out of their security programs, what am I getting for my investment? How do I know I’m investing in the right things? Am I spending in the right places? All of those things are really important, right?

    And the role of the CISO, by definition, is evolving as well. And you can see the people that were running security teams 10 years ago are very different than the model who’s out there now. Many people have grown into that role. Other people have come from the business side of the house, and they’re not the technology experts that were good at configuring firewalls that were really trying to deliver business value to the organization. And if you’re gonna step from infrastructure into the world of application security, you need to be able to have value to the business, and this data is a great way to deliver value to the business.

    Ashley: Really important point, because the CISO role is evolving, and oftentimes, it’s becoming a CIO/CISO kind of role, which means you’re getting into the application stack, not just the network stack. Not saying that they weren’t worried about the app stack, but now you’re getting much more into the nitty gritty of it. And are your security people software people? Well, not necessarily, but they are very accustomed to getting data and responding to that data, and in this case, maybe helping development teams, developers, others with understanding what the true severity, not what some scanning tool might say, right?

    Worrell: You bet.

    Ashley: Which is sort of our history with vulnerability management, [Laughter] to really help them understand, “Okay, great”—and then demonstrating to the business, to compliance, to whoever you may need to be reporting to, it could be customers as well, to say, “Here’s what we’re doing. Here’s the actions that resulted from investment in these tools, people, and processes.”

    Worrell: And that customer angle is actually really important. And I hate to use the word SolarWinds, but I’ll use it. It’s just put, yet again, a big spotlight on software supply chain, software security, and how critical that is. And so many organizations now are asking of us, as well as our customers, “What is your software development process? Prove to me you’ve got one.” Right? Great use of this data.

    When you think about your business line risk owners, the people that are actually responsible for data, they work for the GM or the SVP of that business unit, they’ve got the risk management challenge. They need this data to do their job well. If you take that data to the board, our data is being used as part of the quarterly board package to talk about how they are addressing and how they are improving the risk posture of the organization.

    And then, I wanna go down, really, one level deeper. Because when you think about this, you know, integrating into the DevOps process means that it’s not necessarily just about the developer making that independent decision. Typically, there’s a product owner in there some place that’s gonna be making some decision about what’s gonna get done now versus what’s gonna get done later, right?

    So, being able to feed that product owner with the data they need to help do the triage and make those decisions, that’s another great use case of this data we’re pulling together.

    Ashley: You know, you mentioned a lot of roles, too, including the product person. In your experience with ZeroNorth—and I really appreciate your background in security, too, so we can relate about those days as well—what’s most often, when people come to ZeroNorth and say, “Hey, we need your help, here’s what we need your help with?” Is it the software team, is it the product team, is it the compliance group, is it the CISO/CIO? Usually, what’s that initial driver for engaging with you.

    Worrell: Yeah, there’s probably two different ones we hear most commonly. One is, “Hey, I’ve got a somewhat mature program, I’m running two or three different tools. I can’t make heads or tails out of the data, or if I try to do it, it’s manual, I can only do it once a quarter or once a month, it’s just too labor intensive. So, I think I’m getting good, I’ve got a decent process right now for scanning, but I wanna start pulling that data together.”

    The second use case is those that wanna start out on building what we would call a federated or a shared governance model, and it’s led by the security team, perhaps, but they know they’ve gotta bring in the business line owners into this process, or business line risk owners into the process. And they say, “Hey, can you help me get data out there so I’ve got a single source of vulnerability truth or my applications?” And I’m looking at the same data that they’re looking at, we’re all making decisions off that same set of data so we can build a governance model that makes sense. 

    The security guys wanna work in partnership with the business owners and the people that are actually doing the software development and risk owners in the business. But that relationship is gonna get built on having really good data and everyone using the same data. I think that’s another use case.

    The third one, which is a variation of that, is people that are coming in and really saying, “Hey, listen, I need to go to full DevSecOps right away. I know that I can put this layer of reporting and analytics on top of my current tooling and that’s a great start, and I’m gonna build this over time.” It’s a fine strategy. Other people are coming in because they’re just getting started or they just have a business imperative to do it and they’re saying, “We’re going to DevSecOps right now, and we wanna do the pipeline integrations as well as the reporting and analytics that we’ve been talking about so that I do get that data in real time, I do have a consistent scanning process, it is automated, I don’t have to worry about the developer forgetting or not doing it, and I’m gonna get that data value out of this in that particular way.” 

    Ashley: Yeah, “Help me accelerate through the learning curve instead of making all those mistakes myself to get to DevSecOps, if you will.”

    Worrell: You bet, you bet.

    Ashley: That second example especially kind of is a multiplication of the first one, which is, you have Dev teams with all this data, maybe it’s the primary group—usually, it’s multiple, right? How do you pull it together into something? They can’t make sense of it. How do you do that across the enterprise to meet compliance requirements or even just your own internal assessment and reporting up the management chain maybe to the board.

    Worrell: You bet. If you step back, the biggest concern we get out of just about everyone we talk to is a CISO or a Risk Manager saying, “Hey, help me, I’m flying blind. I don’t know what highs and criticals I’ve got in my production environment right now. I just don’t know. I don’t have visibility into it. And for me to get it, it takes too much effort, and once I get it, it’s outdated, because it’s 30 days old. I’m pushing code every day” or every week or whatever it might be.

    And that is the primary motivator or so many people who just say, “Gimme that visibility, and that’s the starting point. And then at least I know what I’m dealing with and then I can start building out a program that’s gonna help me address that over time.”

    Ashley: You know, you brought up SolarWinds, so I’ll bring up COVID. What have you seen happening over the last 12 to 18 months with the acceleration to the cloud, acceleration of digital transformation projects? Has it taken these same issues and just made them even more visible critical to address, or have new problems emerged?

    Worrell: I think it’s more of an acceleration of the entire process. In general, COVID has driven people to be more comfortable or more reliant on relationships with their customers, with their partners, with their employees. And that means more software, by definition.

    So, as digital transformation accelerates because of more online interaction and new business models that this opens up, it’s just really accelerated the pace and some of the maturity of people coming. The types of questions we’re getting from our prospects now are much more advanced than they were a year ago. And that just shows that the market is really getting smart about this and they’re kind of moving beyond the tool only approach to say, “Okay, how do I get some business value out of this that can really, really help us move our risk management program forward?

    Ashley: That’s one of many indicators I see, kind of data points that are saying the technology groups, the software groups are getting out from under, looking at themselves and starting, trying to get aligned with the business and meet where the business is headed.

    Worrell: And as a big consumer of software, as we all are in our lives today, I’m really happy to see that, right? I mean, that’s what I—I know it’s gonna take bringing the two together and working cooperatively in that federated or shared model to make this work, and I’m really happy to see that kind of a progress. Because, like you and everybody else, software just drives so much of our daily experience.

    Ashley: It is. Can’t do much without it these days. Well, it’s been great talking with you, John. Where can folks learn more about ZeroNorth?

    Worrell: Well, that’s easy, ZeroNorth.io. We’d be happy to talk to you. Take a look, see what we have, and if we can help you out or you wanna hear more about it, we’d love to chat with you. Thank you very much.

    Ashley: Yeah, I think we’ve had a great conversation. Hopefully, that’ll spark some folks to elevate that conversation with you and the ZeroNorth team, so take care, we’ll talk to you again soon.

    Worrell: Thank you, Mitch. I appreciate it.

    Ashley: You bet. 

  • Linux Foundation Project Secures Software Supply Chains

    Linux Foundation Project Secures Software Supply Chains

    The Linux Foundation today embraced a sigstore project founded by Red Hat, Google and Purdue University to make it simpler for developers to employ cryptographic software, enabled by transparency log technologies, to secure software supply chains.

    Luke Hinds, security engineering lead in the office of the CTO at Red Hat, said he and Dan Lorenc, a staff software engineer for Google, led a small team creating a lightweight mechanism for signing software artifacts, such as release files, container images and binaries, in way that creates an immutable, tamper-proof log.

    Now, the goal is to take that project to the next level by contributing it to the Linux Foundation, at a time when the focus on software supply chain security has increased in the wake of a spate of high-profile breaches. As part of that effort, a free service, based on sigstore, will also be made available by the Linux Foundation to make the cryptographic software easily accessible to a wide range of developers.

    Hinds said one of the reasons that artifacts are not signed by developers is that the current processes employed are too cumbersome. The cryptographic software is designed to be integrated as an extension of existing workflows, Hinds noted.

    Because the log created is immutable, everyone participating in the software development process, including auditors, can now transparently track chain of custody for any software artifact, Hinds said. At its core, sigstore creates a set of short-lived ephemeral keys with a trust root based on an auditable public log.

    Existing approaches rely on digests stored on insecure systems that are susceptible to tampering. Cybercriminals can, for example, swap out digests if they can gain access to compromised credentials. Those types of targeted attacks can be thwarted using sigstore, because the log management system that underpins it is immutable, said Hinds.

    Of course, no system for securing software supply chains is going to be useful if developers won’t employ it. Key management, key compromise/revocation and the distribution of public keys and artifact digests are challenging to manage. Developers, then, need to determine which keys to trust in addition to learning how to sign them when different platforms are employed. Digests and public keys are also often distributed across vulnerable websites susceptible to hacks, or stored as a README file on a public git repository.

    As organizations embrace DevSecOps best practices, it’s only a matter of time before the entire software supply chain comes under greater scrutiny. Ultimately, the issue will come down to finding a way to secure those supply chains, either from the top down by deploying and maintaining a platform, or simply making it easier for developers to better secure artifacts from the bottom up.

    Regardless of approach, the signing of software artifacts should, one way or another, become more routine in the weeks and months ahead. The biggest challenge will be overcoming the simple inertia that currently exists among developers who routinely ignore the whole process.

  • API Security in DevOps: Are We Too Comfortable?

    API Security in DevOps: Are We Too Comfortable?

    Postman performed its annual survey of developers, and as this survey and many other surveys have shown, developers are generally comfortable with the level of API security that their organization has implemented. This has been a consistent trend, with most developers being comfortable that they are protected.

    The problem is most organizations also put API security and management in the hands of developers.

    Knowing the rate of change going on in Agile/DevOps shops, and knowing the complexity of environment, I think this mentality is akin to the pre-WW2 French belief in the Maginot Line.

    While API security is certainly a necessity, it is not managed in a vacuum. You really are only as secure as your weakest link, and there are a lot of links in an API-focused environment. User credentials, tokens, SSL, requesting apps and their authentication, (still) code injection concerns…The list goes on and on.

    I’m never comfortable with the level of API security currently implemented in DevOps environments. And I think it’s dangerous for you to be. It only takes one bad code iteration for one of the client instances to be vulnerable. That vulnerability exposes every API the client in question has access to, no matter what is being used as access control.

    In the end, APIs must trust someone, or they’re useless. And that someone is most often a server app running on an instance that your users connect to. That app most often has its own credentials. How are they stored? How are they protected? What code changes could impact that instance’s security?

    API gateways and management tools, when properly configured, are a big help here, but they are not a panacea. Since incoming connections must be trusted at some point, and outgoing connections must be assumed clean at some point, when one of those two things is a wrong assumption, what do you have in place?

    Here’s the scenario: User A has rights to call the ShowMeCriticalData API. This API is called from a server app running on instance B. The server app authenticates user A, then calls the API for him–either with his credentials (best case) or its own (bad for tracking problems). If instance B receives an update that makes it vulnerable to compromise, user credentials are still needed to call ShowMeCriticalData, which is relatively secure, as long as User A is protecting their credentials adequately. However, if the app running on instance B is proxying requests and using its own credentials, Instance B becomes a weak point and attackers could use the vulnerability to call APIs without authenticating users. Worse, if User A is careless with their authentication info, there is little an organization can do to stop at least some misuse.

    API logging and review can be put in place to notice odd behavior by username, and offer early warning that a user has been compromised. But will only catch most issues.

    Of course, there are smart people working on the whole array of API security threats from cross site scripting to authentication spoofing even into TLS weaknesses. But that work is only useful if we all remember that our security could be better. With knowledge that there are weaknesses, we can look for things to mitigate the risk. As you all know, if you have public APIs the risk will never be eliminated–the purpose of a public API is for people to use it, and that exposes it to hackers that seem to be more ingenious by the week–but minimizing the risk is worth the effort.

    Being secure in your current situation means you’re not focused on making it better. With API security, particularly in the type of dynamic environment DevOps creates, this is a dangerous road to be on. Assume there is a fatal flaw, and work to reduce vulnerabilities at each access point into the overall system.

    This also goes for APIs you don’t own. We rate limited an API that called out to Google Places to match up with our plan/budget for the project. Our app was never going to come anywhere near Google’s caps on usage, but should a portion of our infrastructure be compromised, we could potentially end up with a massive bill from Google because we called the API a zillion times in a month. Likely? Not really. Possible? Not after a few lines of rate-limiting code.

    You all have made some astounding apps. Make sure they are astoundingly protected. Because both business and users are counting on it.

    — Don Macvittie

  • DevSecOps Adoption and the Web Security Myth

    DevSecOps Adoption and the Web Security Myth

    As DevOps practices have become widespread in the tech community, many people have begun proclaiming the virtues of DevSecOps. As the name implies, DevSecOps is the addition of security into DevOps. Just as DevOps promises better-quality production in less time, DevSecOps promises better security with less time required to achieve and maintain it.

    DevSecOps has many benefits. However, many executives are under the impression they can’t embrace DevSecOps across their entire organization.

    This idea is false. Most organizations that have already adopted DevOps can, and should, take the next logical step and adopt DevSecOps as well.

    ‘It Sounds Great, But We Can’t Use It Here’

    Adopting DevOps requires far-reaching changes throughout an organization. Going to DevSecOps can be similarly disruptive. However, internal disruption is not usually the (perceived) problem with DevSecOps.

    Instead, the problem is usually the network border. Many executives believe that their current web security solutions are incompatible with DevSecOps.

    In many cases, this is actually true. Older approaches to web security are indeed hindrances to DevSecOps practices.

    Therefore, the problem isn’t actually with DevSecOps. The problem comes from the security solutions currently in use.

    Legacy Security Solutions and DevSecOps

    Many organizations are still using appliances (whether physical or virtual) as their primary protection against external threats. A decade ago, appliances were good enough. Today, they’re a competitive disadvantage.

    There are several ways in which appliances are incompatible with DevSecOps:

    • Programmatic control of an appliance is often difficult if it’s even supported at all.
    • Configuring an appliance can be challenging. It often requires a high level of security expertise.
    • Rolling out settings and configuration changes across a bank of appliances can be time-consuming and potentially error-prone.
    • The cost and specialized nature of most appliances make them inconsistent with the DevOps mantra of “cattle, not pets.”
    • Physical appliances do not scale.
    • External-facing applications will be limited in their scaling as well.
    • Appliances are not designed for ephemeral workflows.
    • Appliances are not designed to support infrastructure as code (IaC).

    Security appliances represent a decades-old approach to security, so it’s not surprising they don’t support modern practices. What is surprising is that even many allegedly “cloud-native” solutions aren’t built for DevSecOps, either.

    For example, some require multiple instances to be launched into your environments. Some require additional layers of management just to maintain consistent configuration among them. Many do not provide good support for evolving architectures and expanding deployments. In general, these security products tend to require significant intervention and management, while lacking automated control capability. This all severely limits the use of DevSecOps.

    These appliances and incorrectly labeled “cloud-native” solutions are why many executives believe DevSecOps isn’t an option for their organizations. To be clear, these executives are correct—for as long as they continue to use previous-generation approaches to web security.

    Not Just a DevSecOps Issue

    Older approaches to web security have other problems that go beyond a hindrance to DevSecOps usage.

    For example, appliances are marketed as complete web security solutions, but they cannot actually fulfill this role. Among other things, an appliance cannot defend against a volumetric DDoS. The attack can overwhelm the upstream ISP before the appliance even has a chance to scrub the traffic. This can result in the ISP blackholing all incoming traffic, which makes the targeted site and web applications unavailable to users and customers—which is the exact situation the appliance was supposed to prevent.

    Solving the Problem

    Organizations which are still using older security technologies should re-evaluate this decision. Modern solutions such as cloud web security platforms can provide better protection, along with numerous other benefits. These include full management by the provider, real-time reporting and traffic control, adaptive threat identification based on machine learning and much more.

    When organizations continue to use legacy security solutions, they not only prevent themselves from enjoying the benefits of DevSecOps, they are also missing out on many other benefits as well.

    — Maksim Blishchik

  • 6 Traits That Define DevSecOps

    6 Traits That Define DevSecOps

    How do we define DevSecOps? A combination of DevOps and security is readily apparent, but the philosophy goes much deeper. In a recent eBook, The State of DevSecOps, we asked industry experts to define what DevSecOps meant to them. Below, we’ve condensed their answers into five core attributes.

    Following these principles, CIOs or CTOs now have a DevSecOps doctrine applicable to nearly any software development and release environment.

    Security Automation

    DevSecOps emphasizes security objectives within the automation processes. Ben Newton, director of product marketing and evangelism at Sumo Logic, defines DevSecOps as first making security requirements and objectives a clear part of the continuous integration (CI) and continuous deployment (CD) cycle.

    Culture of Security

    DevSecOps establishes a team culture that embraces security concerns. As stated by Ben, a security team must provide “clear guardrails for developers as to what is fair game and what is not appropriate from a security perspective.” Building security expertise to integrate security into the entire DevOps lifecycle is critical.

    De-Siloing IT

    DevSecOps is about eroding boundaries. Just as DevOps has eroded the traditional separation between software engineering and IT operations teams, “DevSecOps further erases the walls between the DevOps team and IT Security,” said Tim Jarrett, senior director of product management at Veracode. “DevSecOps is about building a bridge between the security and DevOps teams,” echoed Dan Hubbard, chief product officer at Lacework.

    Security Shifts Left

    DevSecOps places security earlier on in the development process. IT security is traditionally viewed from a risk avoidance and compliance standpoint. Rather than viewing security as a gate, Tim noted that within DevSecOps, “security is better positioned to integrate earlier in the development cycle where they can actually make a difference.”

    Security Enables, Not Stalls

    DevSecOps supports, not stalls, agile development. DevSecOps doesn’t have to be sluggish. As Dan described, DevSecOps must “support the need for DevOps to move fast, but in a way where security is not ignored.” By embracing a security-as-code mindset and involving practices such as automated threat detection, agility is not sacrificed.

    — Bill Doerrfeld

  • The DevOps Security Stack

    The DevOps Security Stack

    The whole idea behind DevOps is automation. DevOps automates the development process by eliminating or minimizing human error. If an error occurs, the applications fail quickly so the errors can be resolved promptly on a smaller scale. DevOps and cloud-native applications complement each other perfectly.

    Even though DevOps offers a more efficient and faster way to develop and deploy applications, security remains an issue. On paper, DevOps and cloud-native applications might be more secure than their traditional counterparts, but the reality is not that simple. With the newer, more complex, highly distributed infrastructures common to DevOps come new challenges that need to be identified and addressed.

    This article walks through those challenges to highlight what it takes to secure a DevOps workflow. I won’t focus on specific tools because DevOps is not about specific tools; rather, I’ll focus on the components of DevOps workflows that present special security challenges and explain how to address them.

    The Vulnerabilities You Just Can’t Ignore

    1. Careless use of recycled code

    One way DevOps improves developer productivity is by letting developers recycle code from previous projects or open source repositories such as GitHub. But without proper screening and sanitization, these pieces of code can pose threats to an application.

    1. Multiple platforms make the attack surface larger

    When deploying code in the cloud, you can’t use traditional security services. You need to rethink your security strategy. DevOps teams leverage various platforms, from serverless computing to microservices to multi-cloud in their applications. There’s no single solution to secure cloud-native applications. These applications are highly distributed and have various components, so organizations need a multifaceted approach to better secure these applications.

    1. The dynamic nature of cloud-native applications doesn’t help

    Cloud-native applications are continuously integrated and deployed, which makes it difficult for security teams to identify and mitigate risks. Traditional security tools can’t match the sheer velocity, scale and dynamics of cloud-native applications, leaving them useless.

    1. Secrets need to be secured

    When following the DevOps approach, security professionals need to create privileged accounts and login details and share them over business networks automatically. With serverless applications and the cloud, security teams don’t have a traditional location such as the operating system to execute security policies. A host of useful data, such as keys, storage account credentials, secrets, database passwords and embedded passwords, are stored in repositories. It’s crucial this data is kept extremely secure, as attackers can use this information to cause some serious damage.

    1. Containers can leave your applications more vulnerable

    Since the microservice approach involves a distributed infrastructure, it exposes your system to the network, which makes it easy for attackers to sneak into the system. Containers can be spread across different systems, which makes the threat landscape even larger. Containers are highly replicable, so if there is a vulnerability in one microservice, it can get replicated every time the source code is used.

    Using open source software in containers can make way for an increase in vulnerabilities. DevOps teams should make sure unauthorized containers are not used in production. These containers can be used by attackers to launch an attack on the entire application.

    1. Hosts aren’t automatically safe

    Even if containers are properly secured, the hosts they run on are vulnerable to new threats and day-zero attacks. Container runtime should be up-to-date to avoid risks that can lead to container escape. This can help attackers to take control not only of a container but also the host operating system. So, it is important that hosts are given appropriate protection.

    1. Kubernetes clusters need additional security

    Kubernetes is a widely accepted container orchestration platform. However, it’s difficult for security to be enabled at multiple layers. Kubernetes uses APIs to manage containers, and recently, it was discovered that K8’s API server is quite susceptible to attacks—attackers can make use of the API server to perform malicious activities or install malware in the application.

    Running multiple workloads on one cluster is not a good idea. Sensitive workloads should be run on a dedicated set of machines to avoid attacks through less secure neighboring applications. There’s also a need to secure sensitive metadata, which can be stolen and used to change privileges in a cluster, thereby paving the way for unauthorized changes in an application.

    DevSecOps: The Future of DevOps Security

    In traditional applications, security holds less priority, and security policies are always employed after development is completed. But with DevOps, security can’t be an afterthought. Companies depending on the DevOps approach need to make security their top priority. One of the ways to make sure security is in the foundation of a DevOps implementation is a DevOps security stack. To employ the DevOps security stack, IT teams no longer should be walled off from the DevOps team.

    Security experts’ involvement can help them gain insights into what the application is and how it’s supposed to function so they can identify risks and take steps to mitigate them. This DevSecOps approach has become popular. DevSecOps is a collaboration between the development, security and operations teams who work together to ensure security is at the root of an application, not just on the surface.

    Symantec Cloud Workload Protection: A Modern Solution for Modern Problems

    Most organizations use hybrid cloud storage comprised of workloads distributed among public clouds, private clouds and on-premises, which can lead to an increased attack surface and vulnerabilities.

    Symantec Cloud Workload Protection (CWP) helps organizations monitor and protect their workloads, no matter where they reside. With CWP, organizations don’t have to look for multiple products to meet their many security needs. CWP offers a single console to monitor and manage security across various platforms. It offers automatic discovery of workloads across AWS, Azure and Google Cloud, and visibility into security postures and software, which enables automatic workload monitoring and protection. With continuous delivery workflows and malware prevention, CWP is essential for modern software development.

    Sign up for a free trial of CWP.

    — Twain Taylor

  • DevOps and Security: The Path to DevSecOps

    DevOps and Security: The Path to DevSecOps

    A secure DevOps is highly beneficial for organizations. However, a secure DevOps environment shouldn’t be the end goal. When it comes to DevOps security, it’s important to integrate security right from scratch, secure the entire architecture, automate the security and use this technology to test the environment and the codes and respond to concerns immediately. This can help organizations safeguard their creations, protect the confidentiality of their customers and strengthen their applications, all before data breaches can occur.

    DevSecOps is a philosophy that focuses on introducing a culture of security in the DevOps environment. It is a collaboration of the DevOps teams with cybersecurity and system security personnel, where the focus is on finding faster and more efficient ways to safely deliver codes in an agile architecture. DevSecOps strives to bridge the gaps between IT and security while responding to bottlenecks in the existing environment.

    Some of the biggest benefits that organizations will get when they move from traditional DevOps philosophy to the DevSecOps philosophy is:

    • Improved operational effectiveness and efficiency.
    • Stronger and healthier collaboration between teams across the company.
    • Greater agility for security teams.
    • More conducive environment for automated builds and quality assurance testing.
    • Easier to identify system and application vulnerabilities.
    • Greater freedom for personnel to focus on high-value projects.
    • A greater degree of transparency into the environment.
    • Improved scalability in the cloud.
    • Increased ROI.

    Key Elements of the DevSecOps Environment

    Microservice-Based Infrastructure

    Single-function modules that contain well-defined interfaces and operations is essential for the success of focused DevSecOps. Constantly monitoring, upgrading and tweaking the microservice-based infrastructure will help organizations to be equipped for new developments.

    Hybrid cloud environments, software-defined networking and network micro-segmentation must be integrated into the infrastructure, and these must be used to better define parameters, assess and identify connections, verify access and monitor the organization’s online assets.

    Continuous Feedback Loop

    The next most important element of the DevSecOps environment is feedback. Setting up a continuous feedback loop will help developers and machines get a comprehensive insight into system/platform vulnerability to security threats. This type of real-time, continuous feedback can help organizations set in place the right policies and rule sets, which can keep the application security testing tools updated and relevant regarding the security status of the organization’s software/network/platform, in addition to keeping all parties updated about the potential threats to the DevOps environment.

    This type of continuous feedback loop acts as an enabler, as opposed to an inhibitor of business, by allowing organizations to stay well-equipped and constantly on guard.

    Focused Automation

    Finally, continuous and focused automation is essential to the success of the DevSecOps environment. Automation, when woven into the software development life cycle right from the start, can reduce the friction that may occur between development and security teams over software/platform security by quickly addressing existing and potential concerns at the lowest cost.

    There are certain open source tools on the market that can help organizations automate their security:

    • Continuum Security: This tool works on a BDD-Security framework, which is compatible with unit testing frameworks, issue trackers, SAST and DAST. It offers an open IriusRisk API for anything the tool doesn’t natively support.
    • WhiteSource: This particular tool specializes in addressing open source vulnerabilities, alerting users about the existence of threats throughout the pipeline. It is compatible with 200 programming languages and it actively analyzes the license, quality and security of all open source codes.
    • ThreatModeler: This platform uses the functional information that developers input to analyze the software and provides information about potential threats. It also offers actionable inputs and security test cases to enable easy implementation of security.
    • Evident.io: This solution is designed for the deployment stage and helps assess and manage cloud security risk, particularly on Azure and AWS. Evident Security Platform (ESP) offers continuous monitoring, early threat identification and threat mitigation.
    • Aqua Security: This tool is ideal to manage end-to-end DevSecOps pipeline security. It has very tight runtime security processes and controls in place. It offers full control over the containerized environment and prevents any intrusion or vulnerability across the pipeline.

    There are many other tools that organizations can use to automate their DevSecOps environment. These include: Dome9 (SaaS) Security, Contrast Security RASP & IAST tool, IMMUNIO RASP tool and Checkmarx SAST tool.

    Shifting to the Left

    In today’s agile world of development, following outdated waterfall philosophies is a death sentence for companies.

    For successful implementation of DevSecOps, organizations should shift left. In other words, they should integrate deployments and testing right from the start, and these must continue till the very end when the software/platform in question is vulnerability-free. Etsy and Amazon, for example, conduct more than 1,000 deployments per day, each aimed at recognizing potential threats to their systems. Such a highly dynamic DevSecOps environment ensures that the velocity of new development isn’t slowed, and the quality of new developments is maintained at a high level.

    This type of “shit left” philosophy not only accelerates new developments but also helps limit the security threats and addresses existing threats at the least cost with minimal damage to the software/platform.

    Incorporating AI and Machine Learning into DevSecOps

    Research shows that more than 50 percent of all companies will implement machine learning in their DevSecOps operations in 2019, due in part to the various benefits AI and ML offer to organizations that are actively maintaining a DevSecOps environment. These benefits include:

    • AI and ML will bring down DevSecOps security review time. They will do this by increasing the speed and quality of false positive identification, which will reduce the time spent on threat vector identification. This will help developers improve the speed at which they recognize the threats to their systems.
    • AI and ML will make up for the cybersecurity talent shortage, thereby ensuring that an organization’s DevSecOps environment isn’t left unmanned at any time. AI and ML will dip into human intelligence and use this to fuel their involvement in the DevSecOps environment. Additionally, AI and ML give organizations the ability to understand how other machines function, thereby helping cybersecurity personnel understand the psyche of cyberattackers.
    • AI and ML will actively find defects in the DevSecOps, help developers identify what they’re doing wrong and what they need to do to rectify these concerns. Constant monitoring is a core feature of DevSecOps, and AI and ML will serve to add more advanced technological backup to power this feature. AI and ML will also offer developers customized secure code patterns based on the organization’s unique DevSecOps landscape, which will assist in vulnerability detection.
    • AI and ML can help organizations move their DevSecOps from a small scale endeavor to large-scale. This will be a possibility because of the sheer technical prowess and the real-time information that AI and ML will furnish to developers and cybersecurity personnel.

    7 Rules to Successfully Implementing a DevSecOps Environment

    • Integrate security from the start and across the pipeline.
    • Automate your security.
    • Track and monitor each software stack meticulously to identify which needs patching.
    • Implement code dependency checks such as the OWASP Dependency Check, vulnerability assessment and discovery tests regularly.
    • Put in place robust policies to manage the DevSecOps environment.
    • Break your tasks into manageable chunks—this will improve the consistency of deployments.
    • Set in place one-click compliance reporting to increase traceability and transparency of the pipeline, right from code planning to code changes to release.

    Conclusion

    DevSecOps will bring massive economic and technical advantages to organizations, apart from equipping them with the capabilities to create, run and offer state-of-the-art software/applications. Organizations wanting to remain relevant and competitive in the industry must consider DevOps security as their primary goal in 2019.

    — Subramonian