devopsy.kinsta.cloud

Category: DevSecOps

GitLab Allies With Rezilion to Add Workload Analysis Tool

Rezilion has integrated its workload analysis tool with the continuous integration (CI) framework provided by GitLab. The move is part of an effort to make it simpler for developers to discover issues such as vulnerabilities before they upload code into a repository.

Sam White, a senior product manager for GitLab, said this integration will provide developers with an instant feedback loop that will enable them to address a wide range of issues before their code is ever reviewed. Rather than experiencing the shame of, for example, missing a vulnerability, the Rezilion platform enables developers to quickly write code while relying on a code analytics tool to surface common mistakes that they can remediate as they see fit, he added.

Rezilion CEO Liran Tancman added that the proprietary workload composition analysis engine, dubbed Unison, on which the Rezilion platform is based makes it feasible to shift more responsibility for application security further left without requiring developers to become cybersecurity experts.

Unison automatically creates a model of all the applications that include the underlying infrastructure and runtime environments. It reverse-engineers and maps that environment in memory to dynamically track the inventory, provenance, runtime execution, exposure and interdependencies between each piece of code and can generate a software bill of materials (SBOM).

The code that is most in need of immediate attention is also surfaced directly with the GitLab user interface, said Tancman. That capability can reduce remediation time of the vulnerability backlog an organization might be working through by as much as 70% over time, he added. Rezilion enables organizations to achieve that goal because non-exploitable vulnerabilities are marked as “false positives” that shouldn’t hold back a release.

Tancman said that without this level of automation it would be extremely challenging for an organization to implement a set of DevSecOps best practices. It might take developers years to attain the level of cybersecurity training that would be required in the absence of the automation enabled by the Unison engine, he noted.

Historically, the relationship between application developers and cybersecurity teams has been tense. Cybersecurity teams tend to put off code review until it’s about to be deployed in a production environment. It’s not uncommon for an application to be pulled at the last minute because of a vulnerability discovered by a cybersecurity team. In some instances, that vulnerability is a legitimate concern but in others, it turns out to be a false positive because the flagged piece of code isn’t actually included in the application.

By providing developers with a workload composition analysis tool within the context of a CI framework, GitLab is trying to reduce the level of friction that the process tends to create between developers and cybersecurity teams.

It may be some time before most organizations fully master DevSecOps, but it’s clear that merely telling developers they will be held accountable for application security without providing the appropriate tools isn’t going to make applications any more secure.

March 29, 2022
3 Must-Haves When Implementing DevSecOps

The term DevSecOps is already more than a dozen years old. DevOps—the practice of combining software development with IT operations to deploy applications faster—was first coined in 2008 and refined in a historic conference presentation in 2009. DevSecOps—the concept of baking in security at every stage of Agile development, rather than tackling it at the end—followed soon after.

Despite a huge amount of industry conversation and activity around DevSecOps, it has been hard to tell just how much traction the movement has been gaining.

In a 2020 survey of 1,500 IT professionals around the world by the Synopsys Cybersecurity Research Center, 63% reported that they are incorporating at least some DevSecOps activities into their development pipelines. “At least some” indicates, however, that a portion of the adopters have yet to fully embrace DevSecOps. And what about the other 37% who haven’t incorporated it at all?

Nevertheless, two factors suggest that DevSecOps is starting to pass a tipping point on the way to mainstream adoption.

One, the prodigious number of cyberattacks that have been hitting companies and government agencies demand aggressive and proactive approaches to address security, which is exactly what DevSecOps does.

Two, the pandemic has been a powerful accelerant for digital initiatives. For example, global spending on public cloud services was predicted to grow slightly more than 23% in 2021, to $332.3 billion from $270 billion in 2020, according to a forecast from Gartner. In addition to the greater reliance on technology driven by COVID-19, increasing use of emerging technologies such as containerization and edge computing are fueling higher cloud spending. All of this points to the need for organizations to take security risk into account throughout the modern software development life cycle.

The logic behind DevSecOps is undeniable. Viewing security as a shared, collaborative responsibility among development, operations and security functions starting at the earlier possible stage—the methodology known as shifting left—rather than in silos enables the deployment of secure software at the speed and responsiveness that today’s business conditions require.

Companies that have successfully meshed the three functions under the broader DevOps paradigm enjoy radically reduced friction in software development and production. These organizations can deploy software faster and more safely with fewer design flaws, misconfigurations and other missteps that can lead to security issues.

Companies not as far along in their DevSecOps journeys tend to have espoused some of its practices but not consistently. Clunky, linear waterfall approaches persist here and there. Vulnerability management remains a reactive box-checking effort later in the life cycle, forcing developers to backtrack and fix the code rather than a proactive, preventative process seamlessly integrated from design to production.

The more advanced organizations typically focus on three key ingredients to assure success.

1. Education. DevSecOps is a cultural shift as much as a technological one. It requires a new mindset that empowers developers to take ownership for creating secure code at the beginning of the software development life cycle. Thus, IT leaders can’t simply snap their fingers and expect DevSecOps to take hold without educating team members on why it is so critical and how it drives rather than disrupts great work.

“Organizational learning and change are key to allowing DevOps to flourish,” senior Gartner analyst George Spafford wrote. “In other words, people-related factors tend to be the greatest challenges, not technology.”

This means that before introducing DevSecOps into an organization, it is important to connect the endeavor with the value it will bring and explain why it will allow the business to deliver more value to customers and improve its competitiveness.

2. Process. The right DevSecOps policies and processes can spell the difference between a nimble software development machine and one still bogged down by inefficiency.

For instance, many organizations that rely heavily on scanning tools to detect vulnerabilities assign nonsensical policies to their use. Let’s say that vulnerabilities of a particular score or type need to be eliminated before code is deployed to production. But what if the flaw resides in an unimportant section of the code and is unexploitable?

The best DevSecOps initiatives include policies and processes that seek to understand the entire context of a project, from design to production and handle everything in a way that reflects the context of true risk to security and compliance requirements rather than merely running all code through the same pipeline.

3. Tooling. Fact: Without the right automated tools that provide deep visibility into code, a DevSecOps implementation doesn’t stand a chance.

To carry out the qualities of collaboration and agility that DevSecOps promises, organizations should gravitate toward tools that give developers all the relevant information they need to make more informed decisions.

These tools must truly simplify the addition of security into DevOps methodologies by automating various manual processes that traditionally have slowed everything down—threat models, security and compliance reviews, pentests and risk assessments and the like. The tools can’t just aggregate data from silos — which fails to provide enough contextual information and generates too many false positives —they need to provide holistic relief to overburdened teams in securing applications at the feature request, commit, pull request and CI/CD stage.

By keeping these three elements in mind, organizations can fulfill DevSecOps’ potential as a true game-changer. As DevSecOps continues to cross the chasm to more universal adoption, organizations can look to earlier adopters for guidance on how to do DevSecOps right.

March 24, 2022
How Work-From-Anywhere Impacts DevOps

Brian Lavallée from Ciena talks with Mike Vizard about how the work-from-anywhere movement has impacted DevOps, networking and security. The video is below, followed by a transcript of the conversation.

Mike Vizard: Hey, guys. Thanks for the throw. We’re here with Brian Lavallée who is senior director for solutions marketing at Ciena. Brian, welcome to the show.

Brian Lavallée: Thanks. Nice to be here.

Mike Vizard: You guys just put out this study talking about what’s happening on the technology impact from working from home. A lot of folks are accessing cloud resources. They have home networks that may or may not be up to the task. Are we looking at some sort of networking crisis coming up as we move to working from anywhere? Do people understand that bandwidth is going to be an issue?

Brian Lavallée: I wouldn’t say it’s a crisis, but it is a work in progress. Just over the last year, year and a half, the global pandemic, our homes have turned into the new branch office, the new classroom. People are learning, playing, and working from home, and it’s put additional stress on the network. One of the main challenges is speeds, because now you have two, three, four people accessing the same broadband connection from home at the same time, and then you have the whole security aspect, where if your home is the new branch office, very often you need additional security to do work, not only with your head office, your colleagues, but also with your cloud applications.

Mike Vizard: I have found myself trying to fight for bandwidth with an 11-year-old, so I know what the issues are and what the problems can be. Do you think companies, as we go forward, are going to turn around and say to their employees, “Here is the router we want you to use”? Are they going to give them that router, or the access point at home? How far will they go to make all this happen and maybe provide that additional layer of security?

Brian Lavallée: Yeah, I think, for sure, adding additional security, even availability, a more resilient connection is necessary going forward. Companies or enterprises are going to implement this differently. Some may do a lot of their security capabilities in the cloud. This allows you to have a centralized location to do all your security, you know, talking about SASE, _____ connectivity, and because it’s centralized, it’s easier to implement and control, rather than having specialized devices or specialized software sitting on everybody’s laptop. If you’re a large enterprise, you can have thousands to many thousands of people working from home, so centralizing your security aspect with a high-speed connection is probably the way moving forward.

Mike Vizard: What impact does that have back on the core networks? I mean, if everybody has suddenly got more bandwidth at the edge, does that have a cascading effect up through the rest of the network infrastructure? Do I need additional core routers or networks? How will this play out in your mind?

Brian Lavallée: Yeah, _____. Content is created and consumed at the edge, and that’s from humans or machines ‒ think sensors, cameras, and so on. Definitely, as we open up the access ramps, 5G is a way of opening up the wireless access ramp, plus you have a variety of wireline broadband technologies for the home, even the offices. For sure, it has a knock-on effect into the core network because, if you think about it, most of the content and applications that you access, whether it’s on your smart phone, your laptop, tablet, or whatever device you’re using, it’s sitting on a datacenter somewhere. That could be a datacenter close to you, an edge datacenter maybe sitting in the same city as you. It could be a datacenter on the other side of the world sitting in Norway or Sweden or Finland.

Mike Vizard: I’m feeling like a lot of the applications that people are starting to build, consume data and analyze data at the point at where it’s created more, it’s more happening out at the edge. Is there a lot more sensitivity around latency? Will the laws of networking physics come back and bite us soon, and are we seeing new classes of applications?

Brian Lavallée: Yeah, and that’s a good question because it leans into the topic of discussion, one of the hotter ones today, which is edge computing or edge cloud. By putting storage and compute access closer to the end-users ‒ again, those end-users could be humans or machines ‒ it does inherently reduce the latency only because you have a shorter physical distance to travel. But, we have to remember latency is not just about distance, it’s about the amount of network nodes that are sitting between you and your content, so the more network devices you go through ‒ switches, routers, and so on ‒ the more latency is incurred and accumulated. Pushing the storage and compute closer to the edge with high-speed connections is a very good way of reducing latency, and it also opens up a whole bunch of new applications where latency is a critical factor beyond just increased speed.

Mike Vizard: I feel like right after the pandemic, we sent everybody home, we gave them their little wireless network, we hooked them up with a VPN, and they accessed the applications in the office. Then, they went out to the cloud and they had this kind of backhauling effect of the network traffic flow. You mentioned SASE. Are we going to change the way we think about how applications are deployed and how we access them, whether they’re up in the cloud? Is the routing concept that we’ve known all these years changing? How do you think we’re going to go forward from here?

Brian Lavallée: Yeah, so I think the increased adoption of cloud-based services is an unstoppable juggernaut. It is happening. There are security concerns. Our survey that we did, the study, we did show that there was a security concern of moving more and more into the cloud, but the cloud providers, the telecom operators, they’ve done a lot of excellent work over the last few years to address security, make increased security for their end-users. The cloud-based service providers, they realize that security is top of mind above everything else, so I think that will continue going forward. As more and more end-users are accessing cloud-based applications and content, the network becomes a critical factor because the network now is sitting in between you and the content you’re trying to access.

Mike Vizard: Do you think the bad guys have figured all this out and they’re targeting people at the end point now and looking at these slightly misconfigured networks? Sometimes we’re using fraud RDP protocols. Are they getting it? Are they after us?

Brian Lavallée: They’ll always be there, nefarious actors, bad actors, whatever you want to call them, so they will always be there, but the cloud providers, the telecom operators, they know this, so they are consistently upgrading, modernizing, and putting increased security capabilities into their network. It’s always going to be a battle of people who want to do not so good things with the network and content, and the people that actually run those datacenters and networks. I think that will continue going forward, it’s kind of human nature, but we’ve made great strides as an industry, and that’s across IT, telecom, and cloud-based services in terms of security. With more and more people from home, I mean right now we’re living in probably the largest working-from-home experiment of our species, and, if you think about it, the network, it has bent in a few places, right?

Most of the networks are running hotter, but it hasn’t broken. The Internet has not broken. A lot of people early on thought that if everybody moves home, it’s not that you have more capacity coming to the home, but the traffic patterns have changed. During the day, when a lot of people went to school and work, the traffic was to the schools and the head offices, but now the whole traffic pattern has moved into the home. Although a lot of people were concerned, I think the telecom operators deserve a lot of credit. The network has bent in a few places, but it hasn’t broken, and they’ve just learned a lot over the last year, and the networks will just be that much more reliable going forward.

Mike Vizard: Prior to the pandemic, the carriers were trying to encourage people to offload from the network and use WiFi. Now we have 5G, and people are talking about 6G and who knows what comes after that. Are we getting to a point where we can just run 5G into the home? Will that get cost-effective at some point and we won’t have all these WiFi routers, or will it always be some sort of a balance between 5G, 6G and WiFi and we have to figure it out?

Brian Lavallée: Yeah, I mean my personal opinion on that, I think it will always be a mix of technologies. I’ve been in this industry long enough to know that a technology network completely disappears, at least not in my career. It always has places of use. The big difference with 5G over 4G LTE and previous technologies, and even WiFi, previous generations of mobile technologies, it brings a lot of new benefits to the table, right? If we look at the faster speeds, this is what gets the most press coverage because that’s what people will see right away, but it also brings much lower latency. We’re talking order of magnitudes lower latency. It connects to a wider variety of connected devices, such as sensors and smart devices, and then we have this whole concept of network slicing, which can guarantee end-to-end performance.

You can guarantee the speed, latency, and resilience. We haven’t had that in the past with any mobile network generation, and, to a large extent, we haven’t even had that on the wireline technologies. So, 5G coming right into your home, if you think about it, if you’re a network operator, it’s much easier just to send a 5G hotspot, it could be a router, residential gateway to somebody’s home, they turn it on, they plug it in the wall, and it discovers the mobile network and then BOOM, you’ve got one-gigabit-per-second capacity right to your home. That is, I think, the way of the future in many cases, but there are some technical challenges that still need to be overcome. But, most operators are looking at 5G fixed wireless access to the residential, but even businesses as something in their toolkit going forward to compete with wireline technologies.

Mike Vizard: Do you think there is a rebellion coming among the end-users who are using this stuff? When you hear them, they’re kind of complaining a lot about the experience, so do you think that at some point they’re going to start picking what companies they want to work for based on what kind of work-from-home, work-from-anywhere experience they can get? Will that factor into hiring and retention more than it has in the past?

Brian Lavallée: Yeah. I mean, the network is critical infrastructure today, especially for the younger generation who were born and raised on having always-on connectivity. I think that will continue going forward. But, as more people work from home, I know we hear a lot of talk about the new normal, what ire the workplace, for instance, or the school systems going to look like after the pandemic subsides? We don’t really know. But, what I think we do know is more people will be working from home. I see it in my personal life, I see it with my colleagues, I see it with friends and family. Their companies are looking at implementing working from home at least a couple of days a week. There are a lot of benefits to the company to do this, as well, so the network will become that much more important.

I think people will shop around if the provider they have, the network provider, is not giving them the performance that they need, and it will become a highly competitive marketplace. I mean, it is pretty competitive now, but I think it will be that much more so going forward in terms of speed, latency, and availability, the three main things that you need with a high-speed connection to your home.

Mike Vizard: I might even one day factor networking costs into my compensation packet, but we’ll see what happens.

Brian Lavallée: There you go.

Mike Vizard: What is your best advice to people who are on the IT side of this equation, they’re the network managers, they’re security people. What is the conversation that they should be having higher up the food chain to the board of directors about what needs to happen to really enable this next wave of work?

Brian Lavallée: Yeah, I mean, the higher speed to the home is something that a lot of people talk about. We just talked about it earlier, more people working, playing, and learning from home, so capacity will always be at the forefront of discussions and having competitiveness for an operator with another operator. But, I think increased security features, this could be wrapped around a broadband connection to your home, so rather than just buying a high-speed pipe to your service provider, they would also provide you with SASE type of capabilities, so security, firewalls, and so on. I think looking at bundling, IT services with the connection is definitely the way to go. It is a differentiator, and it is what people will be looking for because if you’re the average home user, whether it’s for playing, learning, or working, you really don’t care what technology you’re buying, as long as it’s cost-effective and it meets your needs.

I think my advice for the operators is to look at what is available recently, some of the newer technologies ‒ virtualization and SASE technologies and the SD _____, a whole bunch of other technologies ‒ and start to wrap these into a bundle that you solve most, if not all of the needs of the subscriber, and then you could differentiate far above and beyond just a higher speed.

Mike Vizard: All right, guys, you heard it here. Faster things that are insecure do not necessarily represent progress. Hey, Brian, thanks for being on the show.

Brian Lavallée: Happy to be here. Thank you.

Mike Vizard: All right, guys, back to you in the studio.

[End of Audio]

March 22, 2022
Apple Outage Outrage | Linux Random Redo | Okta Hacked (or Not)

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Why Apple services were down, Linux gets a huge RNG overhaul, and we wonder if Okta was hacked again. (more…)

March 22, 2022
Secure Software Summit: Behold the SBOM

With supply chain security becoming more of a focus, the SBOM is now viewed as a critical element in shoring up supply chain security. SBOM stands for software bill of materials. At a very elementary level, an SBOM is a list of ingredients. Think about how every food product in the supermarket lists ingredients so, for example, people who are allergic to nuts can avoid nuts. An SBOM does the same thing for software applications.

If you want to do business with the U.S. government, SBOMs are no longer optional. Executive order 14028 requires an SBOM for all U.S. federal software procurements. Why the push for SBOMs now? Software companies generally do not list all the components included in the software they produce. Knowing what components are included in applications is essential for proactive vulnerability management. That said, the government mandate for SBOMs only requires a minimal amount of included information—supplier name, the component name and version, dependency relationships, time stamps for versions or modification and information about who authored the SBOM itself.

What to Look For in an SBOM

This minimum required data yields minimal benefit for cybersecurity risk reduction and vulnerability management. SBOMs can actually include many other types of information. What information should you include in SBOMs for software applications produced, purchased or consumed by your organization? Here is a list.

Authenticity —Authenticity asks and verifies that someone actually created a listed component. Ideally, this requires a digital signature of some sort. Authenticity can be particularly important in the case of nested components where the supplier of an application includes multiple other components, as is common with JavaScript and Java libraries used for common functionalities.

Integrity —This addresses whether a component has been tampered with or modified without authorization. An SBOM should include some measure of integrity. Equally important is proving the integrity of the SBOM itself. Just as CheckSum can tell you whether software you are loading matches what the source of that software indicates, integrity checks should allow you to verify that components on an SBOM (and the SBOM itself) are exactly what you are expecting.

Services —An inventory of services called by an application is critical. Modern software does not live in a vacuum. It calls out to the Internet to perform all sorts of tasks and communication with other systems. Those services that extend beyond your application are part of the inventory because they can be an indication of vulnerability or indicate a larger problem.

Vulnerability Remediation Status —Development teams are constantly balancing the need to remediate vulnerabilities against the risk that remediation will break the application. Some open source projects, for example, have a history of not backporting security fixes. The latest version of the project may break applications. Organizations are forced to choose between breaking an application or taking the risk of introducing a high-severity vulnerability that might be exploited. Responsible organizations that opt to keep the older version and not break their software will backport their own fixes. An SBOM should describe these variations and any other vulnerability remediation activity.

Inventory of component security advisories —An SBOM should describe security advisories pending against all components. This is likely to be a dynamic list that you will need to automatically update using one of the accepted vulnerability advisory formats (CSAF and CycloneDX, to name two). In fact, this is one of the strengths of SBOMs—they are designed for automation and to act programmatically.

Completeness —It is not always possible to ship complete SBOMs, either for technical or intellectual property reasons. An SBOM should indicate whether it is complete and whether it contains a comprehensive inventory of dependencies and services.

Pedigree —This information describes the DNA of the SBOM and the original makeup and changelog of each particular component. An SBOM should describe the original component, exactly what changes were made, who committed the changes and why, what the differences were between different versions and what security issues were fixed with each variation and version.

Provenance —This relates to the origin of a component or service. It can describe a repository that the component was downloaded from, a country of origin, the name of a software manufacturer or open source project or other information indicating the component or service’s source of origin.

License type —License information is relevant for compliance and to ensure that your organization is not consuming or producing software in a manner that violates licensing terms. Moreso than for security and vulnerability, this information is for risk management, compliance enforcement and auditing.

Customizing SBOMs to Meet Your Needs

The SBOM is a flexible and highly adaptable approach that can be easily customized to fit specific use cases. You can design SBOMs to meet the specific needs of your use case, either as a software provider or consumer. You also may need to modify your SBOM fields and data design as your needs evolve. Fortunately, it is rapidly becoming easier to automatically generate SBOMs using software security tools such as software composition analysis (SCA) and interactive application security testing (IAST) solutions. SBOMs can also be created or read automatically at other points during the build life cycle along the continuous integration/continuous deployment (CI/CD) process. Just as you would not buy a box of cereal that did not list the ingredients, in the future you will not want to consume software that does not have a detailed SBOM. What you don’t know about the software you run can hurt you, and SBOMs will be a crucial tool in providing greater transparency and observability into the software supply chain.

About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay

March 22, 2022
Secure Software Summit: Reachability and Risk for Security Leaders
It is impossible to manage security posture without considering two key factors in any potential vulnerability or security flaw: Reachability and risk. The two factors are related. Reachability defines the degree to which a given security vulnerability that is detected, such as a CVE, can actually be attacked and exploited to gain privileged access and directly or indirectly access critical systems or data. Risk is a business measurement that assesses the potential for a vulnerability to actually damage an enterprise or organization. In general, without reachability, there is less risk.

How to Think About Reachability

Reachability and risk have dramatically shifted in recent years as we have moved from a hard perimeter “fortress” approach to security to more distributed systems, APIs, cloud computing, SaaS and far more potential points for attack ingress and exfiltration egress. To mitigate this new risk landscape, we have actually moved to fortify all the newly exposed internal elements and the critical systems of security. We have built a new perimeter around identity, with authentication and zero-trust. We have built perimeters around connected devices, around our cloud infrastructure and our virtualized environments and our configuration of all these systems. At every point where there is a potential for risk, you need to think about fortifying the new mini-perimeters.

A key aspect of risk and reachability that we have spent less effort on is the human aspect. We do not yet consider people as a perimeter to be defended, even though people comprise the most reachable vulnerability. As people, we create flawed software. We misconfigure our infrastructure, our security tools and our authentication systems. A key part of the new mindset of reachability must be better accounting for and mitigating the risks of people. This means simplifying their decision trees, either through better systems or better design, and by helping them become a better gray-matter perimeter.

Due to the atomization of our IT and application environment, and the resulting miniaturization of our security fortresses, we have created a far more granular surface. In some cases, such as through segmentation, we have enhanced security and reduced risk; in other cases, we have added complexity. Without a doubt, we now have many more points of contact and a broader attack surface exposed to the world. A critical consideration when assessing reachability is how far each of those points can be exploited, and to what breadth and depth—looking at horizontal traversal and immediate compromise of critical systems or subverting supply chain and other pivotal subsystems for subsequent attacks. The miniature fortress mentality is a useful framework for thinking through the potential attack surface and putting in place the right controls, at the right places, to address reachability.

Risk Scoring and Reachability

So what is the interplay between risk scoring and reachability? We cannot say no to new technologies that introduce more complexity, and potentially more risk, if our end users are demanding them. We need to better manage how risk scores are created and applied to help people perform better in this environment of greater complexity.

Today, the responsibility for understanding the complete security picture and applying the proper context to decision-making and judgments has grown more widespread. Everyone who is deploying, using, building or designing applications and technology environments must be constantly thinking about the security implications of whatever project or code they are working on. The lens for those implications is not only at the application or system or infrastructure level but for the general well-being of the business. This means a broader understanding of risk scoring driven by reachability and criticality and taking into better account the implications of reachability and attackability.

In cybersecurity, there is no shortage of risk measurements. Injecting reachability into the scoring process is a way of framing vulnerabilities to prioritize which of these issues are likely to be an actual risk. It is possible to have something that comprises a major business risk but have a very low reachability context, which may lead a security team to treat that vulnerability differently.

Broadly speaking, however, it is insufficient to make decisions solely based on programmatically generated risk scores. Any risk score is a reflection of the system generating the score, and how that system was tuned, either by the vendor or by your team. It is important to always remember that any risk score involves some subjective judgments. This is necessary, but injects opportunities for drift and bias that may skew scores in ways that leave you less secure.

To be clear, risk scores serve a crucial purpose in application security. Risk scores help you reduce the noise and find signals amidst the flood of CVEs and vulnerabilities and alerts. In fact, there is today so much noise around security that any effective team relies on multiple layers of noise reduction in their decision chain and prioritization and policy efforts. Where risk scores fail is with the inability to map accurately to business risk or by inaccurately reflecting the reachability of any CVE or other reported issue. This requires security teams to build their own internal and mental models of risk, modeling threats against the most critical systems and creating context that can be used to verify decisions by the risk scoring systems. Knowing how data flows through applications and infrastructure is crucial if a human agent can quickly judge the reachability and attackability of any given vulnerability presented or recognize when the automated risk scoring engines may not accurately reflect business risks.

In effect, there are two different processes at work: Calculating risk and contemplating risk. Calculating risk is programmatic and automated; human-guided, initially, but driven by models applied against the mass of vulnerabilities surfaced by security testing tools like software composition analysis, static application security testing, fuzzing and linting. Contemplating risk is a more holistic exercise that leverages the human capacity for intuition and innate knowledge of systems driven by experience and hard-won insights.

Contemplation is what will help security teams identify undetected biases, guide threat modeling to better consider drift and highlight areas where human breakdowns might have augmented risk in ways that a risk scoring system could never capture. Above all, contemplation is about context. The more context a human actor has, the better they can understand the big picture. The greater the understanding of the big picture, the better decisions people make. This must also flow through and inform the models and the risk scoring engines so that the value of contemplation is captured in the calculation.

Using Risk Scoring Tools to Minimize Reachability and Attackability

A security team must constantly measure the efficiency and efficacy of all security tooling. The general rule is that security controls lose efficacy over time as vendors lose focus or fail to account properly for emerging classes of threats. When you have a security tool that generates risk scores in tune with your reachability guidelines and proves accurate, you should elevate that system to a higher level of trust and give more weight to its findings. That said, even tools that generate risks you think are accurate and perform well must be constantly evaluated and cross-checked.

Unfortunately, it only takes a small amount of drift to open up an organization to potentially catastrophic attacks. A critical part of this calculation is ensuring that humans stay on top of how the risk scores are created and continue to influence the scores to fight drift. Drift is natural but it is not inevitable. Your code and environment are constantly changing, This may result in changes in reachability which can materially impact your overall risk posture and should alter the way you approach different systems and risks. An automated risk scoring system may not understand that a new type of supply chain attack might result in subsequent related attacks on similar components in other operating systems, to name one example. Human intelligence and contemplation, really, is the secret ingredient that makes risk scoring tools far more valuable and reliable.

Takeaways: Atomized Systems, Reachability and Risk and Humans

The environment and application landscape will continue to atomize further as we move into new infrastructure paradigms like edge computing and all systems on the network gain capacity. For this landscape, security teams and the developers they work with on shifting left should:
- Recognize that we have shifted from a global perimeter to an atomized constellation of mini-perimeters and shift their mindset to consider security and building fortresses at the micro level as a key way to reduce reachability and risk
- Developers and security teams need to assess the systems they build and protect on the basis of reachability and risk to properly prioritize. Risk scoring tools are useful elements for reducing noise but cannot supplant human intelligence to gut-check risk decisions
- All along the way, humans have to continue to guide and inform these risk scoring systems and ensure that risk scores do not become blind proxies for criticality and importance. Allowing automated risk scoring to dominate the decision process can make security drift more dangerous and allow human error to go unrecognized, leaving applications and infrastructure both reachable and at risk.
This article was co-authored by Bryan Smith, CTO, RiskLens and Rob Lundy, director of product marketing, ShiftLeft.

About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay
March 21, 2022
Secure Software Summit: Measuring and Mitigating OSS Risks

Measuring and mitigating the security risks in open source software is becoming a major issue in the software development community. Attacks on open source software (OSS) are on the rise; open source supply chain attacks grew 650% in 2021—a staggering number.

2021 and early 2022 saw significant attacks which impacted more than 1,000 organizations. The most prominent recent OSS attack was the Log4Shell vulnerability disclosed in December 2021. Attacks on the OSS supply chain and on OSS impacted nearly every single software organization on the planet. Everybody is scrambling to continuously patch systems as new vulnerabilities continue pouring in.

To properly navigate this environment and keep our systems safe, we as a community need to understand the security risks of OSS and be better prepared to handle them, collectively and individually.

Open Source Risks

There are a number of risks that make OSS more vulnerable without proper security measures.

OSS is Free; OSS Security is Expensive

To start, we must recognize that open source software may be free to download and require no payment, but it comes with a massive security cost.

A significant portion of open source software is developed by volunteers in their free time. Most of the prominent open source volunteers actually have full-time jobs, and the code they contribute to projects like the Log4j logging libraries are written on nights and weekends; on their own time and not on the company dime. Not surprisingly, security isn’t high on their priority list. They don’t have the resources to invest in a secure development life cycle or have the time and bandwidth to maintain their code. This may mean that security vulnerabilities remain unaddressed for long periods of time, sometimes indefinitely.

All Bugs are Shallow, but There Aren’t Enough Eyeballs

In software development, Linus’s law states “…given enough eyeballs, all bugs are shallow.” However, this is only true if those eyes are looking in the right place. In the case of open source, all code is open but there aren’t enough eyes paying attention to it to spot all the bugs. In addition, organizations may not consider the security consequences of their decision to consume a particular piece of open source software and adding it as a dependency. Dependency trees become more and more complex; harder and harder to monitor and secure, as you keep adding more third-party components to a core codebase. Each new dependency might bring additional nested dependencies—perhaps hundreds or thousands, depending on the size of the library added or the role it plays in the application architecture. For example, the Kubernetes project—one of the most successful and widely adopted OSS projects—has several thousand dependencies, many of which have additional secondary and tertiary downstream dependencies.

So Much Code. So Many Choices. Not all are Good.

The beauty of the open source ecosystem is that there is a solution for everything. The problem with the open source ecosystem is that there’s not just one solution, but several dozen. For example, if you need a JSON parser for your application, there are several to choose from. If your application is running a critical service or infrastructure, then picking the right JSON parser—one that is secure and comes from a responsible project or maintainer—becomes very important. A JSON parser will serve as a critical component for your application, so it will impact the security of your entire system. If you chose the wrong JSON parser, you might pay a big price down the road in security risk and subsequent breaches.

Proposal: An Entry Gate Security Risk Assessment

One potential solution to enhance OSS code security is the creation and implementation of a security risk assessment gate as part of your OSS ingestion pipeline. Putting this process in place will help you understand whether consuming a certain piece of software is high risk and avoid software that does not meet your risk criteria.

Here is an example of how this might work. Consider five different JSON parser libraries in GitHub, all of which have a high number of stars and appear to be highly rated. How can we assess the risk of these libraries and quickly and easily eliminate ones that are too risky? One of these JSON libraries is hosted under a user account. Most of the commits are done by a single developer without reviews. This alone indicates that using this library in a critical system presents a high risk. This lone developer could commit malicious code without review. The maintainer’s token access could be leaked, allowing attackers to insert malicious code. The developer might also turn over committer control of their library to a malicious actor without notifying users (who might not pay attention to the notification anyway). Identifying this risk required no more than a few minutes of review of the repository and cost little in time or effort.

A Broader View of Security Posture for OSS

More broadly, we need to understand the full security posture of an organization or developer and their open source code prior to adding it to critical systems. Because OSS is a complex ecosystem and many pieces of OSS are complex, the risks they generate are similarly complex. Our traditional, myopic view of these risks is that we just need to find all the security vulnerabilities in our codebase and fix them. This does not resolve all risks and may lull us into a false sense of security.

We need to understand the full comprehensive security posture of an open source project. That involves understanding a broad range of risks beyond code vulnerabilities. Here are some of the key risks to consider beyond ensuring that your codebase is well-secured and patched.

Supply Chain Attacks on the Build Pipeline

Attacks like SolarWinds and CodeCove taught us that supply chain risk is a very big deal and that tools are just as much at risk as code. Attackers have found it much easier and much more reliable to attack build pipelines, rather than finding vulnerabilities in code.

Security Risks Revealed by Continuous Security Testing

Continuous security testing is critical to maintaining the security of a codebase. There might not be code vulnerabilities today, but as developers continuously modify code and iterate on applications, vulnerabilities may be introduced. This can mean testing code as part of the CI/CD process with every new code push. For example, we have found numerous critical bugs in OpenSSL libraries. All were caught on a master or main branch and never made it to a production release.

Project Maintenance and Capabilities

One of the really big risks with open source is the quality and frequency of project maintenance. Issues raised may not be addressed in a timely fashion. Maintainers of a project may have a long backlog of issues, indicating that they are not actively maintaining a project. It is not uncommon for critical open source projects used in millions of applications and comprising a significant dependency to the global software supply chain to not have been updated in a year or more. In addition, maintainers of OSS projects may not know how to respond to security incidents and fix security vulnerabilities. This is particularly true if the maintainers do not work at a large organization, do not have cybersecurity or secure development experience or have never suffered through the toil and trauma of addressing a major vulnerability in their code on short notice.

Tools for Measuring and Addressing Supply Chain Risk

A number of efforts are underway to create free tools to help developers and application security teams better measure and address supply chain risk programmatically and analytically.

SLSA—Supply Chain Level for Software Artifacts

As a tool to help organizations better address software supply chain risks, Google created a novel framework that we call Supply Chain Level for Software Artifacts (SLSA). The project has been underway at Google for more than a decade and has been open sourced as part of the OSSF. Contributors include VMware, Citibank and Datadog, among others. Google hopes to standardize SLSA and make it an industry benchmark for evaluating supply chain risk. Right now, SLSA is focused primarily on build process risks but the hope is to expand it in the near future to other areas, such as vulnerability and maintainer risks.

OSS Security Scorecards Project

In order to scale the process of assessing, identifying and mitigating risks in OSS, we need robust automation. In 2020, as part of the OSSF, Google released a new project called Open Source Security Scorecard. The project aimed to automate the analysis and trust aspect when evaluating the security posture of open source code. Further, the Scorecards can provide specific security recommendations. The Scorecard process can incorporate many other automated tools, including checking against vulnerabilities using the Open Source Vulnerability (OSV) database and running dependency checks using open source dependency trackers like Dependabot. The scorecards project also makes it easier to report new vulnerabilities and get those to the right people in a project.

Conclusion: What is not Measured is not Mitigated

No one denies that OSS has become crucial to the vast majority of applications running the world’s technology foundation. The span of OSS continues to grow, as more and more languages and frameworks emerge and specialization increases. No one wants to reinvent the wheel and rewrite fresh code to perform the same tasks with every new application. The model of reuse, dependency and distributed software development works and speeds innovation. That said, the open source community and the supply chain used to build OSS need a more rigorous approach to reducing risk. Given the scope of the problem, automation and standardization are helping to scale toward the stated goal. We are in the early stages of creating the tools and infrastructure to automate the risk identification, measurement, assessment and mitigation processes. Security frameworks like SLSA and new processes like entry gating new OSS code with security scorecards will direct organizations down that path and move us toward a more secure open source future.

About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay

March 17, 2022
Secure Software Summit: Securing Software With Zero-Trust
With the increase of supply chain attacks on everything from logging software like Log4j to takeovers of important JavaScript packages to compromises of network utility tools like SolarWinds, more and more organizations are recognizing the need to adopt a zero-trust mindset. Zero-trust can improve security, reduce risks and give organizations greater confidence in the integrity of their IT infrastructure and applications. To correctly set up a zero-trust architecture, you need to understand what it actually takes to make systems zero-trust.

Zero-Trust Definition and Guiding Principles

Due to the rise of interest in zero-trust, many misconceptions have built up that bear correcting.
- Zero-trust is not a product or tool. In fact, there is no single tool you can apply to an environment to configure a full-stack zero-trust implementation. Rather, zero-trust is more about creating and refining an architecture and applying a consistent mindset.
- Zero-trust is a never-ending journey. It is not a destination. You are never done; never finished setting up zero-trust, because software applications and IT environments continuously evolve. As your environment evolves, part of applying zero-trust is going back and reassessing everything you are doing on a regular basis to ensure it complies with the principles of zero-trust and covers all your software and technology assets.
- Zero-trust does not mean your environment cannot be hacked or breached. On the contrary, under zero-trust you constantly assume your environment has been breached. Zero-trust emphasizes building defensive layers that reduce the potential impact of any hack or breach and protects your assets and data.
So, what is zero-trust? Zero-trust is a framework and approach to security that assumes an organization’s IT assets — software, networks, data — are constantly at risk and must be secured through continuous verification of the identity, and the application of least privilege principles against any identity — human or machine — operating inside the IT environment. The core of zero-trust is the lack of inherent trust for any identity as a managing approach to building more resilient applications and segmented risk.

Zero-trust has several key guiding principles.
1. Verify all identities, all the time, using a strong form of authentication — not just username and password. Specifically, use strong forms of authentication, such as multifactor authentication. This principle applies not only to human users but also to all devices, bots, services, APIs and partner systems accessing your environments. They must be verified every time they seek access and continuously while they are active inside your environment.
2. Always assume your environment has been breached. From this starting point, design your systems to reduce the blast radius and protect your assets as if an attacker is continuously present and moving within your environment to perform malicious acts — data exfiltration, system takeover, or other forms of abuse.
3. Always enforce least privilege for everyone and everything operating in your environment. Least privilege means limiting access only to systems, applications, data and services required for each task or interaction. To properly apply least privilege requires an understanding of expected behavior for applications and services, what they should and should not be doing, and also creating a sense of context. Applying least privilege with context is also dependent on assessing the security posture of any device, service, application, API or identity seeking access. Your zero-trust policy enforcement engines should always interrogate inbound and ongoing connections to understand where an identity is coming from, what it is trying to connect to and the current security posture of that identity. The answers to these questions should inform and shape least privilege permissions and provide additional insight into the risk of providing access.
4. Segment your environment to limit horizontal traversals or pass-through attacks. This is closely related to least privilege. The bottom line: Do not award any identities carte blanche access to your network, applications data or infrastructure. Only allow identities access to the portions they truly need to perform their task, and only while they are performing that task.
Using Zero-Trust to Secure Software and Software Supply Chain

The conceptual language of zero-trust makes it sounds like an identity architecture and network security approach, floating above developers. For building and deploying secure applications, developers and DevOps teams must also adopt a zero-trust mindset. In fact, there are clear synergies between zero-trust and application security — and particularly with regard to open source software (OSS). OSS developers may not spend sufficient time and energy on securing their code because it is not what they like about writing software. For this reason, if you are including OSS code (and nested OSS libraries) in your applications and environments, then you have even more reason to apply zero-trust frameworks.

A useful way to think about this is to consider zero-trust as applicable from the inside out, from the outside in and to everything in between linking internal and external, applications and APIs, code and integration/deployment pipelines and partner environments. Properly executed, zero-trust allows your developers to go faster but also be safer. Additionally, it is useful to view zero-trust through the confidentiality, integrity, availability (CIA) lens. You should design your zero-trust architecture to deliver these elements in a priority hierarchy that maps to your business needs. There are a few different ways to implement zero-trust principles in the software supply chain.

Assume Your Code has Been Breached and is Insecure

Conduct a software asset inventory, looking at and categorizing all code running in your environment. This makes it easier for you to identify unauthorized changes and build more resilient applications. Set up mechanisms to detect unauthorized changes or anomalous software behaviors. You may want to apply these guidelines to close partners and their software and supply chain, as well. Remember that attacks may be from you, through you, or at you.

Manage Identities Carefully

Analyze and categorize user entitlements to set a baseline for least privilege. Conduct an inventory of all identities active in your environment to identify known identities (and quickly recognize unknown identities). Set up mechanisms to analyze the behavior of identities to spot anomalies and create a baseline for context and behavioral privilege and access policy decisions. Include partner identities in this inventory: Your greatest risk may come from unsecured partners that attackers use to gain access to your environment. The inventory should also include service and admin accounts. The inclination is to set their privileges and forget about them, often awarding broad access so as not to slow them down and also potentially providing an unmonitored backdoor to attackers.

Take Inventory and Lock Down Integrations

Inventory your external and internal supply chain. This includes all dependencies in the code, services called, APIs exposed or accessed and tooling for developing, hosting, integrating and building applications. For example, you may want to analyze your CI/CD pipeline and deployment environments through the lens of segmentation and least privilege to limit access depending on the pipeline stage. Put in place measures to segment the software environment and prevent lateral movement, like application sandboxing (but make sure to also tightly secure your sandboxes).

Code Assessment

Developers need to apply threat models to their code and adopt the mindset of considering what an attacker might access and what they might do should they compromise an application or the surrounding supply chain. Obviously, for all code — and especially OSS code – developers will need to continuously look for vulnerabilities and scan code using tools like static code analysis and dynamic application security testing. Zero-trust means continuous verification of your own code, not just the entities using the code.

Zero-Trust Takeaways

As we continue towards a future of more complex and integrated application environments and more usage of OSS code for core application functionality, the zero-trust future becomes zero-trust forever. It will simply become a way of life in application development. That said, we should keep in mind four key guiding principles.

Know Your Current State—Be aware of where your environment stands and your security posture, and direct and indirect impacts of security vulnerabilities.

Assume Breach—Be able to identify anomalous behavior and continuously consider what could be done by an attacker if your app was used maliciously?

Assess—Continuously assess the status and changes in the environment. Ask questions like: Who has what, when and where? Where do your integrated partners and vendors stand in terms of security?

Continuous Improvement—Periodically assess and compare your zero-trust efforts against desired security posture and make improvements that align with objectives around confidentiality, integrity, and availability. Constantly guard against risk.

These four principles map to five key takeaways that any developer, application security team, DevSecOps team or platform team can apply today and forever.
- Review your environment with a zero-trust lens
- Recertify all identities continuously
- Establish baseline app and identity behavior
- Focus on detection and prevention of attacks
- Design applications for resilience
Following these guiding principles and basic framings will ensure your zero-trust journey creates strong defenses that help protect your development supply chain from the outside in, inside out, and with integrated partners.

The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay
March 16, 2022
What Does AIOps Mean for SREs?
It seems SREs are of two minds when it comes to AIOps. On one hand, AIOps’ potential is pretty exciting. By automating complex workflows and troubleshooting processes, AIOps could make your life as an SRE much easier.

But on the other hand, some SREs may choose to view AIOps with disdain and distrust. They might think of AIOps as just another trendy buzzword that doesn’t live up to the hype and that can become a distraction from the SRE tools that really matter.

Which perspective is right? Should SREs embrace AIOps with open arms, or should they resist marketers’ efforts to position AIOps as the latest, greatest tooling innovation in the IT industry?

Those are subjective questions that we can’t answer definitively, but let’s at least gain some perspective by examining what AIOps means for SREs.

What is AIOps?

If you keep up with IT buzzwords, you’ve probably at least heard the term AIOps—which is short for artificial intelligence for IT operations—and that it refers to the use of AI and machine learning (ML) to help automate IT operations workflows.

The big idea behind AIOps is that, by using AI and ML to perform advanced analysis of large volumes of data from IT systems, IT and SRE teams can solve complex problems more efficiently than they could using a traditional, manual approach.

AIOps can, for example, help to surface the root cause of a performance issue in a complex, multilayered environment like Kubernetes. Or it could make recommendations about how best to resolve an incident.

AIOps entered the IT lexicon in 2016, when Gartner coined the term. Now, six years later, it’s a relatively well-established tool domain.

How SREs View AIOps

Despite the fact that AIOps has been around for some time, it doesn’t yet appear that many SREs have bought into the AIOps revolution. Catchpoint found in a 2021 survey that just 7.5% of SREs reported that AIOps tools delivered high value to their organizations.

It’s unclear exactly why there are such low rates of excitement about AIOps, but we’d speculate that there are a few key factors at play:
- AIOps is a new term for an old idea: Many monitoring and observability tools have included at least basic AI and ML analytics features for a long time, even before tool vendors slapped the AIOps label on their products. SREs probably realize this and, to some extent, view AIOps as an effort by marketers to rebrand functionality that is not actually new.
- AIOps is hard to implement: Setting up an AIOps tool requires integrating it with diverse data sources and customizing it to fit your workflows and environment. It’s possible some SREs view this setup work as requiring more effort than it’s worth.
- AIOps can’t replace human insight: While AIOps tools may be useful to a point, it would be unwise to place blind trust in AIOps-based analyses or recommendations. For this reason, some SREs may believe that AIOps encourages organizations to rely too heavily on automated tools at the cost of the expert analysis and perspective that only SREs can provide. (This is similar to the fact that it sometimes makes sense to prioritize human intuition and expertise over playbooks.)
From an SRE’s perspective, then, AIOps may appear over-hyped, overly complicated and underperforming compared to traditional approaches to SRE.

What SREs can Gain From AIOps

SREs’ wariness toward AIOps is valid—but only to a point. It’s important not to let suspicions about the limitations of AIOps turn into excuses not to use AIOps at all. AIOps has some value to offer to SREs, even if it’s not perfect.

For example, AIOps can play a role in reducing toil. To the extent that AIOps tools can recognize complex patterns or discover interrelated data sets more quickly than human engineers, AIOps reduces the time SREs have to spend manually troubleshooting problems or poring over complicated information.

AIOps also helps to enable a more proactive approach to monitoring and incident management. If AIOps tools can alert SREs to emerging issues before they would otherwise recognize them, AIOps can help the SREs to get in front of the problems before they turn into true incidents. That’s better for SREs and end users alike.

There is also an argument to be made that AIOps can help SREs do more with fewer engineering resources. If you can use AI to automate some aspects of monitoring and incident response, you can maintain the same levels of availability and performance with fewer human engineers on hand.

Conclusion: AI Won’t Replace SREs—But it can Help

None of the above is to say that AIOps can replace SREs, or that it magically solves every problem SREs face. Anyone who believes AIOps is a silver bullet has bought into the marketing hype to an unhealthy degree.

Nonetheless, AIOps tools do offer value to SREs by making their jobs easier in some respects and improving reliability outcomes.

So, while it’s wise to maintain a healthy perspective about the limitations of AIOps, SREs shouldn’t rule out AIOps tools as one way to improve reliability engineering.
March 15, 2022
Secure Software Summit: The State of OSS Supply Chain Security

The open source software (OSS) supply chain is under attack. As evidenced by the recent Log4Shell vulnerability, the OSS supply chain is increasingly a focus for attackers seeking to exploit weak links in security. A number of research reports have recorded a significant increase in so-called ‘next-gen software supply chain attacks‘ over the past decade. These attacks look to compromise key links in the software supply chain including source code, build systems, CI/CD tools and code testing tools, among others. Next-gen attacks are happening across all languages, all types of software and are increasing in their frequency. Common forms include typosquatting, malicious code injection and tool tampering (like the CodeCov incident). In 2021, 64% of organizations reported they were affected by software supply chain attacks, according to container security company Anchore.

The increasing wave of next-gen attacks has driven new regulations mandating specific steps to inventory and make more transparent the content of software applications and the software supply chain. Specifically, executive order 14082, issued by U.S. president Joseph Biden, mandated a long list of security practices, including the inclusion of a software bill of materials (SBOM) with every application run by the U.S. federal government. The National Institute of Standards and Technology (NIST) is creating reference architectures and templates for application security as a result of the executive order. In other words, software security regulations will likely impact every company that produces code or sells products running on code (including SaaS products).

To be clear, some of the problems experienced by the OSS software supply chain are not exclusive to open source. The recent supply chain attacks on Microsoft Exchange Server and the SunBurst attack on the SolarWinds network management tool targeted proprietary code bases. That said, OSS is produced and maintained very differently than proprietary software. This does result in some unique risks. Here, we will cover those risks as well as some promising initiatives underway to address them.

Why Is the OSS Supply Chain A Juicy Target?

There are two primary reasons OSS supply chains are a tempting target. One is its omnipresence and the other is its weakness. It is not an exaggeration to say that OSS is omnipresent. According to Sonatype, 98% of all codebases contain OSS. This is likely an underrepresentation. You simply cannot build code from scratch today without using OSS components. It is too labor-intensive and expensive. The ubiquity of OSS makes the supply chain that creates, distributed and manages it an obvious target. At the same time, that widespread use translates into an increasing burden on companies to maintain, update and patch all OSS dependencies in their applications.

This is an impossible task; it is far easier to insert OSS code to solve an immediate problem than to, over time, make sure that code is updated and not vulnerable. For this reason, Sonatype estimated that 92% of all applications contain outdated or vulnerable code. We have seen the impact of this inability of organizations to maintain their OSS code through attacks like the one perpetrated against Apache Struts software run by credit reporting company Equifax, a breach that exposed the sensitive personal and financial information of hundreds of millions of Americans.

In recognizing that OSS is a fruitful channel for attacks, attackers have focused more specifically on the supply chain because it is often the weakest link. Elements of the supply chain reside in the gap between responsible organizations and may be less strictly policed or less actively secured. Continuous integration/continuous deployment (CI/CD) tools, OSS libraries that can easily access other systems, package managers and open source developer tooling are all components of the supply chain that are widely used but which are often the sole responsibility of many part-time maintainers who write and maintain code as an unpaid avocation.

OSS-Specific Challenges

The three key benefits of OSS each create unique security and supply chain challenges. Specifically, OSS is open, free and transparent.

The Two Sides of Open

Linus Torvalds said that many eyeballs make all bugs shallow. To a degree, this is true. But what if those same eyes are looking for bugs to exploit rather than fix? Not everyone on the internet is nice and many seek to exploit the fact that anyone with a GitHub account can contribute code to critical libraries. This attracts a lot of great contributors, but also a lot of bad actors and bad behavior.

That behavior is often hidden or delayed. For example, we often see bad actors ask to contribute to projects or libraries run by single maintainers that are relatively popular and used by big applications. Often, maintainers struggle to keep up with the demands of these projects. A bad actor will offer to help, submit a handful of good PRs and gain commit privileges to the repository. At this point, the bad actor will then stealthily insert malware, such as a cryptojacking script.

Free Puppy. Not Free Beer.

Free is a good price. There are no purchase orders. This makes it super easy to use. Free also serves as a great distribution strategy. However, sometimes free is not worth it. If an organization depends on an application and it becomes critical, they might require fast patching, better compliance and official support relationships. Free OSS comes with no guarantee, and this can be a major liability. An application’s maintainers might be writing the code and applying fixes in their spare time on weekends because they already have a full-time job. In fact, research has shown that a significant percentage of the critical software powering the internet is maintained by volunteers; this was the cause of the Heartbleed vulnerability in the OpenSSH encryption libraries.

Free code also may not be updated for years. If a vulnerability is disclosed, you may not be able to get permission to merge fixes you write. If the code is abandoned and the author no longer checks comments or requests, then you are stuck and may have no choice but to fork the project. A hard fork on short notice can cause chaos and its own problems. This is what happened with a popular Golang library called JWT Go, which made its way into dependency trees across the Golang ecosystem. JWT Go was even included in Kubernetes, one of the most popular and important open source projects. When a vulnerability to JWT Go emerged, everyone realized the maintainer had abandoned it. Multiple forked versions emerged with different patches for the same bug. Chaos ensued.

It is a common refrain to ask companies to fund volunteer maintainers. There are numerous programs for this, including GitHub Sponsors, Open Collective and Tidelift. These programs are helpful but not sufficient. You can’t just hand people a small amount of money and get more secure code and professional-grade maintenance. Even if you offered full-time salaries, it might not translate into well-supported, secure code.

The Illusory Benefit of Transparency

The transparency of OSS is real, but the idea that this code is getting audited and reviewed constantly is an illusion. Yes, the source code is distributed and open. But the sheer quantity makes it impossible to review all the code you use. If you are using millions and tens of millions of lines of code, you don’t have time or sufficient eyeballs. Even if you are spending considerable resources on code review, it is still easy to miss problems and vulnerabilities and relatively simple for malicious contributors to projects to hide malware or other bad inclusions. Reviewing the source code also does eliminate all risks. When most OSS code is consumed, it is running as a package compiled by another entity or developer.

Further, there is no guarantee that the contents of the package comes from the same source code you audited. We have actually seen this attack often with package repositories like npm and Maven, where malicious actors insert tampered builds or take over packages and insert code that is not listed as part of the package.

The CodeCov incident is one of the more recent and egregious examples. This free service for checking the code coverage of unit tests used BASH script running inside of a CI server for installation. The script was compromised and attackers inserted malware that could steal code secrets and export them. Those secrets might be tokens used to publish packages in npm or Maven and exposing those secrets allowed attackers to then compromise those packages. In another example, a PHP server managing source code distribution for the PHP project was compromised and multiple commits injected malicious code into the source code repository. The incident was caught quickly, but it demonstrated how attackers could tamper with source code directly, bypassing code review and security processes.

But All is Not Lost

Innovative efforts to better secure open source and its supply chain are gaining momentum. The SBOM is gaining acceptance and it is getting easier to automatically generate SBOMs from CI/CD pipelines or build systems. A new initiative to make it easy to cryptographically sign code, called Sigstore, aims to remove barriers to code signing; ubiquitous code signing would make it much harder to compromise the supply chain by attacking the source code provenance problem at its root. Lastly, the Open Source Security Foundation has garnered tens of millions of dollars in funding from major technology companies and is in the process of spinning up mechanisms to issue grants and otherwise provide a more robust financial support mechanism for OSS. These developments point to a brighter future where the OSS Supply Chain is far more robust and companies can more easily audit, track and trust all the OSS dependencies in their applications and the full supply chain that moves OSS code from developer boxes up into production environments.

About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay

March 15, 2022