devopsy.kinsta.cloud

Tag: Software Supply Chain

Aqua Security Acquires tfsec to Advance DevSecOps

Aqua Security today announced it has acquired tfsec, an open source project that provides a static analysis scanner for infrastructure-as-code (IaC) that is designed to be integrated within a DevOps workflow.

Amer Jerbi, Aqua Security CTO, said tfsec extends the security platform the company already provides in a way that advances adoption of DevSecOps best practices. In addition, to prevent developers from misconfiguring cloud infrastructure using tools such as Terraform, the scanner surfaces examples of code that provide developers with insights into how to properly configure infrastructure.

Jerbi noted that tfsec is already integrated with Aqua Trivy, an open source tool that scans container images for vulnerabilities. Integrations with the rest of the Aqua Security portfolio will follow, added Jerbi.

Tfsec co-founders Liam Galvin and Owen Rumney will join Aqua Security as cloud engineers. The challenge Aqua Security is trying to address is the need to enable organizations to proactively prevent cybersecurity issues from arising in production environments rather than only identifying vulnerabilities that need to be fixed, said Jerbi.

The decision to acquire tfsec comes in the wake of a series of high-profile breaches when many organizations are scrambling to make sure software supply chains are secure. Cloud platforms are rife with misconfigurations mainly because developers often lack the expertise needed to ensure infrastructure is secure after they have provisioned it. Organizations are embracing DevSecOps best practices in the hopes of educating developers to assume more responsibility for application security. However, developers are not going to acquire that expertise overnight. The tfsec scanner provides a way to enforce security policies today while at the same time affording the time to teach developers how to avoid misconfigurations.

It’s unclear how many cybersecurity events arise because of cloud misconfiguration issues. The cloud service providers have adopted a shared responsibility model that makes the organization employing the cloud responsible for both the applications deployed on those clouds and for how that infrastructure is provisioned. Many developers that provision cloud infrastructure themselves within the context of a DevOps workflow often assume the cloud service provider is doing more than it is to prevent misconfigurations. Cybercriminals, meanwhile, have become very adept at scanning for those misconfigurations.

There are, of course, a lot of layers and nuances when it comes to DevSecOps. Arguably, Jerbi said, the most critical thing for most organizations is to simply get started. Tools that can be quickly and easily integrated within an existing DevOps workflow with minimal disruption are critical, Jerbi added.

In the meantime, DevOps teams should expect cybersecurity teams to become more involved in their workflows, especially as government mandates aimed at securing software supply chains become stricter. Eventually, as organizations gain confidence in the ability of their DevOps teams to secure their application environments, those cybersecurity teams will refocus their efforts on hunting for threats. In the meantime, DevOps teams would be well-advised to make those cybersecurity professionals feel as welcome as possible, given the application security mandate most of them have been given.

July 12, 2021
Linux Foundation Makes Available Software Manifest Tool

The Linux Foundation is making available a set of free tools for building software bills of material (SBOMs) based on the software data package exchange (SPDX) file format it curates.

Backed by more than 20 organizations, SPDX is an effort to standardize the way metadata describing the contents of a software package is described.

The SPDX specification was submitted last fall to the International Organization for Standardization (ISO) by the Joint Development Foundation, an arm of the Linux Foundation.

The Linux Foundation, as part of its efforts, is also making available a free, online training course to provide IT teams with the foundational knowledge required to create and maintain SBOMs.

Kate Stewart, vice president of dependable embedded systems for the Linux Foundation, said SBOMs play a critical, often overlooked role in maintaining the security of a software supply chain. One of the major challenges organizations have today is that an SBOM isn’t often created, which makes it harder to discover what software components have been included in a package. That makes it difficult for organizations to determine if they are employing a version of a software component that now has a critical vulnerability that was just recently reported, noted Stewart.

Even when there is a SBOM, its construction can vary from one team or organization to the next. The tools provided by the Linux Foundation are intended to standardize how the fields used to describe a SBOM are defined, added Stewart.

The SBOM tools being advanced by the Linux Foundation are, in part, a response to President Biden’s executive order pertaining to the security of software supply chains, added Stewart. The National Telecommunications and Information Administration (NTIA), as a result, is now asking for wide-ranging feedback to define the minimum requirements for a SBOM.

It may take time before there is an agreed upon SBOM standard, but as microservices based on containers become more prevalent, it’s also apparent the issue will become more pressing. More, smaller packages of software being deployed in the enterprise without manifests will only further exacerbate an application security crisis that has deepened in the wake of recent high-profile software supply chain breaches.

Ultimately, a SBOM should automatically be self-generated as part of any DevOps workflow. The challenge is there is still a massive amount of software being created outside of any formal DevOps workflow. On top of that, the massive amount of poorly documented legacy application software already deployed in an enterprise IT environment makes it clear how big an issue the lack of a simple SBOM really is.

It’s not precisely clear who inside an organization should be responsible for standardizing SBOMs. The challenge is that within many organizations it’s not the kind of issue for which developers are rewarded, much less motivated, to solve. Nevertheless, everything from application security issues to the massive amount of technical debt that stems from duplication of software development projects can all be traced back to the lack of a simple manifest.

Of course, now that many organizations are starting to appreciate just how dependent they are on software, more questions are being asked about how it’s constructed. The thing that many of the business executives asking those questions are about to discover (with some incredulity) is how much of the software the organizations depend on is still being built using methods that have more to do with a medieval craft than they do a manufacturing line.

June 21, 2021
Google Proposes SLSA Framework to Secure Software Supply Chains

Google is proposing organizations adopt a framework for securing the integrity of software artifacts across a software supply chain.

Kim Lewandowski, a product manager for open source software security at Google, said the Supply Chain Levels for Software Artifacts (SLSA) is based on an internal framework, known as binary authorization for Borg, that the company has been employing now for more than eight years to secure its software. Borg is the name Google attached to the platform for building and deploying applications upon which Kubernetes is based.

In its current form, SLSA is a set of guidelines that identifies how to attain four different levels of application security. The goal, however, is to create a framework that will automatically generate auditable metadata that can be fed into policy engines to create a SLSA certification.

The lowest level of SLSA requires that the build process be fully scripted or automated and generates metadata that enables provenance to be created. At the highest level, SLSA requires a two-person review of all changes made during a hermetic, reproducible build process. At the highest SLSA security level, organizations should be able to assure others that their software supply chain has not been compromised, said Lewandowski.

In the wake of a recent series of high-profile breaches of software supply chains, the level of scrutiny being applied to how software is constructed has increased significantly. In theory, at least, developers have assumed more responsibility for application security as part of an overall shift left that provided them with more programmatic control over IT environments. As a result, cybercriminals are now focused on compromising either the credentials of developers as part of an effort to insert malware into application as it is developed, or they are trying to exploit vulnerabilities that developers have inadvertently created by programmatically misconfiguring infrastructure.

DevSecOps best practices are, of course, being adopted to teach developers how to better secure their application environments. The issue is that not enough developers are learning how to secure applications fast enough. The volume of attacks against the software supply chain continues to grow at a rate that is now much faster than organizations can thwart simply by educating developers. In some cases, organizations are once again empowering cybersecurity teams to make sure software supply chains are secure regardless of the potential impact on the rate at which applications are being developed.

Google is trying to promote adoption of SLSA as a framework for securing both open source and proprietary application code, said Lewandowski. There’s no silver bullet for securing software, but a framework that ultimately automates as many of the security processes as possible within the context of a DevOps workflow would go a long way toward hardening the software supply chain, noted Lewandowski.

SLSA is being put forward as a response to a cybersecurity executive order issued by President Biden that, among other things, tasks the National Institute for Standards and Technology (NIST), an arm of the U.S. Department of Commerce, to consult with federal agencies, the private sector, academia and other stakeholders in identifying standards, tools, best practices and other guidelines to enhance software supply chain security. Given the level of dependencies that exist across those software supply chains, that executive order, to varying degrees, now applies to almost any entity that builds software for an organization that, on some level, interacts with the Federal government.

June 21, 2021
Linux Foundation Project Secures Software Supply Chains

The Linux Foundation today embraced a sigstore project founded by Red Hat, Google and Purdue University to make it simpler for developers to employ cryptographic software, enabled by transparency log technologies, to secure software supply chains.

Luke Hinds, security engineering lead in the office of the CTO at Red Hat, said he and Dan Lorenc, a staff software engineer for Google, led a small team creating a lightweight mechanism for signing software artifacts, such as release files, container images and binaries, in way that creates an immutable, tamper-proof log.

Now, the goal is to take that project to the next level by contributing it to the Linux Foundation, at a time when the focus on software supply chain security has increased in the wake of a spate of high-profile breaches. As part of that effort, a free service, based on sigstore, will also be made available by the Linux Foundation to make the cryptographic software easily accessible to a wide range of developers.

Hinds said one of the reasons that artifacts are not signed by developers is that the current processes employed are too cumbersome. The cryptographic software is designed to be integrated as an extension of existing workflows, Hinds noted.

Because the log created is immutable, everyone participating in the software development process, including auditors, can now transparently track chain of custody for any software artifact, Hinds said. At its core, sigstore creates a set of short-lived ephemeral keys with a trust root based on an auditable public log.

Existing approaches rely on digests stored on insecure systems that are susceptible to tampering. Cybercriminals can, for example, swap out digests if they can gain access to compromised credentials. Those types of targeted attacks can be thwarted using sigstore, because the log management system that underpins it is immutable, said Hinds.

Of course, no system for securing software supply chains is going to be useful if developers won’t employ it. Key management, key compromise/revocation and the distribution of public keys and artifact digests are challenging to manage. Developers, then, need to determine which keys to trust in addition to learning how to sign them when different platforms are employed. Digests and public keys are also often distributed across vulnerable websites susceptible to hacks, or stored as a README file on a public git repository.

As organizations embrace DevSecOps best practices, it’s only a matter of time before the entire software supply chain comes under greater scrutiny. Ultimately, the issue will come down to finding a way to secure those supply chains, either from the top down by deploying and maintaining a platform, or simply making it easier for developers to better secure artifacts from the bottom up.

Regardless of approach, the signing of software artifacts should, one way or another, become more routine in the weeks and months ahead. The biggest challenge will be overcoming the simple inertia that currently exists among developers who routinely ignore the whole process.

March 9, 2021
Tidelift Secures Open Source Software Supply Chains

Tidelift today announced the general availability of a platform dubbed Tidelift catalogs, through which enterprise IT organizations can more effectively manage their open source supply chains.

Donald Fischer, Tidelift’s CEO, says the Tidelift catalogs platform enables IT teams to curate, track and manage open source components that have been vetted and adhere to compliance requirements by Tidelift. The company is working directly with the maintainers of those open source projects to compensate them for making that additional effort, said Fischer.

Developers can reliably pull components spanning common language frameworks like JavaScript, Python, Java, Ruby, PHP, .NET and Rust, with more programming environments to follow, added Fischer. Those components are made available via a Tidelift Subscription service.

Fischer said Tidelift is going beyond merely scanning open source components for vulnerabilities. The company is ensuring that when developers access versions of open source components, those versions comply with specific sets of policies, said Fischer. As a part of that subscription service, Tidelift monitors components for vulnerabilities as they are discovered, and works with maintainers to implement updates as needed.

While many organizations are starting to employ DevSecOps best processes, Fischer said the discovery of an issue involving open source components often comes far too late in the process. The Tidelift catalog approach ensures there are no last-minute surprise that are only discovered just before an application is about to be deployed in a production environment.

Securing software supply chains has become a higher priority since it was revealed that cybercriminals successfully inserted malware into an application update made to network monitoring software provided by SolarWinds. That malware subsequently rippled across IT environments involving multiple corporate entities, as well as several Federal agencies.

Of course, securing a software supply chain doesn’t happen overnight. Most IT organizations don’t have the processes and platforms required to vet open source components and frameworks. Tidelift is betting most of those organizations would prefer to subscribe to a service, rather than set up that infrastructure themselves. However, for organizations that require in-house monitoring of all the elements of their application development platform, Tidelift will make an on-premises edition of its platform available.

As organizations become more dependent on software, the potential impact malware can have on their operations can be extensive. Organizations of all sizes should expect to be fielding inquiries from customers, suppliers and partners concerning the level of rigor being applied to their software development processes. In effect, they are going to demand to know the provenance of the software they are being asked to implicitly trust. In many cases, those inquiries will provide the incentive to accelerate adoption of DevScOps best practices.

The days when organizations blindly used software with little regard to its origins are coming to a close. That doesn’t mean reliance on open source software is necessarily going to decline; however, it does mean application development teams will be held more accountable for the open source components they rely on.

February 2, 2021
Software Supply Chain Attacks: How to Disrupt Attackers
Supply chain attacks—compromising an organization via insecure components in its software supply chain—are a growing concern for organizations. Throughout the past three years, an increasing number of open source software package repositories have been found to contain malware, making it clear that all installation and update pathways for software and library code must have security controls applied to prevent and mitigate supply chain attacks.

Growing Risks of Open Source Software to the Business

Digital transformation has made every company a software company. As a result, open source software has grown 75% over the past two years. Despite this growth, few companies accurately track the components used in their code. Most lack the policies, processes and tools to keep up with the choices made by their developers, which introduces risk. In fact, 60% of the codebases audited in 2018 contained at least one vulnerability.

Bad actors have turned to exploiting these weaknesses to compromise organizations. If production code is infected with malware or vulnerabilities, inadvertently sourced from the repository, it may contaminate all organizations that come in contact with it—whether by using the code already in their SDLC or by launching presumed trusted applications from third parties who failed to validate their own code.

Attackers are also shifting their techniques to mask malware among large packaged objects using obscure formats we refer to as destructive objects. They infiltrate at various points in the development processes opportunistically, knowing they can infiltrate systems outside a managed operational environment. They take advantage of third-party dependencies and use payloads that can sit and wait until an unsuspecting developer implements their software. Hence, software supply chain attacks are difficult to detect, and often achieve dwell-time of hundreds of days.

Assessing Supply Chain Surface and Sources

In order to assess weaknesses and identify what warrants continuous monitoring, we need to understand both the surface and sources.

When evaluating surface risks, consider the following questions regarding the development environment:

What technologies, repositories and packages are in use?
- Inventory technologies used in development: Python, Ruby, JavaScript, Node.js, .NET, etc. This can also refer to operating systems.
- Third-party repositories of code: PyPI, RubyGems, NPM, NuGet, PEAR or PECL, and OS repositories such as DebianRepository, CentOS, Ubuntu, AUR and more.
- Packages: There can be multiple packages from a single repository. How often are packages updated? How often are new packages added?
Are there update procedures in place?
- What are the dependencies for every package? How many other packages are pulled from a single package update, and how often are they updated and/or reviewed?
Who’s managing the verification processes across all these dependencies? What update procedures are in place if there is a personnel change?

Once these questions have been answered, they should be addressed regularly. The process must be iterative to be successful.

When considering attack sources, it’s important to understand the characteristics of repositories teams draw from in order to understand potential areas of weakness. Third-party repositories rarely infect a big project as they are typically found elsewhere (e.g. GitHub) and have thorough code reviews. Official repositories, unofficial mirrors and internal code repositories represent increasing risk potential.

Random malicious packages, typosquatting, bad code review and phishing developer credentials to hijack accounts are all less technical ploys used by attackers, but equally dangerous.

The Software Supply Chain Challenge

The size, breadth of formats, trusted third-party and open source risks, and misused certificates have made it nearly impossible to stop malware-infected files and objects with today’s signature based, sandbox and vulnerability-based solutions.

Threats evade dynamic analysis remaining unknown to defenders or lacking details when detected. The explicitness of signature blacklists misses new or never-seen-before code, and sandboxes or dynamic analysis solutions are resource intensive and unable to scale to the breadth and volumes of formats required. Without a way to inspect large or cross platform archives, installers and source code, it’s impossible to extract indicators of compromise. These destructive objects can also impact SOC triage, response and continuous improvement.

Rising to the Challenge

How can you address these risks, not only in the software development life cycle (SDLC), but as SOCs address vulnerabilities that may come through other applications and repositories? How do you evaluate and inspect wrappers, digital certificates, executables, scripts, files and libraries, resources, documents and icons?

It’s vital to establish security practices within SDLC, from training developers on secure coding practices using open source libraries to factoring in detection capabilities including static analysis, dynamic analysis, software composition analysis and manual penetration testing. Implementing a secure SDLC process ensures that the development effort is protected by these activities, augmenting code reviews and infrastructure analysis.

Teams can extend security controls into the SDLC with technology that provides deep visibility into all aspects of the supply chain, from open source dependencies through continuous integration and delivery of packaged applications. Capabilities that provide greater application security across code development and deployment activities include:
- File reputation identifies and classifies “known bad” and “known good” files by applying a file’s unique hash against a database of goodware and malware.
- Automated static analysis detects and classifies unknown malware in code repositories by deobfuscating and decomposing objects and files without executing, and extracting threat indicators.
- Dynamic analysis executes files in a safe environment and captures the behaviors and results of malware, allowing the security team to understand what is going to happen.
- Software composition analysis analyzes applications to detect vulnerabilities, dependencies and licensing requirements in software.
- Threat intelligence is served up at various stages of the analysis cycle, and represents evidence-based knowledge including context, mechanisms, indicators, implications and action-oriented advice about existing or emerging threats.
By incorporating these technologies into the SDLC, development teams can obtain continuous validation across the software supply chain. Further automation, implementing new security guidelines and advancing the activities within the existing framework will allow the enterprise to progress its security measures and close gaps in the SDLC and supply chain operations.

At the same time, it’s important to recognize that threats can still make their way into the software supply chain, and it’s equally important that the SOC remain vigilant about maintaining a clear triage and escalation process. Continuous monitoring of software repositories should be an integral part of good hygiene processes, and incorporated into best practices applied to operational infrastructure, including the SOC. The SOC serves as a centralized management infrastructure to respond, contain and remediate attacks. This includes aggregating and consolidating threat data, building a case for managing a response which can include actions to formulate triage and incident response plans, and moving beyond SIEM.

— Tomislav Pericin
January 17, 2020
Why Your Software Supply Chain Might Be Your Achilles Heel

Historically, cybercriminals are as lazy as they are innovative. They can come up with clever exploits and attack vectors, but they still generally focus on the low-hanging fruit in the most target-rich environments. Recently, attackers seem to have shifted focus away from directly targeting companies with strong security or a wealth of resources, instead going after weak links in the software supply chain.

No matter how many resources or how much effort you dedicate to securing your networks and protecting your data, it can all be for nothing if you trust a third-party supplier or vendor that is vulnerable. It is also true of your software supply chain, because the process of developing and deploying software offers opportunities that attackers can exploit.

The Case of Xcode Ghost

iOS devices are considered to be exceptionally secure. iOS is not impervious to attack or exploit, but it is more secure than the competing Android platform. Apps for iOS are also considered to be more secure, because Apple’s App Store is a “walled garden” with a stringent review process for an app to be approved.

A few years ago, however, it was discovered that more than 4,000 apps in the Apple App Store contained malicious code. Attackers had figured out how to leverage a weakness in the software supply chain to get past the gatekeepers of the walled garden.

How did they do it? iOS apps have to be written in Xcode. The Xcode software is provided for free, but the Apple servers are often slow—especially when trying to download from China, so developers often just search the web to get Xcode from a third-party site. Cybercriminals developed a malicious version of Xcode, then gamed the systems to ensure their version of the software would show up at the top of online searches.

Apps developed with the malicious Xcode software contained extra code that would phone home and provide the attacker a backdoor—and an opportunity to inject malicious code or execute commands within the compromised apps.

Low-Hanging Fruit

The Xcode Ghost incident is an excellent example of a software supply chain attack, and a perfect illustration of attackers going after the low-hanging fruit. iOS is relatively secure. The Apple App Store is protected and apps are reviewed to ensure they meet strict criteria. So, instead of trying to go after Apple or iOS, attackers figured out how to strike the platform used to develop the apps and sneak in through the back door.

I spoke with Ryan Olson, Intelligence Director for Palo Alto Networks, about the rising threat of attacks on the software supply chain. He explained that attackers know that when they are going after a hard target that would be difficult to defeat directly, there is a better and easier way: Just figure out who they trust and go after the low-hanging fruit.

This becomes a potentially larger issue in the context of DevOps and automation. The recent NotPetya ransomware was propagated through a malicious update. If your systems or software are configured to update and/or deploy automatically, a software supply chain attack can be in the wild and compromising your applications or data before you know it.

There are tools available to scan images, applications or containers before deploying them, but they typically test for known vulnerabilities or overt crashes or conflicts. They are not designed to look for latent backdoors that might be buried in the code.

Olson recommends that you start taking a closer look at the software vendors you rely on. It is important to understand how they are securing and protecting their code—because their weaknesses expose you to threats by virtue of the trust you give them and their apps.

— Tony Bradley

April 2, 2018
DOES 2015 – Josh Corman, John Willis on Immutable Awesomeness

Continuing recapping my favorite sessions of DevOps Enterprise Summit 2015, here is Josh Corman and John Willis talking about Immutable Awesomeness. What is Immutable Awesomeness you ask? Have a listen to my interview with John and Josh and then listen into the actual session where they go over it.

Just a couple of hints: 1. Software supply chains are a great way to look at your development/deployment process 2.In a world where software controls everything, it really is a case of life and death 3. If you can’t get excited about what this all means, listen to the session again.

Of course what makes this interview and session great is the unbridled enthusiasm that both John and Josh bring to the subject. In fact there was a palpable buzz in the theater when they walked off the stage and over to my video studio. The same enthusiasm carried over to our interview too. It was a lot of fun to interview these two.

So have a listen and enjoy. Also if you enjoy John Willis presenting, he will be headlining the DevOps Connect: cdSummit SoCal on December 3rd, joining some other great speakers.

Here is John and Josh’s actual presentation:

November 13, 2015
The Software BOM Squad
In my previous post, “When Good Code Goes Bad“, I shared new research showing the average large development organization consumes over 15,000 known vulnerable and defective components annually. While we can’t stop software from going bad, there are practices from traditional manufacturers that we can use to improve our ability to recall and fix the “bad” software components.

The Software BOM

A Bill of Materials (BOM) is used in traditional manufacturing supply chains to list the suppliers and parts used in a product, a “software bill of materials“ (BOM) is an inventory of the third party and open source components used to build an application.

A sample software bill of materials lists all open source and third party components, including their version, age, license type, and any known security vulnerabilities.

As noted in Wikipedia, “The concept of a BOM is well-established in traditional manufacturing as part of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. If defects are later found in a specific part, the BOM makes it easy to locate affected products.

“A software BOM is useful both to the development organization (manufacturer) and the buyer (customer) of a software product. Builders often leverage available open source and third-party software components to create a product; a software BOM allows the builder to make sure those components are up to date and to respond quickly to new vulnerabilities. Buyers can use a software BOM to perform vulnerability or license analysis, both of which can be used to evaluate risk in a product. Understanding the supply chain of software, obtaining a software BOM, and using it to analyze known vulnerabilities are crucial in managing risk.”

A software bill of materials not only inventories what is used, but in some cases it also syncs with real-time component defect data to indicate which components have known vulnerabilities or license risks. Various software supply chain automation tools expand the bill of software much further, alerting stakeholders automatically when a defect alert occurs. In advanced tools, the entire software lifecycle is automated to ensure that defective components are avoided and continuous monitoring instantly identifies newly announced vulnerabilities as soon as they are discovered.

Breaking Bad

As mentioned in my previous post, When Good Code Goes Bad, there is no way to prevent software from “going bad”. No one is immune from these issues. Therefore, we need to focus on our responses to these issues.

To paraphrase Vince Lombardi, “It’s not how you get knocked down, it’s how you get up”. Following the OpenSSL vulnerability announcement in 2014 (or pick any other new security vulnerability with or without an assigned logo), every development and IT operations team on the planet was asking two questions:
1. Did we ever use that open source component?
2. if yes, where is it?
For those organizations tracking their use of components with a software bill of materials, the answer was easy to find. For other organizations, weeks and often months of research were required. To get a sense of the lack of maturity around traceability practices, take a look at this image from Sonatype research showing announcements of OpenSSL vulnerabilities in products following the initial vulnerability disclosure in April 2014.

Automating Traceability Across the Software Supply Chain

Software supply chain complexities, coupled with the volume and velocity of open source and third-party components flowing through them, are forcing us to think differently about our current practices across development and operations. We are not the first to face these challenges and we can learn a great deal from the mature practices of other high-velocity manufacturing organizations and industries.

One thing is certainly clear. Manual practices of monitoring or occasional checkpoints cannot keep pace with volume and velocity of components being consumed. And if you extend these considerations beyond just open source components and into containers, microservices, or other images used in development and operations, the problem is further out of reach for manual processes to keep pace. The volume and velocity within our software supply chains will not diminish—and without a new approach, the volume of unchecked quality and integrity of parts being consumed will continue to build up as technical debt.

But the answer is here today: automation. Automation can unleash the potential of an organization’s development capacity. Rarely is there such an opportunity to simultaneously increase speed, efficiency, and quality. Solutions that facilitate comprehensive software supply chain automation are poised to usher in the next wave in development productivity—with gains on par or even greater than possible with agile, Lean, and DevOps.

More information on the current state of software supply chain practices as well as examples of best practices being employed by software development teams, can be found in the 2015 State of the Software Supply Chain Report.
July 22, 2015
When Good Code Goes Bad

Milk spoils. Iron rusts. And in software, good code goes bad. Yet the difference is, with the first two, you know the change has occurred. With software, those changes are not always obvious.

Your 5,100 Binaries Went Bad

There is no way to prevent software from “going bad”. As with all products, bugs and defects,are bound to happen at some point. No one and no code is immune from these issues. But who’s looking for the 5,100 software components in your organization that went bad last year (meaning new security vulnerabilities were discovered in them)? It’s all too likely, no one.

Earlier this year, I took a deep dive into the analysis of software supply chains that fuel high velocity development practices and IT operations. The analysis revealed that some of the largest development organizations were consuming an average of 240,000 open source components to expedite development, accelerate innovation, and meet time to release deadlines. While reaping huge benefits from this consumption, these same organizations were unintentionally growing their technical and security debt. How you ask?

Here is a summary of the their average debt:

Source: 2015 State of the Software Supply Chain Report

The analysis showed that over 15,000 (7.5%) of the open source components being consumed by these organizations in 2014 had known security vulnerabilities. Of those 15,000 components, an average of 66% (9,900) had known vulnerabilities dated 2013 or older. That means, they were known vulnerable components (“bad”) before they were downloaded.

The remaining 34% (approx. 5,100) might have actually been “good” components at the time they were downloaded by development teams from public open source repositories, but at some time during 2014 a new security vulnerability was discovered and a CVE identifier was assigned.

Even with the best controls in place to protect developers from downloading open source components with known vulnerabilities, the fact that new vulnerabilities are continuously discovered — about 50 new CVEs are announced daily — means that continuous integration and delivery practices also need to be complemented by continuous governance and visibility practices.

We’re Not Alone

Software development is not the only industry facing such issues of things good things going bad. I bet all of us have received notification from our car’s manufacturer that a certain part is being recalled. While the part was known to be safe when the manufacturer sourced it from suppliers, placed it on the car during assembly and shipped it, discoveries of defects are often found years after the vehicle has been on the road. The same scenario has been played over and over in the food, pharmaceutical, defense, toy, electronics, and other industries.

The difference is those other industries are known to have more mature supply chain practices where they can track and trace every part or ingredient ever used in every product they have shipped. And when a part goes bad, they can recall and repair that defective component.

In software, track and trace practices are still in the earliest stages of maturity. In a 2014 survey of 3,300 development professionals, 63% stated that their organizations have incomplete or no records of the components used in their applications. This lack of traceability means that even when good components go bad, development and IT operations teams have no way to quickly find the defective component and replace it with a better, safer version — dramatically extending mean times to repair, or in the case of no action, growing their technical and security debt. And when applications with those defective components are in the hands of customers, the ability to “recall” the impacted software or distribute newer versions of the application are hindered.

Did he recommend a “Bomb”?

A Bill of Materials (BOM; pronounced “bomb”) is used in traditional manufacturing supply chains to list the suppliers and parts used in a product. According to Wikipedia, “The concept of a BOM is well-established in traditional manufacturing as part of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. If defects are later found in a specific part, the BOM makes it easy to locate affected products.”

In Part 2 of this blog, “The Software BOM Squad“, I will introduce the importance of automating the creation of a “software bill of materials“ (Software BOM) to take inventory of the third party and open source components used to build an application. When software components age more like milk than wine, a software bill of materials can help you quickly identify where your 5,100 “bad” component were used.

July 22, 2015