Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: The npm registry suffers spam infestation, and Microsoft makes Google sad. (more…)
Tag: supply chain attacks
-
npm is Scam-Spam Cesspool ¦ Google in Microsoft Antitrust Thrust
-
Dev of core-js Will Flip Table ¦ Another 451 PyPI Maldeps
Welcome to
The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Denis Pushkarev is fed up with core-js freeloaders, and hundreds more malicious packages found at PyPI. (more…)
-
To Prevent Supply Chain Attacks, Build Secure Code
More than a year after the massive SolarWinds cyberattack, targeted companies continue to feel its ramifications in both reputation and financial cost. Moreover, the global software supply chain remains vulnerable to severe attacks, whether from a hostile nation-state like Russia–now increasingly in the cybersecurity spotlight due to fears of retaliation due to U.S. sanctions–or from any other cybercriminal party.
As long as the world continues to choose fast over secure, nothing is safe. No one knows how many potential backdoors the SolarWinds attack alone created, which could allow the persistent presence of malicious actors on a seemingly infinite number of networks; nor do we know when the next attack will happen with potentially more dire consequences. For example, six months after the discovery of the Log4j vulnerability and its associated patches, we know that networks and products all over the world remain unpatched, vulnerable and exploitable in all sorts of companies both big and small.
This must change. And that change must start from the inside out; meaning programmers, developers and DevOps teams should start building code more securely in the first place. That also means that those who use cloud services, third-party black box software or the services of hub companies, whose networks connect both suppliers and customers, should better understand how code development affects security. This increased focus on the underlying technology itself will not completely prevent attacks, but it will reduce the number of attacks.
There are signs that the industry is moving in this much-needed direction—or that there are at least more discussions about it. The White House recently hosted meetings with technology experts focused on this very topic. And this spring, the SEC announced that it may begin requiring some companies to report cyberattacks within four days and also provide details about their cybersecurity posture. This increased government attention is raising awareness and could also open the door to regulations, or at least universal best practices for coding protocols. But regardless of eventual, still-hypothetical regulation, developers—and cybersecurity professionals at hub companies—should give serious thought to the process they use to securely build their products.
Beyond Open Source Code
Of course, an oft-recommended solution is to move to open source software (OSS). Although open source code is still vulnerable to attacks, the processes and practices involved in creating OSS code are transparent, which goes a long way toward crippling the ability of bad actors to conceal themselves. But such an approach will never work on a large scale in the private sector as there is no way to prevent programmers from simply copying others’ novel code, leaving no way to protect intellectual property among software developers.
However, there are other steps that could be nearly as effective, including integrating security features and testing into development tools and processes. For example, making gray and black box penetration testing a routine part of the software production process would go a long way in helping find vulnerabilities in the code that outsiders could find and access. Ideally, developers could then also remedy or repair these vulnerabilities. Also, assessing security at every step of the development process could give crucial insights into vulnerabilities before they are discovered in the wild by malicious actors.
Code-Signing and Scanning Innovations
Implementing other methods to spot security flaws in code, including in open source code, would also help. Using innovations like code-signing would alert developers if any changes were made to the code before they use it, helping to spot hackers or backdoors before the code is implemented into the product. In addition, developing better source code scanning technology is important for flagging vulnerabilities. Developers should also undergo training focused on building a product with a secure life cycle and implementing security protocols for merging and accessing the product code; this could prevent cyberattacks from occurring as a result of a faulty (or nonexistent) SDLC process—the same way an electrical engineer couldn’t design a building electric plan without formal training and certification and participating in regular continuing education and recertification.
CISOs Need to Be Proactive
CISOs and IT professionals also have an active role to play. They need to evaluate the security of their products’ underlying code , taking full advantage of the software bill of materials (SBOM), which lists all components of software (and that could soon be required by law) but that is only one side of it. This needs to be coupled with measures like code signing, scanning for flaws and keeping an updated list of components to spot newly-published weaknesses and vulnerabilities in libraries and frameworks. And it is increasingly important to make sure that contracts with software service providers specify they are adhering to secure development practices.
With the world on high alert and the private sector especially vulnerable to state-sponsored cyberattacks, there has never been a better time to focus on building more secure code. This does not mean that all of the other aspects of cybersecurity—including threat intelligence, incident response plans and hiring and retaining talented professionals—will be any less important. But if the code underlying the software can itself become more secure, these attacks could become fewer in number and their damage less far-reaching.
-
Secure Software Summit: The State of OSS Supply Chain Security
The open source software (OSS) supply chain is under attack. As evidenced by the recent Log4Shell vulnerability, the OSS supply chain is increasingly a focus for attackers seeking to exploit weak links in security. A number of research reports have recorded a significant increase in so-called ‘next-gen software supply chain attacks‘ over the past decade. These attacks look to compromise key links in the software supply chain including source code, build systems, CI/CD tools and code testing tools, among others. Next-gen attacks are happening across all languages, all types of software and are increasing in their frequency. Common forms include typosquatting, malicious code injection and tool tampering (like the CodeCov incident). In 2021, 64% of organizations reported they were affected by software supply chain attacks, according to container security company Anchore.
The increasing wave of next-gen attacks has driven new regulations mandating specific steps to inventory and make more transparent the content of software applications and the software supply chain. Specifically, executive order 14082, issued by U.S. president Joseph Biden, mandated a long list of security practices, including the inclusion of a software bill of materials (SBOM) with every application run by the U.S. federal government. The National Institute of Standards and Technology (NIST) is creating reference architectures and templates for application security as a result of the executive order. In other words, software security regulations will likely impact every company that produces code or sells products running on code (including SaaS products).
To be clear, some of the problems experienced by the OSS software supply chain are not exclusive to open source. The recent supply chain attacks on Microsoft Exchange Server and the SunBurst attack on the SolarWinds network management tool targeted proprietary code bases. That said, OSS is produced and maintained very differently than proprietary software. This does result in some unique risks. Here, we will cover those risks as well as some promising initiatives underway to address them.
Why Is the OSS Supply Chain A Juicy Target?
There are two primary reasons OSS supply chains are a tempting target. One is its omnipresence and the other is its weakness. It is not an exaggeration to say that OSS is omnipresent. According to Sonatype, 98% of all codebases contain OSS. This is likely an underrepresentation. You simply cannot build code from scratch today without using OSS components. It is too labor-intensive and expensive. The ubiquity of OSS makes the supply chain that creates, distributed and manages it an obvious target. At the same time, that widespread use translates into an increasing burden on companies to maintain, update and patch all OSS dependencies in their applications.
This is an impossible task; it is far easier to insert OSS code to solve an immediate problem than to, over time, make sure that code is updated and not vulnerable. For this reason, Sonatype estimated that 92% of all applications contain outdated or vulnerable code. We have seen the impact of this inability of organizations to maintain their OSS code through attacks like the one perpetrated against Apache Struts software run by credit reporting company Equifax, a breach that exposed the sensitive personal and financial information of hundreds of millions of Americans.
In recognizing that OSS is a fruitful channel for attacks, attackers have focused more specifically on the supply chain because it is often the weakest link. Elements of the supply chain reside in the gap between responsible organizations and may be less strictly policed or less actively secured. Continuous integration/continuous deployment (CI/CD) tools, OSS libraries that can easily access other systems, package managers and open source developer tooling are all components of the supply chain that are widely used but which are often the sole responsibility of many part-time maintainers who write and maintain code as an unpaid avocation.
OSS-Specific Challenges
The three key benefits of OSS each create unique security and supply chain challenges. Specifically, OSS is open, free and transparent.
The Two Sides of Open
Linus Torvalds said that many eyeballs make all bugs shallow. To a degree, this is true. But what if those same eyes are looking for bugs to exploit rather than fix? Not everyone on the internet is nice and many seek to exploit the fact that anyone with a GitHub account can contribute code to critical libraries. This attracts a lot of great contributors, but also a lot of bad actors and bad behavior.
That behavior is often hidden or delayed. For example, we often see bad actors ask to contribute to projects or libraries run by single maintainers that are relatively popular and used by big applications. Often, maintainers struggle to keep up with the demands of these projects. A bad actor will offer to help, submit a handful of good PRs and gain commit privileges to the repository. At this point, the bad actor will then stealthily insert malware, such as a cryptojacking script.
Free Puppy. Not Free Beer.
Free is a good price. There are no purchase orders. This makes it super easy to use. Free also serves as a great distribution strategy. However, sometimes free is not worth it. If an organization depends on an application and it becomes critical, they might require fast patching, better compliance and official support relationships. Free OSS comes with no guarantee, and this can be a major liability. An application’s maintainers might be writing the code and applying fixes in their spare time on weekends because they already have a full-time job. In fact, research has shown that a significant percentage of the critical software powering the internet is maintained by volunteers; this was the cause of the Heartbleed vulnerability in the OpenSSH encryption libraries.
Free code also may not be updated for years. If a vulnerability is disclosed, you may not be able to get permission to merge fixes you write. If the code is abandoned and the author no longer checks comments or requests, then you are stuck and may have no choice but to fork the project. A hard fork on short notice can cause chaos and its own problems. This is what happened with a popular Golang library called JWT Go, which made its way into dependency trees across the Golang ecosystem. JWT Go was even included in Kubernetes, one of the most popular and important open source projects. When a vulnerability to JWT Go emerged, everyone realized the maintainer had abandoned it. Multiple forked versions emerged with different patches for the same bug. Chaos ensued.
It is a common refrain to ask companies to fund volunteer maintainers. There are numerous programs for this, including GitHub Sponsors, Open Collective and Tidelift. These programs are helpful but not sufficient. You can’t just hand people a small amount of money and get more secure code and professional-grade maintenance. Even if you offered full-time salaries, it might not translate into well-supported, secure code.
The Illusory Benefit of Transparency
The transparency of OSS is real, but the idea that this code is getting audited and reviewed constantly is an illusion. Yes, the source code is distributed and open. But the sheer quantity makes it impossible to review all the code you use. If you are using millions and tens of millions of lines of code, you don’t have time or sufficient eyeballs. Even if you are spending considerable resources on code review, it is still easy to miss problems and vulnerabilities and relatively simple for malicious contributors to projects to hide malware or other bad inclusions. Reviewing the source code also does eliminate all risks. When most OSS code is consumed, it is running as a package compiled by another entity or developer.
Further, there is no guarantee that the contents of the package comes from the same source code you audited. We have actually seen this attack often with package repositories like npm and Maven, where malicious actors insert tampered builds or take over packages and insert code that is not listed as part of the package.
The CodeCov incident is one of the more recent and egregious examples. This free service for checking the code coverage of unit tests used BASH script running inside of a CI server for installation. The script was compromised and attackers inserted malware that could steal code secrets and export them. Those secrets might be tokens used to publish packages in npm or Maven and exposing those secrets allowed attackers to then compromise those packages. In another example, a PHP server managing source code distribution for the PHP project was compromised and multiple commits injected malicious code into the source code repository. The incident was caught quickly, but it demonstrated how attackers could tamper with source code directly, bypassing code review and security processes.
But All is Not Lost
Innovative efforts to better secure open source and its supply chain are gaining momentum. The SBOM is gaining acceptance and it is getting easier to automatically generate SBOMs from CI/CD pipelines or build systems. A new initiative to make it easy to cryptographically sign code, called Sigstore, aims to remove barriers to code signing; ubiquitous code signing would make it much harder to compromise the supply chain by attacking the source code provenance problem at its root. Lastly, the Open Source Security Foundation has garnered tens of millions of dollars in funding from major technology companies and is in the process of spinning up mechanisms to issue grants and otherwise provide a more robust financial support mechanism for OSS. These developments point to a brighter future where the OSS Supply Chain is far more robust and companies can more easily audit, track and trust all the OSS dependencies in their applications and the full supply chain that moves OSS code from developer boxes up into production environments.
About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.https://go.shiftleft.io/secure-software-summit-2022-replay
-
DAOPS Meetup 2021: Software Engineering, DevOps and DevSecOps
DAOPS Foundation—a non-profit organization committed to accelerating global digital transformation through technical standardization—is hosting the first ever DAOPS Meetup on May 18. The virtual meetup will bring together DevOps leaders from Sonatype, DevOps Institute, TARS Foundation and DAOPS Foundation to discuss the latest in software engineering and the latest DAOPS developments.
Here are the four sessions scheduled for the DAOPS Meetup.
ASEM, Next Generation Software Engineering Infrastructure
Software is the core of digital transformation. In the digital era, the way we produce software also needs to be digitized. Forest Jing, chief evangelist at DAOPS Foundation, explains how they plan to help global enterprises build a high-performance, technical organization based on Lean, Agile, CI/CD, SRE, DevSecOps, Cloud Native, etc., by developing the next generation software engineering infrastructure named ASEM.
Best DevOps Practice with TARS
TARS is a high-performance RPC framework based on name service and Tars protocol, also integrated administration platform, and implemented hosting-service via flexible schedule. During this session, Mark Shan, chair at Tencent Open Source and TARS Foundation, will share the best DevOps practice using TARS.
DevOps Hybrid Skills — Upskilling towards EverythingOps
Upskilling is a human endeavor and a key contributor to personal and career growth. This past year was like no other. As we progress through the “next normal” of this decade, one thing is certain—digital transformation relies on human skills transformation. DevOps has continued to shift left in recent years, becoming more integral to operational processes as manifests in domains such as secure-development-focused DevSecOps, secure-operations-focused SecOPs, artificial intelligence-based AIOps, machine learning-focused ModelOps, and data management-related DataOps. As each of these evolves, its specific required skill set is being fleshed out and internal IT staff have the opportunity to upskill to meet requirements. Join Dheeraj Nayal, global community ambassador at DevOps Institute, to learn all about DevOps upskilling.
The Rise of Next Generation Software Supply Chain Attacks
Cameron Townshend, Solution Architect from Sonatype
The adoption of open source in software development has grown exponentially over the past decade. Yet, the time required for hackers to exploit a newly disclosed open source vulnerability has shrunk by 93.5%. As development teams deliver new innovations, adversaries are also upping their game; something we’ve seen in recent high profile and devastating cyberattacks, such as Dependency Confusion.
Adversaries have the intent and ability to exploit security vulnerabilities in the software supply chain—and in some cases plant vulnerabilities themselves. They have increased scale through automation and improved breach success through precision targeting. Cameron Townshend, solution architect at Sonatype, urges developers to fight back by doing the same: automating security directly in the DevOps pipeline.
For more information and to register visit the DAOPS Meetup website.
-
Software Supply Chain Attacks: How to Disrupt Attackers
Supply chain attacks—compromising an organization via insecure components in its software supply chain—are a growing concern for organizations. Throughout the past three years, an increasing number of open source software package repositories have been found to contain malware, making it clear that all installation and update pathways for software and library code must have security controls applied to prevent and mitigate supply chain attacks.
Growing Risks of Open Source Software to the Business
Digital transformation has made every company a software company. As a result, open source software has grown 75% over the past two years. Despite this growth, few companies accurately track the components used in their code. Most lack the policies, processes and tools to keep up with the choices made by their developers, which introduces risk. In fact, 60% of the codebases audited in 2018 contained at least one vulnerability.
Bad actors have turned to exploiting these weaknesses to compromise organizations. If production code is infected with malware or vulnerabilities, inadvertently sourced from the repository, it may contaminate all organizations that come in contact with it—whether by using the code already in their SDLC or by launching presumed trusted applications from third parties who failed to validate their own code.
Attackers are also shifting their techniques to mask malware among large packaged objects using obscure formats we refer to as destructive objects. They infiltrate at various points in the development processes opportunistically, knowing they can infiltrate systems outside a managed operational environment. They take advantage of third-party dependencies and use payloads that can sit and wait until an unsuspecting developer implements their software. Hence, software supply chain attacks are difficult to detect, and often achieve dwell-time of hundreds of days.
Assessing Supply Chain Surface and Sources
In order to assess weaknesses and identify what warrants continuous monitoring, we need to understand both the surface and sources.
When evaluating surface risks, consider the following questions regarding the development environment:
What technologies, repositories and packages are in use?
- Inventory technologies used in development: Python, Ruby, JavaScript, Node.js, .NET, etc. This can also refer to operating systems.
- Third-party repositories of code: PyPI, RubyGems, NPM, NuGet, PEAR or PECL, and OS repositories such as DebianRepository, CentOS, Ubuntu, AUR and more.
- Packages: There can be multiple packages from a single repository. How often are packages updated? How often are new packages added?
Are there update procedures in place?
- What are the dependencies for every package? How many other packages are pulled from a single package update, and how often are they updated and/or reviewed?
Who’s managing the verification processes across all these dependencies? What update procedures are in place if there is a personnel change?
Once these questions have been answered, they should be addressed regularly. The process must be iterative to be successful.
When considering attack sources, it’s important to understand the characteristics of repositories teams draw from in order to understand potential areas of weakness. Third-party repositories rarely infect a big project as they are typically found elsewhere (e.g. GitHub) and have thorough code reviews. Official repositories, unofficial mirrors and internal code repositories represent increasing risk potential.
Random malicious packages, typosquatting, bad code review and phishing developer credentials to hijack accounts are all less technical ploys used by attackers, but equally dangerous.
The Software Supply Chain Challenge
The size, breadth of formats, trusted third-party and open source risks, and misused certificates have made it nearly impossible to stop malware-infected files and objects with today’s signature based, sandbox and vulnerability-based solutions.
Threats evade dynamic analysis remaining unknown to defenders or lacking details when detected. The explicitness of signature blacklists misses new or never-seen-before code, and sandboxes or dynamic analysis solutions are resource intensive and unable to scale to the breadth and volumes of formats required. Without a way to inspect large or cross platform archives, installers and source code, it’s impossible to extract indicators of compromise. These destructive objects can also impact SOC triage, response and continuous improvement.
Rising to the Challenge
How can you address these risks, not only in the software development life cycle (SDLC), but as SOCs address vulnerabilities that may come through other applications and repositories? How do you evaluate and inspect wrappers, digital certificates, executables, scripts, files and libraries, resources, documents and icons?
It’s vital to establish security practices within SDLC, from training developers on secure coding practices using open source libraries to factoring in detection capabilities including static analysis, dynamic analysis, software composition analysis and manual penetration testing. Implementing a secure SDLC process ensures that the development effort is protected by these activities, augmenting code reviews and infrastructure analysis.
Teams can extend security controls into the SDLC with technology that provides deep visibility into all aspects of the supply chain, from open source dependencies through continuous integration and delivery of packaged applications. Capabilities that provide greater application security across code development and deployment activities include:
- File reputation identifies and classifies “known bad” and “known good” files by applying a file’s unique hash against a database of goodware and malware.
- Automated static analysis detects and classifies unknown malware in code repositories by deobfuscating and decomposing objects and files without executing, and extracting threat indicators.
- Dynamic analysis executes files in a safe environment and captures the behaviors and results of malware, allowing the security team to understand what is going to happen.
- Software composition analysis analyzes applications to detect vulnerabilities, dependencies and licensing requirements in software.
- Threat intelligence is served up at various stages of the analysis cycle, and represents evidence-based knowledge including context, mechanisms, indicators, implications and action-oriented advice about existing or emerging threats.
By incorporating these technologies into the SDLC, development teams can obtain continuous validation across the software supply chain. Further automation, implementing new security guidelines and advancing the activities within the existing framework will allow the enterprise to progress its security measures and close gaps in the SDLC and supply chain operations.
At the same time, it’s important to recognize that threats can still make their way into the software supply chain, and it’s equally important that the SOC remain vigilant about maintaining a clear triage and escalation process. Continuous monitoring of software repositories should be an integral part of good hygiene processes, and incorporated into best practices applied to operational infrastructure, including the SOC. The SOC serves as a centralized management infrastructure to respond, contain and remediate attacks. This includes aggregating and consolidating threat data, building a case for managing a response which can include actions to formulate triage and incident response plans, and moving beyond SIEM.