devopsy.kinsta.cloud

Author: Tomislav Pericin

  • Addressing Software Supply Chain Security

    Addressing Software Supply Chain Security

    It’s essential for organizations to learn more about the software supply chains they rely on and the steps needed to secure them. In just the past few years, we have seen a major uptick in malicious actors exploiting vulnerabilities in software supply chains to facilitate attacks on organizations. However, it’s important to remember that these attacks are not a new phenomenon. In fact, they’ve been around for decades. Therefore, the need to secure the software we rely on is not new, though the industry turning its attention to this problem is.

    Software supply chain attacks are also far from simple. Given the fact that software supply chains are highly complex and made up of a number of constantly changing components, the avenues that malicious actors can take to pull off an attack are plentiful. For example, software artifacts are partially made up of components downloaded from open source software repositories. These platforms have become a favored attack avenue, given how frequently they are used by developers and the trust extended to them by development organizations. According to GitHub, anywhere from 85-97% of enterprise codebases come from open source repositories. Two of these repositories, npm and PyPI, have seen a nearly 300% spike in the number of attacks on their platforms in the past four years, impacting thousands of software supply chains globally.

    One example of the kinds of software supply chain attacks that are now common is IconBurst, a major attack from last year where more than two dozen typosquatted packages were discovered on npm. As with other recent attacks, the IconBurst attackers used typosquatting – uploading malicious packages to repositories with names that closely resemble popular and legitimate packages on the same platform. Developers, in a rush to complete their work, are often tricked into downloading these typosquatted packages. In the case of the IconBurst campaign, for example, one of the malicious packages discovered was downloaded over 17,000 times!

    Given the significant increase in the number of software supply chain attacks, organizations must make more of an effort to understand how these attacks work and then take steps to secure their software supply chains from compromise.

    Say Goodbye to “Shift Left”

    To begin with, development organizations need to reevaluate the mantra to “shift left.”

    For years, industry leaders promoted the notion of “shifting left” as a call to marry application security with application development: Embedding secure development tools and practices into the continuous integration/continuous delivery (CI/CD) process. But times are changing and the industry is beginning to understand that while shifting left is necessary, it is not sufficient to secure software supply chains.

    “Don’t shift left; shift everywhere!” said Tanya Janca, founder and CEO of WeHackPurple. Janca was speaking as a member of a recent Techstrong panel, “Software Supply Chain: It’s All About The Code” at the Predict 2023 virtual event. Janca pointed out the industry’s past fixation on “shift left” and the gaps in security that this caused since threats to the software supply chain not only happen during development but also throughout the entire software release process.

    In fact, the notion of shifting left may do development organizations and developers a disservice: Leaving the work of securing applications entirely to them. “It’s unfair to expect developers to write perfect code,” said Jeff Williams, co-founder and CTO of Contrast Security, another member of the panel.

    Software engineers are already under pressure to develop software quickly to meet development and project milestones. That makes it difficult for them to implement perfect security measures, like vetting every single line of third-party code they use while also watching out for social engineering attacks.

    A New Approach to Secure the Software Supply Chain

    For organizations looking to secure the software supply chains they rely on, they must choose the right tools. But much of the secure development and application security testing (AST) technology out there won’t cut it. For instance, tools such as dynamic or static application security testing (DAST/SAST) are critical components of the secure development life cycle (SDLC). However, they don’t provide holistic coverage of all of the supply chain threats out there, including software tampering and the risk posed by compromised open source and third-party libraries. On the other hand, tools such as software composition analysis (SCA) can help vet open source components but often fail to detect malicious modules or provide total security coverage of software supply chains. Knowing that your application depends on a malicious open source package after the fact is of little consolation when you release code at the speed of DevOps.

    This is why modern tools that go beyond these capabilities should be considered. “It’s about giving developers the tools they need to be successful,” said GitLab’s CPO David DeSanto at the event. Experts in the space agree that change is needed but that developers and development teams aren’t set up for success as it stands.

    According to Mat Matthews, an experienced practitioner in the DevOps space: “While developers are responsible for ensuring clean code, it takes a cross-functional team with a specific focus on security to reduce the attack surface against supply chain attacks.”

    To accomplish that, organizations need to look to their security operations centers (SOCs) to take on a greater role in helping to secure software supply chains. Close collaboration between development and SOC teams is the surest way to provide end-to-end security for an organization’s software supply chains. And, supplied with the right tooling, SOCs and development teams working together will be better positioned to scan software packages; achieve greater visibility into development processes; detect deviations from a set baseline; and pursue possible supply chain compromises before they can cause real harm to software publishers and their customers.

    As threats and attacks on software supply chains evolve, so must protections. A modern software supply chain security program should aim to protect the entire software development process, from the developer IDEs and their plugins to the open source components, CI/CD workflows and the release pipelines, while also being equipped to spot and pursue compromises at any point in the development life cycle. Only then will development teams be able to prevent harmful attacks like SUNBURST, IconBurst and countless others that have happened since.

  • Supply Chain Security: Has the Next SolarWinds Already Happened?

    Supply Chain Security: Has the Next SolarWinds Already Happened?

    More than two years after the now-infamous hack of the firm SolarWinds, the incident is very much in the foreground of conversations about cybersecurity defense, threat detection and incident response. As evidence of that: The SolarWinds incident was the subject of five separate talks and panels at the recent RSA conference in San Francisco. Dozens more talks addressed issues related to supply chain security that–directly or indirectly–invoked the SolarWinds incident. 

    That’s because the SolarWinds attack raises important—if not existential—questions for both software development and DevOps teams and the companies running the software. First: If sophisticated cybercriminals can insert malicious backdoors directly into application code from reputable developers, what organization can afford to blindly trust software? Second: If it is now the responsibility of software teams  to spot sophisticated, methodical compromises of their development pipelines, what is the best and most reliable means of doing that? Finally, if it is also now the job of software consumers to monitor for—and detect—compromised binaries from otherwise reputable vendors (like SolarWinds), how should they go about doing that given ongoing staffing and technology constraints? 

    The answers to these questions are matters of debate within DevOps, software development and information security communities. What is beyond debate is that the SolarWinds incident was not a fluke. In the months since that attack, attackers—both criminal and nation-state-backed—have leveraged software supply chain security blindspots to target development organizations as means of gaining access to sensitive IT environments. While we haven’t yet heard of another incident of the scale or importance of SolarWinds, it is entirely reasonable to assume that the next SolarWinds-caliber compromise could already be unfolding, but has yet to be discovered and/or publicly acknowledged. 

    Software Supply Chain Security Weaknesses

    SunBurst, the malicious implant in SolarWinds software, endures as a topic of interest and conversation for information security professionals and for DevOps and development organizations precisely because of the ways it exposed glaring weaknesses in the security of development organizations and software supply chains. As noted in my analysis of the hack in December 2020 and FireEye’s analysis of the same compromise, the SolarWinds hackers gained access to the company’s development environment months before they actually launched their attack. During that time, they closely studied the underlying source code and mimicked SolarWinds style and nomenclature as they added malicious code and integrated it into the SolarWinds Orion application via a compromise of the company’s build system.  

    Though carefully planned and conducted, the SolarWinds attack wasn’t foolproof. The attackers left clues, such as their additions of native Windows functions, use of cryptography, compression and more. The attackers obfuscated these using compression and Base64 encoding designed to fool simple threat hunting rules that looked for their plaintext variants and other suspicious strings. Still, the obfuscation repeated throughout the implanted code was enough to send up a red flag to anyone who bothered to look closely. That left the attack open to discovery. 

    The problem is this: Very few software publishers have the staff, expertise or tooling to comb through massive codebases to spot a handful of sketchy obfuscated strings or analyze large binaries for deviant behaviors before deploying them. As evidence of this, ReversingLabs sponsored a survey of more than 300 IT professionals at software-driven enterprises. It found that 87% were aware that software tampering, like what occurred in the SolarWinds attack, could lead to enterprise security breaches. About 40% of those surveyed said CI/CD toolchain exposures posed a risk to their organization. Still, barely a third of respondents (37%) said they had the means to detect such software-tampering attacks. 

    Threats to Software Development Organizations Grow

    Attackers are aware of this weakness within the software development pipeline, which is why threats to development organizations are metastasizing. Just in the last year alone, we have seen a series of attacks on development infrastructure and software supply chains. In one April 2021 incident, attackers were successful in pushing malicious, unreviewed code directly to the PHP main branch, ultimately resulting in a compromised, formal version of PHP being pushed to all PHP websites. That same month, there was a breach at the firm CodeCov, which involved unauthorized access to and modifications of the company’s Bash Uploader script. An investigation of that incident ultimately concluded that the alteration of a single line of code in that script by an unauthorized third party enabled the attackers to (potentially) export information stored in the users’ continuous integration (CI) environments. 

    More recently, ReversingLabs observed proof-of-concept and “white hat” activities testing so-called “dependency confusion” attacks, in which attackers set up malicious packages in public repositories like GitHub that squat on the names of private, non-public repositories and sport high version numbers. Misconfigurations in package management systems can result in these malicious components being imported in lieu of the legitimate, privately managed namesakes. As ReversingLabs found, a number of malicious modules discovered on the npm platform were used to target German technology, manufacturing and entertainment firms. It was later revealed to be the work of a white hat firm conducting red team exercises on behalf of those firms. Red team exercises aside, RversingLabs research also detected other, malicious packages in npm, as well as similar attacks targeting Ruby Gems and Python Package Index (PyPi). 

    And that doesn’t even count issues like the Travis CI flaw that exposed (potentially) millions of credentials to developers’ accounts on GitHub, Amazon Web Services and Docker Hub. Lapses like that can fuel supply chain attacks, giving malign actors the ability to insert malicious code into otherwise legitimate open source or proprietary code repositories. After reading such headlines, and without the means to check code intent, how can developers trust their software to update? And how can they do so while at the same time being forced to frequently update their dependencies to reduce or eliminate critical software vulnerabilities?

    Dev Teams Need New Tools

    For software development organizations struggling to meet delivery deadlines and satisfy customer expectations, reorienting to address amorphous software supply chain threats and attacks is a bridge too far. The same is true for enterprises struggling to staff security teams and simply keep on top of targeted phishing attacks or threats to public-facing IT assets. Too often, these entities only find religion about software supply chain attacks after they have occurred. 

    What software development organizations need is a way to secure the CI/CD pipelines and release packages, ensuring that their software builds are free of code tampering and other supply chain risks introduced through build systems. That should include the ability to detect compromises by monitoring for malicious software changes and enforcing security controls and failure conditions on software build processes to ensure that you can release your code with confidence.

    On the customer end, organizations need a way to detect malicious behaviors and unauthorized software changes in compiled binaries before they are released into production, while also monitoring for risks like exposed secrets or exploitable software vulnerabilities that might lurk in compiled code. 

  • Securing the Software Supply Chain with Behavioral Analysis

    Securing the Software Supply Chain with Behavioral Analysis

    Lately, software supply chains find themselves in a very interesting and uncomfortable position—the industry spotlight—and not in a good way. While significant and costly breaches such as  SolarWinds or Kaseya make front-page news, supply chain attack tactics (e.g. typosquatting or dependency confusion) that target package managers such as npm, PyPI or WinGet can poison downstream applications or compromise developer credentials.  

    The software engineering community has been steadily increasing its use of third-party and open source solutions to help create new offerings. In fact, over the last five years, there has been a 259% increase in the use of open source.

    One reason for this is the simple fact that creating new applications takes time and can strain budgets that are already tight. With open source code, software engineers gain fast access to pre-existing code that’s already demonstrated a proven ability to address the very unique challenge they are looking to solve. By removing the need to reinvent the wheel, the development timeframe is accelerated, the costs are lowered and teams are freed to focus on the creation of new and unique features that the customers demand.

    At this point, turning to third-party code may seem like a no-brainer. However, it can also open the door to potential risks. This is especially true with engineers who, rather than checking software to ensure it is secure and compliant, take the providers’ assurances as gospel. 

    They are not alone. Other groups will try to conduct their initial due diligence at the onset if that’s possible. Soon after they then ease into a “set it and forget it” approach, where they fail to update the code over time. As a result, while the code may appear secure initially, vulnerabilities can emerge over time, and if the software is not continually updated, bad actors can leverage an unprotected window of opportunity. In fact, according to Enterprise Security Group’s “Modern Application Development Security” eBook, 48% of organizations regularly push vulnerable code. The same research also found that 60 percent of organizations have had production application vulnerabilities exploited.

    Organizations conduct static application security testing (SAST) and dynamic application security testing (DAST) to assess pre-production code vulnerabilities. They will also scan with AV before releasing software, examine signing certificates, and code quality. Yet, despite these efforts, they remain challenged by gaps in security processes and the ability to sufficiently inspect large, complex and multi-layered software packages. What’s missing is the ability to conduct in-depth analysis to pinpoint behavioral differences between compiled software versions.  

    Malicious tampering can often involve changing how the software interacts with the system on which it is deployed.  Differential behavior analysis highlighted below makes it easier to detect those unusual changes. For example, having an application suddenly add a pre-install script that downloads files via certutil is highly unusual. If the addition was planned, reasons for the change would be well documented and it would be associated with a significant new feature. For this type of change to happen in any other scenario is suspicious and deserves further investigation before the software is released. In this case, such action would have caught ua-parser-js developer’s account compromise.

    supply chain

    With the software supply chain at risk and with shared code the new normal, what’s needed is software assurance. 

    Software assurance allows teams to inspect every application layer within an installation package for the presence of software tampering, malware and other indicators of software supply chain attacks.   

    Begin the software assurance process with the deep inspection step through recursive package decomposition. The resulting software bill of materials (SBOM) becomes so much more than a build manifest (a list of components that should be in the final build). It becomes the means to determine exactly which components have unacceptable build quality or those that contain unwanted software behaviors. This enables developers to verify the code they are working with and take appropriate action to protect their customers and partners from the majority of software risks. These assessments include:

    • Software grading: The software package and every included component can be graded based on the types and severity of security issues found. Using a grading system makes it easy for software buyers to gauge the level of risk posed by a software package and determine whether additional deployment measures can be taken to mitigate those risks. For developers, component grading makes it easier to prioritize their remediation efforts and aids in the selection of higher-quality third-party components.  
    • Application hardening: Activating specific build or compiler options can mitigate vulnerabilities and protect applications against attack—but only if they are implemented correctly. Assessing whether these mitigations are correctly implemented ensures that these protection mechanisms will work as expected. 
    • Binary behavior analysis: This allows teams to fully understand how each software component interacts with the system on which it will run. The ability to understand third-party component behaviors, even when developers have no access to the source code, delivers a more comprehensive understanding of supply chain risks. Checking for unusual software behaviors or unexpected behavioral changes between one version and the next is often the only way to identify software tampering by a malicious actor. The ability to have this final check before distribution enables software developers to validate that the software being released is trustworthy.

    For companies that are using software to create business value or producing software that is mission-critical for their customers, turning a blind eye or making software assurance assumptions can no longer be tolerated. In today’s climate where increasingly sophisticated threats loom, inspecting software and monitoring updates for signs of tampering, malicious or unwanted additions is essential and must become a part of standard software development practice. 

  • Software Supply Chain Attacks: How to Disrupt Attackers

    Software Supply Chain Attacks: How to Disrupt Attackers

    Supply chain attacks—compromising an organization via insecure components in its software supply chain—are a growing concern for organizations. Throughout the past three years, an increasing number of open source software package repositories have been found to contain malware, making it clear that all installation and update pathways for software and library code must have security controls applied to prevent and mitigate supply chain attacks.

    Growing Risks of Open Source Software to the Business

    Digital transformation has made every company a software company. As a result, open source software has grown 75% over the past two years. Despite this growth, few companies accurately track the components used in their code. Most lack the policies, processes and tools to keep up with the choices made by their developers, which introduces risk. In fact, 60% of the codebases audited in 2018 contained at least one vulnerability.

    Bad actors have turned to exploiting these weaknesses to compromise organizations. If production code is infected with malware or vulnerabilities, inadvertently sourced from the repository, it may contaminate all organizations that come in contact with it—whether by using the code already in their SDLC or by launching presumed trusted applications from third parties who failed to validate their own code.

    Attackers are also shifting their techniques to mask malware among large packaged objects using obscure formats we refer to as destructive objects. They infiltrate at various points in the development processes opportunistically, knowing they can infiltrate systems outside a managed operational environment. They take advantage of third-party dependencies and use payloads that can sit and wait until an unsuspecting developer implements their software. Hence, software supply chain attacks are difficult to detect, and often achieve dwell-time of hundreds of days.

    Assessing Supply Chain Surface and Sources

    In order to assess weaknesses and identify what warrants continuous monitoring, we need to understand both the surface and sources. 

    When evaluating surface risks, consider the following questions regarding the development environment:

    What technologies, repositories and packages are in use?

    • Inventory technologies used in development: Python, Ruby, JavaScript, Node.js, .NET, etc. This can also refer to operating systems. 
    • Third-party repositories of code: PyPI, RubyGems, NPM, NuGet, PEAR or PECL, and OS repositories such as DebianRepository, CentOS, Ubuntu, AUR and more.
    • Packages: There can be multiple packages from a single repository. How often are packages updated? How often are new packages added?

    Are there update procedures in place?

    • What are the dependencies for every package? How many other packages are pulled from a single package update, and how often are they updated and/or reviewed?

    Who’s managing the verification processes across all these dependencies? What update procedures are in place if there is a personnel change?

    Once these questions have been answered, they should be addressed regularly. The process must be iterative to be successful. 

    When considering attack sources, it’s important to understand the characteristics of repositories teams draw from in order to understand potential areas of weakness. Third-party repositories rarely infect a big project as they are typically found elsewhere (e.g. GitHub) and have thorough code reviews. Official repositories, unofficial mirrors and internal code repositories represent increasing risk potential.

    Random malicious packages, typosquatting, bad code review and phishing developer credentials to hijack accounts are all less technical ploys used by attackers, but equally dangerous. 

    The Software Supply Chain Challenge

    The size, breadth of formats, trusted third-party and open source risks, and misused certificates have made it nearly impossible to stop malware-infected files and objects with today’s signature based, sandbox and vulnerability-based solutions.

    Threats evade dynamic analysis remaining unknown to defenders or lacking details when detected. The explicitness of signature blacklists misses new or never-seen-before code, and sandboxes or dynamic analysis solutions are resource intensive and unable to scale to the breadth and volumes of formats required. Without a way to inspect large or cross platform archives, installers and source code, it’s impossible to extract indicators of compromise. These destructive objects can also impact SOC triage, response and continuous improvement.

    Rising to the Challenge

    How can you address these risks, not only in the software development life cycle (SDLC), but as SOCs address vulnerabilities that may come through other applications and repositories? How do you evaluate and inspect wrappers, digital certificates, executables, scripts, files and libraries, resources, documents and icons?

    It’s vital to establish security practices within SDLC, from training developers on secure coding practices using open source libraries to factoring in detection capabilities including static analysis, dynamic analysis, software composition analysis and manual penetration testing. Implementing a secure SDLC process ensures that the development effort is protected by these activities, augmenting code reviews and infrastructure analysis.

    Teams can extend security controls into the SDLC with technology that provides deep visibility into all aspects of the supply chain, from open source dependencies through continuous integration and delivery of packaged applications. Capabilities that provide greater application security across code development and deployment activities include:

    • File reputation identifies and classifies “known bad” and “known good” files by applying a file’s unique hash against a database of goodware and malware.
    • Automated static analysis detects and classifies unknown malware in code repositories by deobfuscating and decomposing objects and files without executing, and extracting threat indicators. 
    • Dynamic analysis executes files in a safe environment and captures the behaviors and results of malware, allowing the security team to understand what is going to happen.
    • Software composition analysis analyzes applications to detect vulnerabilities, dependencies and licensing requirements in software.
    • Threat intelligence is served up at various stages of the analysis cycle, and represents evidence-based knowledge including context, mechanisms, indicators, implications and action-oriented advice about existing or emerging threats.

    By incorporating these technologies into the SDLC, development teams can obtain continuous validation across the software supply chain. Further automation, implementing new security guidelines and advancing the activities within the existing framework will allow the enterprise to progress its security measures and close gaps in the SDLC and supply chain operations.

    At the same time, it’s important to recognize that threats can still make their way into the software supply chain, and it’s equally important that the SOC remain vigilant about maintaining a clear triage and escalation process. Continuous monitoring of software repositories should be an integral part of good hygiene processes, and incorporated into best practices applied to operational infrastructure, including the SOC. The SOC serves as a centralized management infrastructure to respond, contain and remediate attacks. This includes aggregating and consolidating threat data, building a case for managing a response which can include actions to formulate triage and incident response plans, and moving beyond SIEM.

    — Tomislav Pericin