Category: DevSecOps

  • The Software Extinction Event That Wasn’t

    The Software Extinction Event That Wasn’t

    The world may have just avoided a cybersecurity disaster, with potential impact of CrowdStrike x1000. 

    Imagine if the world’s most pervasive programming language, used in the majority of organizations, services, websites and infrastructure today, was itself made to be malicious?

    Cybersecurity researchers from JFrog recently discovered a GitHub Personal Access Token in a public Docker container… due to the popularity of Python, inserting malicious code that would eventually end up in Python’s distributables could mean spreading your backdoor to tens of millions of machines worldwide!

    – Bruce Schneier, Cybersecurity Guru, NY Times best-selling author

    Picture this: You wake up in the morning, and notice your mobile device has no connectivity. You check your wifi. It’s also out. You turn on the TV to see if there’s any big news – only a couple stations are even functioning and only over the air via antenna. Before you can get oriented, the power goes out.

    You feel like this is the beginning of every disaster movie, but dramatically it’s happening in real life. What’s to blame? An EMP? Terrorism? Nuclear war?

    No, but something potentially just as devastating. A widespread cyberattack that derails nearly every aspect of billions of lives – even digital services based in space are vulnerable.

    What Enabled this Unimaginable Disaster? 

    For the slightly more technical: One wayward Docker container hosted on a public hub, which contained an access token to provide administrator access to Python, PyPI and Python Foundation’s GitHub repos. In the wrong hands, this leaked token would have provided access to the entire Python infrastructure, allowing a nefarious actor to potentially – very sneakily and quietly – take control of systems running Python. 

    In even more simplistic terms: An engineer accidentally posted login/access information online that would allow someone to get a “back door” into many of the world’s computer systems.

    It’s not science fiction. It almost happened. 

    And it could have taken down some of the most critical systems in the developed world. At JFrog we are not prone to hyperbole, but this wasn’t an exaggerated risk. It was a very scary reality. In fact, if not for the dedicated work and investment from security researchers at JFrog, you may have tragically experienced it.

    What is Python?

    Python is a multi-purpose programming language that is designed to be flexible and easy to use. As such, it is a default language and framework for a wide variety of computing systems from website development to software applications to workflow automation to data analysis – even as the default language that enables many of the AI technologies used today. Due to its flexibility and simplicity, it is often one of the first languages used by developers as they learn to code. Software libraries and other packages written in Python are some of the most prevalent open source components used by modern developers.

    Who Relies on Python?

    To understand the potential scope of this attack, it’s important to point out the prevalence of Python in digital systems. It is not a niche use case. The vast majority of critical computing systems utilize Python in some form and could be targets for such an attack. This is where the potential for CrowdStrike’s update impact x1000 becomes a reality.

    • YouTube, Instagram, Facebook, Reddit, Pinterest and other popular sites and services are written in majority or whole in Python (NIT Academy, Probytes). Say goodbye to the majority of services that run and display social media as they black out in an attack.
    • Nearly all machine learning models and AI-enabling software utilize Python to execute. (NIT Academy) Imagine if suddenly everything that runs on AI is taken over by a nefarious actor and trained or retrained with any data they wish.
    • Amazon, Google, Microsoft – All the cloud services and infrastructure from the major cloud players could be blacked out, including all applications and services that run on them. This alone would bring global commerce and interactivity to a halt. (Learnenough, Manpreet Singh)
    • P2P financial services like Venmo, and traditional firms with TRILLIONS of dollars under management (such as JPMorganChase and Goldman Sachs) utilize Python heavily. This attack would have crashed global financial markets, as well as crippling most of the world’s major stock exchanges. (Planeks)
    • Governmental organizations and election systems rely on Python, creating a very real possibility of complete government shutdowns and democratic disruptions. (NIT Academy)
    • Space exploration, satellite-based networks and critical research missions (including NASA, space telescopes, SpaceX and more) all depend on Python, making it very likely all advances in these areas would be orphaned for an extended period (perhaps permanently), leaving a very lonely Mars Rover on the red planet, even dangerously trapping astronauts in space. (Builescu Daniel)

    In this disastrous case, the “end of the world” would come about not with a whimper, but a cyber-bang that crashed most global systems. Identities and information all stolen or missing. Companies destroyed, fortunes wiped out, records deleted, medical care unobtainable, workers unpaid or without access to funds. It could take decades to recover fully.

    Thankfully it didn’t happen. But it was a close call.

    How was the Potential Attack Prevented?

    JFrog is investing heavily not just in products and product development, but also takes our responsibilities to the community very seriously as part of our security research team’s efforts. Part of that ongoing reinvestment effort is to proactively download and examine some of the world’s most popular packages for potential vulnerabilities. Some discoveries – while important – may have a limited blast radius. Some may affect specific operating systems, such as the CrowdStrike issue or maybe certain solution sets such as with SolarWinds. In rare cases, issues are more prevalent, such as the Log4Shell instance. This Python takeover could have made them all look like child’s play.

    From our technical blog on the discovery, we find that JFrog’s secret scanning engines detected a “classic” GitHub token in a binary Docker container hosted in the public Docker Hub repository. The risk with “classic” GitHub tokens is that, unlike newer, more fine-grained tokens, they grant permissions across all repositories the user has access to. 

    PyPI Supply Chain Attack Vector

    The discovered token had permissions to change any piece of code related to Python, from the simple Python executable to any Python package that’s hosted on the official PyPI repository.

    The leaked Python admin token, inside a compiled binary file

    This showcases the “daisy chain” of consequences that could have come about, as an infection moves from one system to the next.

    Why Only Scanning Source Code Guarantees Blind Spots

    In the case of this secret leak, the token was found ONLY in the compiled binary file. It was not discovered to be in the source code for this Docker container, indicating the developer may have temporarily utilized the credentials to build the binary, then cleaned up the code before making it available alongside the binary in the Docker container, forgetting to scrub the artifact. 

    Why does this matter? According to JFrog’s recently published Software Supply Chain State of the Union report, a staggering 27% of companies are utilizing ONLY code scanning to provide software supply chain security, and only 56% of companies are using both code and binary scanning together to secure their supply chain. This means more than half of companies have a glaring blind spot that may not have detected this vulnerability.

    Software Supply Chain State of the Union Report

    What is the Community Saying?

    Thankfully, the gravitas of this scenario has not been lost on the community, spurring all of us to not only provide even-better tools, but also to be more diligent in our approaches.

    What is most interesting, however, is how JFrog found the token. They discovered it in a compiled Python binary file! 🐸📂 It turns out that searching for secrets in the code itself may not be sufficient. Artifacts in the binaries themselves can also pose a serious threat to supply chains!

    Interestingly, this secret wasn’t exposed directly in the code base. That’s something developers, especially those mindful about security, are well aware of. But you might not have known that when .pyc files are generated, they can inadvertently store sensitive information, including secrets. Jfrog (sic) notes that many secrets they uncover are found within binary artifacts. FWIW, I’m not calling out the admin for this. It’s an honest mistake; they wrote an excellent summary and investigated what they could, identifying no indicators of malicious usage.

    The full blog post by the Python engineer who inadvertently made the error may be found here. As you see in his (very transparent and authentic) blog recapping the issue, it is very easy to make an honest mistake that has potentially devastating consequences. And with 20+ million developers globally, mistakes will inevitably happen and secrets will be leaked if proper guardrails are not applied.

    Key Takeaways

    The majority of industries and most digital infrastructure rely on Python in some way, making threats to this core framework highly consequential and possibly more devastating than recent software supply chain attacks.

    The digital world may have just avoided a landmark Python event thanks to researchers’ diligence. Many thanks to them for saving us from consequences we will hopefully never realize.

    Scanning both source AND binaries is essential to ensure your company doesn’t accidentally reveal secrets or introduce vulnerabilities. If you are not currently doing this practice, there is far too much risk to waste time (if you need a place to start, JFrog powers all of our security products with this approach, led by the same research team that found this secret).

    Supporting the community is key. There are earth-shaking discoveries that need to be shared with the community and fixed quietly before nefarious actors can take advantage. We should learn from the close calls to ensure none of us is next in the headlines, and build better tools and automation to safeguard our companies and posture.

    Huge thank you to the JFrog team for finding this before it could have been much much worse, and as always thank you to the volunteer team running Python’s infrastructure for handling this so well. These things can and do happen to anyone, and their response was world-class.

    Let’s find, fix and fortify together.

  • Securing the DevOps Pipeline: Tools and Best Practices

    Securing the DevOps Pipeline: Tools and Best Practices

    The DevOps pipeline refers to the set of practices and tools designed to automate and integrate the processes between software development (Dev) and IT operations (Ops) teams. This pipeline facilitates continuous integration (CI), continuous delivery (CD) and rapid deployment practices, aiming to shorten the development life cycle and provide continuous delivery with high software quality. 

    The DevOps pipeline typically includes planning, coding, building, testing, releasing, deploying, operating and monitoring, enabling teams to develop and deploy software more efficiently and with fewer errors.

    By automating workflows and integrating various development and operational activities, the pipeline supports the rapid iteration of software releases. This not only accelerates the pace of software development but also ensures that quality and performance standards are met throughout the application’s life cycle.

    Due to the critical nature of the DevOps pipeline, security is becoming a top priority. Some of the most devastating cyberattacks in recent years were related to breaches in DevOps pipelines, which led to global-scale supply chain attacks.

    The Need for Security in DevOps

    DevOps, by its very nature, encourages speed and efficiency. However, this drive for rapid software development and deployment can sometimes overlook the critical aspect of security. That’s why it’s essential to weave security into the fabric of the DevOps pipeline, an approach known as DevSecOps. By doing so, we can ensure that security isn’t an afterthought but an integral part of the development process.

    Incorporating security into the DevOps pipeline offers multiple benefits. First, it helps identify and rectify security vulnerabilities early in the development process, reducing the risk of security breaches. It also fosters a culture of shared responsibility, where the development, operations and security teams work together toward the common goal of creating secure software.

    Tools for Securing the DevOps Pipeline

    Securing the DevOps pipeline requires a set of tools that align with the DevOps philosophy of automation and continuous improvement. Let’s explore some of these tools.

    Static Application Security Testing (SAST)

    SAST, also known as white-box testing, is a tool that helps locate security vulnerabilities in the codebase without executing the application. It makes it possible to identify issues early in the development process, even before the code is run.

    SAST works by analyzing the source code, bytecode or binary code, looking for patterns that might indicate a security flaw. The strength of SAST lies in its ability to pinpoint the exact location of the security vulnerability in the codebase. This makes it easier to fix the issue, thus saving time and resources.

    Dynamic Application Security Testing (DAST)

    DAST, a type of black-box testing, involves testing the application in its running state. It’s designed to find security vulnerabilities that can’t be spotted during static analysis.

    DAST works by simulating attacks on the application and observing its response. It can identify security issues like cross-site scripting (XSS), SQL injection and security misconfigurations. The beauty of DAST is that it views the application from an attacker’s perspective, thus providing a real-world assessment of its security. However, DAST can’t identify vulnerabilities in the source code and relies on the application being in its running state for testing.

    Software Composition Analysis (SCA)

    Software composition analysis (SCA) is designed to provide visibility into the open source components used in the software. It helps identify any security vulnerabilities, licensing issues or outdated libraries in the open source components.

    SCA works by creating a bill of materials (BOM) that lists all the open source components used in the software. It then compares this list against various databases that track open source vulnerabilities. If a match is found, SCA flags the component, allowing the developers to take necessary actions.

    Endpoint Detection and Response (EDR)

    Endpoint detection and response (EDR) is a cybersecurity technology that addresses the need for continuous monitoring and response to advanced threats. EDR tools continuously monitor endpoint and network events and record the information in a central database where further analysis, detection, investigation, reporting, and alerting take place.

    EDR is like a surveillance system for your DevOps pipeline. It keeps an eye on endpoint activity, and if it detects a potential threat, it alerts developers and security staff. EDR is not just about detection; it also provides tools for responding to security incidents. It helps isolate affected systems, block malicious activities and even roll back systems to their pre-attack state.

    Container Security Tools

    In a DevOps environment that increasingly relies on containerization, container security cannot be overlooked. Container security tools provide a means to enforce security policies and protect containerized applications.

    These tools offer functionalities like image scanning, runtime protection and network segmentation. Image scanning helps identify vulnerabilities in the container images. Runtime protection monitors the container’s activities in real-time, looking for any signs of compromise. Network segmentation allows for isolating containerized applications, thus limiting the blast radius in case of a security incident.

    Best Practices for Securing the DevOps Pipeline

    Integrate Security Early (Shift Left)

    The concept of ‘shifting left’ in DevOps refers to integrating security measures early in the development process. Instead of treating security as an afterthought, it becomes a part of the software development life cycle from the get-go.

    Implementing security early allows us to identify and address potential vulnerabilities before they become a significant issue. It saves us time and resources by preventing security breaches instead of responding to them. Besides, it fosters a proactive approach to security, emphasizing prevention over remediation.

    SAST and DAST are useful in integrating security early. SAST tools analyze code in its static state to identify vulnerabilities, while DAST tools test applications in their running state to detect security flaws.

    Automate Security Processes

    Automated security scans can be incorporated into the continuous integration/continuous delivery (CI/CD) pipeline. Each time a code push occurs, these scans are triggered, allowing for immediate feedback on potential security issues. Automated scans can detect a broad range of vulnerabilities, from code defects to configuration errors.

    Configuration management tools like Chef, Puppet, Ansible and Terraform can help manage configurations and enforce security policies. They ensure that all systems are configured correctly and consistently, reducing the risk associated with manual configuration.

    Regular Security Audits and Compliance Checks

    Regular security audits and compliance checks are essential in securing the DevOps pipeline. They provide an opportunity to review and update security measures, ensuring they are effective and up-to-date.

    Security audits should be comprehensive, covering all aspects of the DevOps pipeline. They should evaluate security controls, identify potential vulnerabilities, and recommend remedial measures. Compliance checks ensure that security practices align with relevant regulations and standards.

    In conclusion, securing the DevOps pipeline requires a holistic approach that integrates security into every aspect of the development process. It involves shifting security left, automating security processes, conducting regular audits and compliance checks and fostering a culture of collaboration.