devopsy.kinsta.cloud

Tag: software security

Squaring the Circle: How to Make Public APIs Private
APIs have become an integral part of enabling seamless data exchange and functionality between systems and businesses. In fact, APIs are trebling every year. Public APIs are, by their nature, designed to be open and discoverable. By definition, this also means that APIs are open to attacks from the networks. This is why APIs have become the number one attack vector.

APIs becoming the top attack vector makes sense. All our other business apps are hidden behind private networks–usually MPLS (AWS Direct Connect, Azure ExpressRoute, etc.), VPN or zero-trust network access (ZTNA). But APIs are exposed to the public Internet.

Day 2 API Security Solutions and Snowflakes

This network exposure is why there are so many ‘Day 2′ API security solutions – WAFs, API security platforms, IDS, etc. These solutions do a great job of blocking threats that have already been identified, yet there are successful API attacks each day. Why?

Each API is unique and evolves rapidly–constant changes in business logic, authentication, authorization and configurations. APIs are snowflakes. The result is that many API attacks are effectively zero-day attacks–novel attacks that exploit recent and unique changes to specific APIs.

A Difficult Race to Win

Network exposure and snowflake characteristics are why API-related vulnerabilities multiply each day.

The ‘snowflake’ issue means many attacks are unknown to the Day 2 solutions – they attack vulnerabilities that didn’t previously exist. So, after the novel attack, we race to apply a fix. Who are we racing against? Every attacker on the internet. The API is public–exposed to the networks–not like our other business apps, which are private behind an MPLS, VPN or zero-trust solution. Racing against attackers on the internet is difficult–and coming second may result in a service outage, data breach or customer compromise.

Why Not Avoid the Race?

So why not make the APIs private, avoid exposing them to the internet, put them behind MPLS, VPN or ZTNA, like we do for all our other business apps? API clients are too distributed and dynamic; there is no good way to manage agents or dedicated circuits across all those API clients. Those solutions were not made for APIs–it is like fitting a square peg in a round hole.

Zero-Trust API Networking Needs to be Built for APIs

To make our APIs private like the rest of our business apps, we must first meet the business need. API providers need to achieve the business velocity and agility they require. On the technical side, we need a secure networking solution which is built to manage the distribution of the API clients, and assume that there will always be snowflake-like vulnerabilities.

Let’s look at an open source solution for zero trust APIs which is purpose-built for those business, technical and operational requirements:

There are usually three reactions to the diagram above:
1. OK, tell me how it works. See below.
2. Looks complex. Nope. All software, spun up in minutes, centrally managed.
3. Seems like a proprietary solution for users accessing their applications. Yes! It uses principles similar to the user-to-app zero-trust world but secures APIs.
How Zero-Trust API Networking Works

Here are the four pillars of zero-trust API networking:
1. Eliminate public access. Your API clients will use private IPs or domains to reach your APIs. If your company name is Acme, then your clients might now use api7.acme.unreachable (you make up any domain you want). Envision an attacker trying to interrogate *.acme.unreachable to find snowflake vulnerabilities. Not possible because it is not a public domain—your API server is no longer reachable.
2. Identity, authentication, authorization, mutual TLS. Authorized API clients do need to access your APIs from the internet. Your zero-trust API overlay network is accessed from the internet only with X.509-certificate cryptographically authenticated identities, continuous authentication and least privileged access authorization via mTLS links. You move the policy enforcement point (PEP) from your DMZ to your API clients. Your API clients add a few lines of zero-trust networking code to enable this model. This code replaces VPNs, MPLS, etc. (there are non-code options too). For example, some of your API clients likely use Resty, a popular open source REST client library for Golang. Their existing Resty code adds a few lines of zero-trust API networking code:
3. Outbound-only. You eliminated the inbound access. Your server will instead open identified, authenticated and authorized sockets to your zero-trust API overlay network (you’ll close all your inbound ports and only enable outbound ports to specific overlay network router IPs or to your x.acme.unreachable domain).
4. Management. API networking security is not very valuable without simple management. The zero-trust API overlay cloud is software-only and cloud-native. You spin up your private API clouds in minutes, anywhere, without deploying hardware. If an API client is compromised, one click or automated action removes its access. Network usage for each client/API is in your web console or management dashboard.
The above is based on the open source OpenZiti zero-trust networking platform, but any zero-trust API platform should provide similar functionality. Specifically, it should enable:
- Speed. Securing APIs but slowing down business or complicating operations is not the trade-off most organizations want. Security and speed is the goal.
- Minimize the threat surface. Close all inbound ports in front of your API servers. Use private, unreachable addresses or domains to minimize your attack surface and stop racing against network-based attackers.
- Extensibility. Software-only, cloud-native architectures. Preferably open source based.
- Operations excellence. Centralized visibility, controls, policies and identities. Rich telemetry data. Performance-optimized overlay networks.
- Zero-trust security. Identity-based networking, continuous authentication, strong authorization, mutual TLS (mTLS), encrypted delivery, microsegmentation.
- Flexibility. API clients add a few lines of code to get zero-trust API networking. Other API clients use API-specific agents or DNS to access the overlay, if desired, while your API server/gateway is still private.
- Recovery. Assume breaches. Implement microsegmentation, prevent data exfiltration and provide simple controls to immediately remove threats.
Zero-Trust API Networking

While APIs will always be snowflakes, they no longer need to be our only business app that is exposed to the networks. By minimizing the attack surface, without compromising business velocity and API client experience, we can finally mitigate against our greatest API security risk. This also simplifies our day two API security, the solutions that inspect API queries after they enter our DMZ.

Rather than trying to filter the internet, day two solutions can focus on detecting authorized API clients that may have been compromised. Similarly, our operations teams are freed to focus on sessions from authorized API clients rather than wading through oceans of data from port scanning, DDoS attacks, unauthorized API queries and attempts to interrogate public APIs to find new snowflake vulnerabilities.
February 14, 2024
OX Security Optimizes DevSecOps to Improve Application Security

OX Security this week updated its application security posture management (ASPM) platform to enable DevSecOps teams to instantly identify applications running in production environments that are running code that could be potentially exploited.

The ASPM platform from OX Security leverages large language models (LLMs) to identify vulnerabilities. The latest version adds an ability to confine the number of alerts generated to instances of code that can actually be exploited.

OX Security CEO Neatsun Ziv said that approach reduces the number of alerts that would otherwise be generated by 99% to enable DevSecOps teams to focus their limited resources on the 1% of alerts that represent an actual risk to the business. OX Security achieves that goal in part because its scanners are integrated with source control, continuous integration/continuous delivery (CI/CD) platforms, registries and cloud computing environments.

The overall goal is to reduce the amount of manual effort currently required to consistently maintain DevSecOps best practices.
The primary reason many vulnerabilities are never addressed is that DevSecOps teams are simply overwhelmed by alerts. Most of the alerts generated by a cybersecurity team are not immediately relevant because either the vulnerability is not present in the application running in a production environment or the application doesn’t connect to the internet.

The OX Security platform makes it possible to identify application security issues that need to be prioritized because it has been confirmed a vulnerability exists that can be exploited, said Ziv.

No matter how precise the alert process becomes, a DevSecOps team will still need to manage it, added Ziv. However, as the alerts generated become more relevant, it should become easier to bridge the historic divide between application developers and cybersecurity teams. Too many developers routinely ignore vulnerabilities identified by cybersecurity teams simply because previous efforts to identify the actual severity they represent have ultimately been a waste of their time.

The challenge just about any organization encounters when trying to better secure their software supply chains is, not surprisingly, getting application developers to buy into the DevSecOps process. Most developers are going to actively resist any approach to remediating vulnerabilities that takes time away from building new applications. Most of them only allocate about 10% of their time or less to remediating vulnerabilities. As such, it’s incumbent on a DevSecOps team to find ways to address application security issues in ways that don’t adversely impact developer productivity.

Organizations could mandate that vulnerabilities be remediated. In fact, there is a growing body of regulations that will require organizations to do just that unless they can prove the vulnerability in question cannot be exploited. The only way to effectively make that case will be to rely more on automation to document that the code running in a production environment is actually secure.

In the meantime, developers should expect to see their code scrutinized more than ever. Of course, the better part of valor is to ensure that as many vulnerabilities as possible are never introduced.

February 1, 2024
Vulnerability Management for DevOps Teams: A Practical Guide

Vulnerability management is a process that includes the identification, categorization, prioritization and resolution of security vulnerabilities in software systems. It involves a continuous cycle of protection to ensure that your digital assets are secure and your operations are running smoothly.

With the rapid evolution of technology, fast-moving cloud-native systems and evolving cyber threats, it’s critical to stay ahead of the game. The goal is to close the gap between the discovery of vulnerabilities and their resolution, thereby minimizing the window of opportunity for potential attacks.

Vulnerability management is not just about finding and fixing vulnerabilities. It’s also about understanding the nature of the vulnerabilities and the potential impact they could have on your network and systems. This understanding informs the prioritization process, helping teams decide which vulnerabilities need immediate attention and which ones can be addressed later. See this blog post for an in-depth review of vulnerability management.

Why is Vulnerability Management Important for DevOps Teams?

With the DevOps approach, development and operations teams work together throughout the software life cycle. This collaboration results in faster development times and more reliable software. However, the fast development cycle also introduces new security challenges. This has given rise to the DevSecOps paradigm, where developers, operations teams, and security experts collaborate during all stages of the development lifecycle.

Here are a few reasons vulnerability management is a critical part of DevSecOps practices:

Early Detection of Vulnerabilities

DevOps teams are under constant pressure to deliver high-quality software quickly. However, the speed of delivery should not compromise the security of the software. Vulnerability management helps DevOps teams identify vulnerabilities early in the development process. This early detection allows teams to address security issues before they become serious problems.

Early vulnerability detection is not just about finding vulnerabilities; it’s also about understanding the potential impact of those vulnerabilities. It involves analyzing the vulnerabilities to determine their severity and the risk they pose to the system. This analysis informs the prioritization process, ensuring that the most critical issues are addressed first.

Building Continuous Security Into CI/CD Pipelines

Integrating vulnerability management directly into continuous integration/continuous deployment (CI/CD) pipelines is essential for maintaining continuous security. CI/CD pipelines automate the steps in software delivery, from code integration to deployment, ensuring faster and more consistent releases. By embedding vulnerability management into these pipelines, DevOps teams can automatically scan for and address security vulnerabilities at each stage of the software development life cycle.

Incorporating vulnerability management tools into CI/CD pipelines allows for automatic scanning of code repositories, container images, and even infrastructure-as-code (IaC) configurations for known vulnerabilities. This integration means that any security issues are identified as soon as code is committed or containers are built, not just at the deployment stage. The advantage of this approach is twofold: It reduces the risk of deploying vulnerable software, and it allows developers to fix issues in their context of development, making the process more efficient.

Cost-Effective Development and Operations

Implementing vulnerability management can lead to cost savings for DevOps teams. By identifying and resolving vulnerabilities early in the development process, teams can avoid the high costs associated with fixing security issues after the software has been deployed. In addition, vulnerability management can help prevent costly security breaches that could damage the company’s reputation and result in financial loss.

Moreover, vulnerability management can contribute to more efficient operations. By systematically managing vulnerabilities, teams can avoid wasting time and resources on low-priority issues. Instead, they can focus their efforts on addressing the most critical vulnerabilities.

Enhancing Cross-Team Collaboration

Vulnerability management can also enhance cross-team collaboration within the DevOps environment and between DevOps and security teams. By providing a clear framework for identifying, categorizing, prioritizing and resolving vulnerabilities, it fosters a shared understanding of security issues among team members. This shared understanding can facilitate communication and cooperation, ultimately improving the software product’s security posture.

Strategies for Integrating Vulnerability Management Into DevOps Processes

Adopt a ‘Shift-Left’ Approach

The ‘shift-left’ approach is a strategy that involves shifting the security measures to the early stages of the DevOps pipeline. Traditionally, security checks and vulnerability assessments were conducted toward the end of the development cycle, often leading to late discovery of security flaws and vulnerabilities.

By shifting these security measures to the left, i.e., to the earlier stages of the development cycle, DevOps teams can catch and fix security vulnerabilities early on. This approach aligns well with the DevOps philosophy of early error detection and continuous improvement.

Automation of Vulnerability Scanning and Detection

Automation is at the heart of DevOps. It allows for rapid, reliable and consistent processes, freeing up the team to focus on what they do best – delivering quality code and features. Automation should also extend to vulnerability management.

Automating vulnerability scanning and detection means integrating security tools into the DevOps pipeline that can automatically scan the code, detect vulnerabilities and even suggest fixes. This approach ensures that security checks are not skipped or forgotten and that vulnerabilities are dealt with promptly.

Security as Code

Another strategy for integrating vulnerability management into DevOps processes is embracing the principle of security-as-code. This principle treats security infrastructure and configurations as code, allowing them to be part of the DevOps pipeline.

By adopting security-as-code, DevOps teams can ensure that the security configurations and infrastructure are always up-to-date and follow the best practices. Like the rest of the code, security configurations can be version-controlled, reviewed, tested and automatically deployed.

Collaboration Between Security and DevOps Teams

One of the key tenets of DevOps is collaboration between different teams. In a DevSecOps paradigm, this collaboration extends to the security team as well. Integrating vulnerability management into DevOps processes fosters close collaboration between the security and DevOps teams.

The security team can provide valuable insights into the security landscape, vulnerability management tools and best practices. On the other hand, the DevOps team can ensure that these insights and tools are integrated into the DevOps pipeline.

Continuous Monitoring and Feedback Loop

Continuous monitoring is crucial for effective vulnerability management. Continuous monitoring ensures that the security state of the system is always known and that any changes or vulnerabilities are promptly detected. The feedback loop ensures that the insights gained from monitoring and vulnerability assessments are fed back into the development process, leading to continuous improvement in security.

Regular Training and Awareness

Finally, regular training and awareness are crucial for integrating vulnerability management into DevOps processes. DevOps teams should be trained in security best practices, threat landscape, and use of vulnerability management tools.

Regular training ensures that the team stays up-to-date with the latest security trends and vulnerabilities. Awareness sessions can help the team understand the importance of security and vulnerability management, encouraging them to take an active role in securing the system.

Conclusion

In conclusion, integrating vulnerability management into DevOps processes is not only possible but crucial for secure and reliable systems. Adopting a shift-left approach, automating vulnerability scanning and detection, embracing security-as-code, fostering collaboration, implementing continuous monitoring and feedback and ensuring regular training and awareness are some of the strategies that can help achieve this integration.

Remember, vulnerability management is an ongoing process that requires commitment, collaboration and continuous improvement.

December 7, 2023
How Encryption Innovation Will Power Cloud Privacy for Developers in 2024

For developers, the surge in cloud computing adoption has brought both new opportunities and challenges. On the one hand, developers are turning to cloud services for data storage, processing and accessibility due to their flexibility, convenience and cost-effectiveness. And because the cloud takes care of these infrastructure and operational concerns, it’s allowing developers to focus on coding, innovation and delivering value to end-users. It also allows companies to scale more easily when their market grows or when new hardware becomes available.

However, this paradigm shift has led to the accumulation of vast volumes of sensitive data on remote cloud servers—including personal, health, financial and proprietary business information. As the volume of sensitive data on the cloud continues to grow, it has naturally raised mounting security concerns, from data breaches and cyberattacks, to unauthorised access and the complexities in complying with specific industry regulations, such as GDPR or CCPA.

For some, having to rely on the cloud provider to maintain robust security practices and data protection has also led to hesitancy in sharing data, both within and between organizations. This not only limits the potential for data-driven collaboration, research, and innovation—essentially halting progress in fields like health care, science, and social sciences—but businesses of all types and sizes could miss opportunities for cost savings, scalability, and efficiency if they’re not willing to adopt innovative technologies. There are also cases when companies are forbidden to do something by law because of privacy, again causing roadblocks.

All of this has meant that up until now, maintaining data privacy while making the most of cloud computing has been somewhat of a balancing act for developers and companies. Companies want to fulfill their clients’ needs, while developers have to find technical solutions that provide the level of privacy required.

Traditional Encryption methods Vs. Fully Homomorphic Encryption (FHE)

Currently, there are several technologies and practices employed to protect data in the cloud, one of the most commonly used being traditional data encryption, which provides protection only during data transmission and storage.

However, when data is processed, it must be decrypted—at which point it can expose data to eavesdropping, rendering it susceptible to potential threats. The aim of FHE is to ensure that the computations can be carried out on cloud-based machines while the data itself remains encrypted. With FHE, the data is really encrypted end-to-end: not only is it encrypted during transport and storage, but also it is encrypted during computations. It is only when the data or results are sent back to the client side that they are decrypted.

This means that in the event of a data leak, the exposed information would be rendered meaningless without the decryption key, which stays securely in the hands of the owner. In other words, neither the cloud provider nor any potential spyware on the cloud machine would have the capability to decrypt the encrypted data.

This approach, which ensures sensitive data remains encrypted throughout its entire life cycle on the cloud, even during processing and analysis, significantly enhances data security and privacy.

Current Limitations and FHE’s Future Developments

While developers using traditional cloud services previously had to decide whether to use the cloud with some privacy concerns or forgo its benefits altogether, with FHE we are starting to see a possible shift in attention from users’ reliance on the cloud provider to a more user-controlled approach.

As FHE becomes more widespread, users gain the ability to make their computations over any cloud provider, even those they may not fully trust. This means the upfront costs of building new cloud platforms can be lowered by providing only FHE computation.

While the encryption of data ensures that computations are secure and no unauthorized access or breaches can occur, this new tech is not without its limitations. For the enhanced security FHE offers, there may come a trade-off of longer execution times. Presently, we are mainly reliant on CPUs for computations. However it’s worth noting that as the technology advances and hardware accelerators become available, the execution time is expected to improve in the future. In fact, we anticipate that by 2026, FHE implementations will see a considerable increase in speed, potentially by factors of 100x or 1,000x, expanding the applicability of FHE to a wider range of use cases.

Other benefits currently being worked on include usability. FHE had a reputation of being very complex and so used to be reserved for those with cryptography PhDs. By developing open source tools and extending the community’s knowledge around FHE, the aim is to democratize FHE so that anyone can use it, companies can experiment free of charge and check if it fulfills their needs.

Looking to the future, privacy will likely remain high on the agenda for almost everyone. Thankfully, with new encryption technologies rapidly developing in terms of speed and usability, developers look set to achieve this, ensuring their cloud-based applications and services are as secure as possible.

December 1, 2023
CISA, NSA Issue Supply Chain Security Guidance Report

The National Security Agency (NSA), Office of the Director of National Intelligence (ODNI) and Cybersecurity and Infrastructure Security Agency (CISA) have issued guidance to assist software developers and suppliers to shore up software integrity and security across various stages of the software lifecycle.

The report, Securing the Software Supply Chain: Recommended Practices for Software Bill of Materials Consumption, was developed by the Enduring Security Framework (ESF) Software Supply Chain Working Group.

The report offered recommended practices as a framework for assessing and improving security practices concerning software development, release, updates, vulnerability notifications and mitigation.

The guidance also addressed the escalating cybersecurity threats targeting software supply chains, which have exposed vulnerabilities susceptible to exploitation by state adversaries.

It highlighted potential risks such as design flaws, integration of vulnerable third-party components, infiltration of supplier networks with malicious code and malware injections in deployed software.

To combat these threats, the report underscored industry best practices, emphasizing open source software management and software bills of materials (SBOM).

Specific areas covered include SBOM consumption, life cycle, risk assessment and operational implementation, ultimately promoting enhanced transparency and risk management throughout software life cycles for organizations.

Alex Kreilein, vice president of product security at Qualys, said while the concept of an SBOM is not new, organizations now have concrete and foundationally important guidance on how to make operational use of SBOMs while developing, managing or consuming them.

“The guidance supplied by the Enduring Security Framework reduces the complexity of SBOM management, making it accessible to everyone for the first time,” he noted. “SBOMs are part of a healthy software ecosystem, but are by no means the only factor.”

He explained that while the SBOM is an effective output, it cannot be trustworthy if the system that created it cannot validate the provenance of the product.

“It’s critical for organizations to understand the whole supply chain from code through pipeline to production,” Kreilein said.

He added that this is why The Linux Foundation and Google started the Supply-chain Levels for Software Artifacts, or SLSA framework, as a practical set of methods to action NIST recommendations in the Secure Software Development Framework (SSDF).

John Gallagher, vice president of Viakoo Labs at Viakoo, called the report a “major step forward” in providing an overall framework and transparency for how SBOMs are constructed, maintained and used.

“One of the key ways this helps relieve concerns over software supply chain security is by providing clear guidance on the risk scoring mechanisms,” he said.

This augments other software supply chain efforts (though it doesn’t replace them) in areas like the use of SBOMs in SaaS environments or version control mechanisms for SBOMs.

He noted two significant points on implementation that come out of this framework; first is that it gives guidance to departments like procurement as to what they should be requiring from their vendors, and second it gives a framework for solution providers to base their consumption and generation of SBOM and Vulnerability-Exploitability eXchange (VEX) data on.

“While this document does not address the methods of remediation, it does help to more quickly identify software supply chain vulnerabilities so that they can be addressed early on and ideally before the widespread use of that vulnerable code,” Gallagher said.

He noted that the more vulnerabilities that are caught early or before they are broadly used the later stage of remediation is either much easier or not needed.

Anthony Tam, manager of security engineering at Tigera, said when vulnerabilities have been identified, it is important to prioritize them based on severity and potential impact.

“This allows organizations to allocate their resources more effectively and focus on the most critical vulnerabilities first,” he explained.

Prioritization can be done by using a risk-based approach, considering factors such as the likelihood and potential impact of a vulnerability being exploited.

“By prioritizing vulnerabilities, organizations can ensure that they are addressing the most critical security risks first and reducing the overall risk to their software system,” Tam noted.

November 21, 2023
Survey Surfaces Software Supply Chain Security Challenges

A survey of 268 security leaders and 252 developers published this week found that while there is general agreement on the importance of software supply chain security, there is a significant disconnect on what is required to achieve that goal.

Conducted by the Harris Poll on behalf of Chainguard, a provider of curated container images for building secure software, the survey found 92% of developers acknowledged software supply chain security is at least very important, with 39% noting it as absolutely essential. Similarly, 96% said effective software security practices are important to meeting government or regulatory requirements, with 93% noting effective software security as a critical component of their threat and risk mitigation strategy.

Overall, most security leaders (70%) and CISOs viewed software supply chain security as a top priority. However, only slightly more than half of developers (52%) agreed. Nearly three-quarters of developers (72%) said they are very security-conscious, but only 50% of security leaders rated developers as very security-conscious.

Roughly two-thirds of security leaders (69%) and developers (64%) agreed that lack of communication and collaboration between developers and security teams is a problem. More than three-quarters of security leaders (77%) and more than two-thirds of developers (68%) agreed that the need to prioritize security causes tension between their teams. Nearly three-quarters of developers said the work/tools their security team requires them to use interferes with their productivity and innovation. Tools in place included software bills of materials (SBOMs) (40%), and nearly half are implementing software supply chain security frameworks such as Supply-chain Levels for Software Artifacts (SLSA) (47%) and the Secure Software Development Framework (47%) defined by the National Institute of Standards and Technology (NIST).

The report also found that 36% of security leaders and 34% of developers viewed an overwhelming number of scanner false-positive vulnerability alerts as being among the biggest software supply security obstacles an organization faces.

Most security leaders (74%) and developers (85%) believed that prioritization of software supply chain security would increase over the next five years.

However, many developers don’t believe enough security teams understand core software. Developers said security teams are generally familiar with open source software libraries and projects (61%), source code repositories and source code management systems (60%) and software build tools (59%), but only 43% said security teams are very familiar with container images.

On the plus side, many security leaders and developers understand that best practices and tooling in software security to drive certain business outcomes is essential, including customer retention (43% and 40%), meeting or satisfying procurement contract obligations (36% and 32%), fewer breaches or compromises (34% each), and developer/engineer productivity (32% and 34%).

Kim Lewandowski, chief product officer at Chainguard, said the survey makes it apparent that while progress toward securing software supply chains is being made, much work still needs to be done as silos between the developers and security leaders who are ultimately responsible for application security persist. While more organizations than ever have embraced DevSecOps best practices to achieve that goal, the cultural and technical challenges many organizations encounter are often underestimated, she noted.

The issue, as always, is that given the limited resources available, many organizations, despite their best efforts, are clearly still finding it difficult to stay current with updates to remediate vulnerabilities as they become available.

November 9, 2023
3 Steps to Secure Your CI/CD Pipelines

More and more software teams are waking up to the fact that their CI/CD pipelines are prone to risk. These pipelines often interface with various toolsets and automation, which may be prone to misconfigurations and abuse. This is not to mention the potential for insecure code being pushed through this funnel and external factors that threaten to circumvent the pipeline.

In recent years, these risks produced some CI/CD tooling breaches—concerns that gained high-level attention recently after a joint CISA and the NSA alert about the importance of securing CI/CD pipelines. “This is something that moves the tectonic plates of the industry,” said Daniel Krivelevich, former founder of Cider Security and current CTO of AppSec at Palo Alto Networks.

I reconnected with Krivelevich, the ‘godfather of CI/CD security,’ to reevaluate the CI/CD threat landscape and understand how IT should respond. According to Krivelevich, we’re now seeing more maturity around CI/CD security, both in terms of recognizing that these risks exist as well as emerging frameworks to help engineering teams respond. He shared a general three-step framework organizations can use to secure the CI/CD pipeline and surrounding areas, which we’ll explore below.

The Evolving Threat Landscape

The traditional approach to application security (AppSec) used to be simpler. Defenders could implement more standard controls to make their code more secure, said Krivelevich. However, throughout the years, the software engineering process has transformed dramatically. It’s become faster, dynamic and reliant upon third-party frameworks. “The entropy of the engineering ecosystem is very high,” he said.

This paradigm shift has introduced a slew of new risks, which requires rethinking since most traditional security solutions deal solely with AppSec. “Software engineering is a big part of what makes the business tick,” said Daniel. Therefore, having a common language and understanding of this area is crucial to respond to evolving risks that threaten business operations.

A Three-Step Framework For CI/CD Security

Krivelevich is involved with creating the OWASP Top 10 CI/CD Security Risks; therefore, he has excellent insight into the types of vulnerabilities that most engineering practices and their CI/CD pipelines are facing. His three-step framework for CI/CD security, which Palo Alto Networks has applied in practice, looks to holistically secure not only the pipeline itself but also internal and external factors.

1. Security in the Pipeline (SIP)

SIP can be thought of as the traditional AppSec problem space. This involves ensuring developers are not shipping anything inherently insecure into production. Therefore, whatever code flows through the CI/CD pipeline must be vetted to ensure it has no security flaws or misconfigurations. Of course, this is easier said than done, given that teams are juggling more technologies and frameworks than ever, each of which has different knobs and requires unique security scanning capabilities.

2. Security of the Pipeline (SOP)

This area has to do with the security of the software delivery chain itself. Attackers may attempt to breach these pipelines by running malicious code within the pipeline or stealing credentials to CI/CD tools to harm the system. Poisoned dependencies, access management holes or insufficient credential hygiene could all be doorways to disrupting a CI/CD process. Therefore, SOP focuses on the security posture of the actual systems and tools that compromise the software delivery chain.

3. Security Around the Pipeline (SAP)

However, even if SIP and SOP are secure, what if hackers can bypass the pipeline altogether? That would render even a perfect set of controls null and void. Therefore, SAP is necessary to ensure that attackers can’t push code directly into production. It’s good to see the bigger picture so that everything that reaches production actually originates from the pipeline.

To deploy SAP effectively, Krivelevich recommended leveraging both detective and preventative measures. You need detective measures to issue alerts when anomalies occur, such as an artifact deployed by a human account instead of a service account. You also need preventative measures that set the proper branch protection rules across all repositories. For instance, this could be a rule that no single developer can publish to production unless approved by two reviewers.

The Role of AI in Software Security and CI/CD

By using both detective and preventative measures, you can detect when settings are disabled or abused and have better configurations across all stages of the software delivery chain. But what other tactics can be used to secure CI/CD? For example, does machine learning have any merit in this area?

Well, Krivelevich described AI as “both a blessing and a curse.” Attackers are already using AI to operate much faster and in a much more agile way. And unfortunately, he viewed the engineering ecosystem as the new path of least resistance. “I expect the power of AI will be used by sophisticated attackers who are motivated to get access to the ‘crown jewels’ by abusing the engineering ecosystem,” he said.

On the defensive side, Daniel doesn’t see a specific opportunity for deploying AI to protect CI/CD. That being said, there’s a lot of potential for security builders to utilize AI to advance their efforts and accelerate the pace at which they detect anomalies, which could help respond more quickly to emerging threats.

Final Thoughts on CI/CD Security

The recent chatter around CI/CD threats appears to be garnering more attention around securing this niche area of agile software development. To thwart CI/CD attacks, it will be necessary to assess the settings of systems, understand their technical natures and analyze them against an attacker’s perspective, Krivelevich explained. Therefore, understanding what realistic attacks can be carried out will help build measures to correct these gaps.

In doing so, the SIP, SOP and SAP descriptions serve as a solid reminder of what needs to be addressed when attempting to secure the software delivery process and the supply chain at large. Code that goes through the pipeline needs to be free from vulnerabilities, and the channel itself must be vetted for gaps that could be abused. We also need to consider the possible existential threats if the entire process can be undermined.

September 26, 2023
Summit Highlights Open Source Software Security Progress

More than a year after the Open Source Security Foundation (OpenSSF) summit kicked off an initiative to better secure open source software-based software supply chains, a lot of progress has been made—but much work remains to be done.

The OpenSSF this week hosted a Secure Open Source Software (SOSS) Summit 2023 event in Washington, D.C., during which it made available a Secure Open Source Software Vision Brief 2023. The brief describes the various efforts being made to improve open source software security, including, for example, providing maintainers of open source software projects with free DevSecOps tools.

This week, the Cybersecurity and Infrastructure Security Agency (CISA) also published an Open Source Software Security Roadmap that defines its role in providing more visibility into how open source is being used and the associated risks.

However, it’s not clear how much maintainers of these projects are embracing DevSecOps workflows. Many projects are led by a handful of maintainers that often lack the time and skills required to ensure the components they employ have not, for example, been compromised by malware that might find its way into downstream applications.

Sonatype CTO Brian Fox said the focus of the summit’s participants will be on education and securing public code repositories. While there is a lot of focus on maintainers, he added that there is not enough attention to all the organizations that have not updated open source components within a software build with known vulnerabilities. Many IT teams are reluctant to upgrade components for fear of breaking applications with complex dependencies, but the amount of code that is potentially insecure is massive. “There’s a huge mountain to climb,” he said.

The challenge is determining which code to upgrade first in a way that will have the most impact in terms of improving application security, added Fox.

Shawn Ahmed, chief product officer for CloudBees, also noted that organizations should evaluate the DevSecOps processes being applied by maintainers of open source software projects. CloudBees, for example, reviews the plugins provided by the open source community for the Jenkins continuous integration/continuous delivery (CI/CD) platform for vulnerabilities every month, he noted.

It’s also critical for organizations that consume open source software to contribute code to improve the security of open source software, added Ahmed. Too many organizations are focused on open source software security without devoting resources that would help resolve the issue. he added. “We need to make sure open source software is not a vulnerability threat,” said Ahmed.

Moran Ashkenazi, chief strategy officer (CSO) and vice president of security engineering for JFrog, said on the plus side, there is a lot more awareness of open source software security issues since the disclosure of a zero-day vulnerability in the Log4j shell tool widely used to manage logs in Java applications. But added that this issue is not going to be resolved overnight. “That was a wakeup call,” she said.

Tapabrata Pal, vice president of architecture for Fidelity Investments, noted that the fundamental problem is that cybercriminals have become more adept at injecting malware into software supply chains, so no one knows whether their application environments might be compromised. “It’s a serious issue,” he said.

It’s not known whether the open source software that many developers routinely incorporate into their applications might contain vulnerabilities that could be exploited, but the level of technical debt that has accrued over the last several decades is considerable. The issue is finding a way to address it without becoming overwhelmed by the enormity of the task.

September 14, 2023
Survey Surfaces Spike in Generative AI Usage Across IT

A survey of 800 DevOps and application security operations (SecOps) leaders published today found nearly all (97%) are making use of generative artificial intelligence (AI).

Conducted by the market research firm Sago on behalf of Sonatype, the survey found that beyond writing code, a full 45% of the 400 security operations professionals have already embedded generative AI into their workflows compared to 31% of the 400 DevOps professionals.

Security operations teams also reported greater time savings than their DevOps counterparts, with 57% saying generative AI saves them at least six hours a week compared to only 31% of DevOps respondents that saw the same benefit.

The top benefits cited by security operations teams is increased productivity (21%) and faster issue identification/resolution (16%) compared to faster software development (16%) and more secure software (15%) cited by DevOps professionals.

It’s still early days as far as embedding generative AI in workflows is concerned, but it’s clear the ability to write code faster is trumping concerns about the increased number of vulnerabilities that might find their way into application environments. The most widely used generative AI platforms are trained on a massive corpus of data that includes code of varying quality. As such, a lot of the code generated by a general-purpose AI platform will likely have as many errors, including vulnerabilities, as the code used to train it.

The level of faith organizations will place in that code will be heavily influenced by use cases. A script used to automate DevOps workflows that are not externally facing may be one example of where a general-purpose AI platform can boost productivity. In other instances, code that contains vulnerabilities that find their way into a web application might be much more problematic.
Despite those concerns, however, the survey finds nearly three-quarters (75%) of all respondents feel pressure to use generative AI.

There are, of course, already large language models trained on code that has been vetted by software engineers. In the months ahead, the code generated by AI platforms specifically designed for that purpose should significantly improve.

Sonatype CTO Brian Fox said it’s only a matter of time before IT organizations focus more on ensuring the generative AI platforms are tuned for specific tasks. In the meantime, however, application security may get worse before it gets better, with more than three-quarters of DevOps respondents noting that generative AI will result in more vulnerabilities in open source code. Only 58% of security operations professionals are similarly concerned.

It will be up to the engineers that make up these teams to bring discipline to code quality practices and make sure the code running in a production environment is of high quality, regardless of how it was created, noted Fox. For the moment, the issue is that AI augmentation is now occurring at a faster pace than the AI advances that should one-day help DevOps teams cope with increasingly larger codebases, he noted.

Survey respondents are also concerned about who might own the code generated by AI. A total of 40% said developers or the organizations they work for should own the copyright for AI-generated output, and most of them agreed developers should be compensated for the code they wrote if it’s used in open source artifacts in LLMs (DevOps 93% versus SecOps 88%). A total of 42% of DevOps respondents and 40% of SecOps respondents said a lack of regulation could deter developers from contributing to open source projects.

DevOps and SecOps leads both want more regulation: Asked who they believe is responsible for regulating the use of generative AI, 59% of DevOps leads and 78% of SecOps leads said both the government and individual companies should be responsible for regulation.

At this point, the proverbial generative AI genie is out of the bottle, so there is no going back. The issue that remains to be seen is how DevOps and security operating teams will adjust to that new reality.

September 12, 2023
Styra Makes Source Code Available for Enterprise Edition of OPA

Styra is now making the enterprise edition of the Open Policy Agent (OPA) available under a source license that enables IT teams to modify or change the underlying code as they see fit when integrating it with other applications.

Enterprise OPA is a drop-in replacement for OPA, agent software for implementing policy-as-code that is already available under an open source license. Styra’s enterprise edition provides access to a version that provides better performance, native integrations for a suite of data sources and additional capabilities such as policy decision analysis.

Previously, Enterprise OPA was only available in either a binary or container format. While these formats enable Enterprise OPA to be widely used, they limited it from being built directly into applications for performance optimization because no access was provided to source code.

Chris Hendrix, director of product management for Styra, said the source license also provides organizations with an assurance that they will be able to continue to modify Enterprise OPA in much the same way as they modify the open source edition.

It’s not clear how many DevOps teams are managing policy-as-code within their pipelines, but as application security regulations become more stringent, it’s only a matter of time before most of them do so in one way or another. Styra’s approach relies on a Rego programming language that some DevOps teams have found challenging, however. To make it simpler, the company is working on a low-code approach that would help make Enterprise OPA more accessible, said Hendrix.

In addition to encouraging more DevOps teams to use Enterprise OPA under a source license, Syra is clearly moving to make it more attractive to replace existing instances of OPA with a more robust implementation licensed from the company. No one knows how many instances of the open source edition of OPA advanced under the auspices of the Cloud Native Computing Foundation (CNCF) are running in production environments. But as those applications become more complex, the need for deeper levels of integration enabled by a faster version of OPA becomes more apparent.

OPA enables developers to programmatically embed compliance controls without waiting for a compliance team to make them available. That ultimately makes it simpler to ensure that the proper controls have been implemented before an application is deployed in an on-premises IT environment. That capability also makes it easier to document which controls have been applied to make any audit less time-consuming.

Shifting responsibility for compliance controls to the left also enables DevOps teams to better understand which controls address specific regulatory mandates incorporated into most applications versus which controls might be specific to a specific regulation. Armed with those insights, it then becomes simpler to incorporate policy-as-code within a DevSecOps workflow.

Hopefully, more applications running in production environments will soon be secure by default. Naturally, one of the first places to start is by making sure the compliance controls needed to achieve a baseline level of security have been pervasively implemented.

August 28, 2023