devopsy.kinsta.cloud

Category: IT as Code

Securing APIs at the Speed of DevOps

In the 2021 State of DevOps Report, 83% of IT decision-makers told Puppet that their organizations were in the process of implementing DevOps practices to improve the quality of their software, the speed of their delivery and the security of their systems. Those DevOps organizations varied in their stages of evolution, however. For example, respondents in the middle stages of their journey reported that culture blockers such as unclear responsibilities and insufficient feedback loops were making it difficult for them to continue their DevOps journeys and join the ranks of high-evolution organizations. Such obstacles help to explain why 97% of high-evolution organizations had already applied automation to their competitive tasks compared to just two-thirds of mid-level and a quarter of low-evolution teams.

The Importance of APIs to DevOps

The prevalence of automation ties into the importance of application programming interfaces (APIs) for any organization’s DevOps journey. In the words of IBM, an API “enables companies to open up their applications’ data and functionality to external third-party developers, business partners and internal departments with their companies.” APIs, therefore, enable the services and products of different companies to communicate with one another in a way that streamlines and/or augments functionality for the user.

That said, users aren’t the only ones who benefit from APIs. DevOps teams do, too. Many of these team members deal with increasingly automated processes as part of their daily tasks, so they need APIs that can help them to deploy and configure their infrastructure. APIs can also help DevOps teams gain detailed information about an app during runtime. DevOps staff can use these insights to proactively address issues before they lead to disruption.

Security Obstacles of APIs and DevOps

Despite their benefits, APIs and DevOps aren’t without their security challenges. Contemporary Computer Services, Inc. (CCSI) identified several security hurdles that organizations may encounter along their DevOps journeys. These include the following:

A fast development process: DevOps introduces a faster speed of software delivery. In their efforts to keep up with this demand, developers might make coding mistakes that leave applications vulnerable to attack. Absent a timely security review, those vulnerable products and services might make it into production, granting attackers an opportunity to exploit the flaws and to try to access affected customers’ digital assets.

Poor collaboration: No DevOps journey will succeed without collaboration between development and operations. When there’s a lack of joint processes, these teams may stick to their silos. This way of working may leave security gaps if development and operations aren’t communicating about the management of their credentials, tokens, SSH keys and other secrets, for example.

Those security challenges can then spill over into APIs. The speed of development along with poor team communications may give rise to multiple versions of the same API. This redundancy contributes to a sense of what’s known as API sprawl, a phenomenon where the number and reach of organizations’ APIs grow exponentially. API sprawl comes with its own obstacles including maintenance difficulties and broken clients, issues that provide attackers with more opportunities for infiltrating the organization’s systems. This reality helps to explain why 91% of enterprises experienced an API security incident in 2020 and why malicious API traffic increased by 300% in the middle of 2021.

How Can Organizations Address These Security Challenges?

For an answer to this question, we look back to what highly evolved DevOps organizations are doing. Most organizations are focusing on DevSecOps. In Puppet’s report, 51% of respondents said that they are integrating security into their DevOps requirements, with even more saying the same about design (61%) and build (53%). An additional 52% of survey participants indicated that they’re bringing security into their testing. By contrast, mid- and low-evolution organizations mainly engaged in security when they were in the process of preparing for an audit of their production systems or when they learned of an issue affecting production.

Beyond those areas of focus, organizations need to emphasize runtime protection. Salt Security agrees with this statement, noting how organizations need to “security test your APIs, but know that you will also need runtime protection to catch changes that don’t go through standard build process and abuses that testing tools aren’t designed to find.”

Finally, organizations need to treat DevSecOps as the cultural change that it is. They can do this by assigning security responsibility to one person from their DevOps team. Doing so will help to ensure that teams won’t overlook security during their jobs. Simultaneously, organizations can conduct security awareness training for their developers and regularly invite both development and operations to share their stories, priorities and successes with one another. This will help to foster collaboration, thereby minimizing instances of API redundancies and API sprawl across their systems.

March 23, 2022
Apple Outage Outrage | Linux Random Redo | Okta Hacked (or Not)

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: Why Apple services were down, Linux gets a huge RNG overhaul, and we wonder if Okta was hacked again. (more…)

March 22, 2022
Secure Software Summit: Behold the SBOM

With supply chain security becoming more of a focus, the SBOM is now viewed as a critical element in shoring up supply chain security. SBOM stands for software bill of materials. At a very elementary level, an SBOM is a list of ingredients. Think about how every food product in the supermarket lists ingredients so, for example, people who are allergic to nuts can avoid nuts. An SBOM does the same thing for software applications.

If you want to do business with the U.S. government, SBOMs are no longer optional. Executive order 14028 requires an SBOM for all U.S. federal software procurements. Why the push for SBOMs now? Software companies generally do not list all the components included in the software they produce. Knowing what components are included in applications is essential for proactive vulnerability management. That said, the government mandate for SBOMs only requires a minimal amount of included information—supplier name, the component name and version, dependency relationships, time stamps for versions or modification and information about who authored the SBOM itself.

What to Look For in an SBOM

This minimum required data yields minimal benefit for cybersecurity risk reduction and vulnerability management. SBOMs can actually include many other types of information. What information should you include in SBOMs for software applications produced, purchased or consumed by your organization? Here is a list.

Authenticity —Authenticity asks and verifies that someone actually created a listed component. Ideally, this requires a digital signature of some sort. Authenticity can be particularly important in the case of nested components where the supplier of an application includes multiple other components, as is common with JavaScript and Java libraries used for common functionalities.

Integrity —This addresses whether a component has been tampered with or modified without authorization. An SBOM should include some measure of integrity. Equally important is proving the integrity of the SBOM itself. Just as CheckSum can tell you whether software you are loading matches what the source of that software indicates, integrity checks should allow you to verify that components on an SBOM (and the SBOM itself) are exactly what you are expecting.

Services —An inventory of services called by an application is critical. Modern software does not live in a vacuum. It calls out to the Internet to perform all sorts of tasks and communication with other systems. Those services that extend beyond your application are part of the inventory because they can be an indication of vulnerability or indicate a larger problem.

Vulnerability Remediation Status —Development teams are constantly balancing the need to remediate vulnerabilities against the risk that remediation will break the application. Some open source projects, for example, have a history of not backporting security fixes. The latest version of the project may break applications. Organizations are forced to choose between breaking an application or taking the risk of introducing a high-severity vulnerability that might be exploited. Responsible organizations that opt to keep the older version and not break their software will backport their own fixes. An SBOM should describe these variations and any other vulnerability remediation activity.

Inventory of component security advisories —An SBOM should describe security advisories pending against all components. This is likely to be a dynamic list that you will need to automatically update using one of the accepted vulnerability advisory formats (CSAF and CycloneDX, to name two). In fact, this is one of the strengths of SBOMs—they are designed for automation and to act programmatically.

Completeness —It is not always possible to ship complete SBOMs, either for technical or intellectual property reasons. An SBOM should indicate whether it is complete and whether it contains a comprehensive inventory of dependencies and services.

Pedigree —This information describes the DNA of the SBOM and the original makeup and changelog of each particular component. An SBOM should describe the original component, exactly what changes were made, who committed the changes and why, what the differences were between different versions and what security issues were fixed with each variation and version.

Provenance —This relates to the origin of a component or service. It can describe a repository that the component was downloaded from, a country of origin, the name of a software manufacturer or open source project or other information indicating the component or service’s source of origin.

License type —License information is relevant for compliance and to ensure that your organization is not consuming or producing software in a manner that violates licensing terms. Moreso than for security and vulnerability, this information is for risk management, compliance enforcement and auditing.

Customizing SBOMs to Meet Your Needs

The SBOM is a flexible and highly adaptable approach that can be easily customized to fit specific use cases. You can design SBOMs to meet the specific needs of your use case, either as a software provider or consumer. You also may need to modify your SBOM fields and data design as your needs evolve. Fortunately, it is rapidly becoming easier to automatically generate SBOMs using software security tools such as software composition analysis (SCA) and interactive application security testing (IAST) solutions. SBOMs can also be created or read automatically at other points during the build life cycle along the continuous integration/continuous deployment (CI/CD) process. Just as you would not buy a box of cereal that did not list the ingredients, in the future you will not want to consume software that does not have a detailed SBOM. What you don’t know about the software you run can hurt you, and SBOMs will be a crucial tool in providing greater transparency and observability into the software supply chain.

About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay

March 22, 2022
Secure Software Summit: Reachability and Risk for Security Leaders
It is impossible to manage security posture without considering two key factors in any potential vulnerability or security flaw: Reachability and risk. The two factors are related. Reachability defines the degree to which a given security vulnerability that is detected, such as a CVE, can actually be attacked and exploited to gain privileged access and directly or indirectly access critical systems or data. Risk is a business measurement that assesses the potential for a vulnerability to actually damage an enterprise or organization. In general, without reachability, there is less risk.

How to Think About Reachability

Reachability and risk have dramatically shifted in recent years as we have moved from a hard perimeter “fortress” approach to security to more distributed systems, APIs, cloud computing, SaaS and far more potential points for attack ingress and exfiltration egress. To mitigate this new risk landscape, we have actually moved to fortify all the newly exposed internal elements and the critical systems of security. We have built a new perimeter around identity, with authentication and zero-trust. We have built perimeters around connected devices, around our cloud infrastructure and our virtualized environments and our configuration of all these systems. At every point where there is a potential for risk, you need to think about fortifying the new mini-perimeters.

A key aspect of risk and reachability that we have spent less effort on is the human aspect. We do not yet consider people as a perimeter to be defended, even though people comprise the most reachable vulnerability. As people, we create flawed software. We misconfigure our infrastructure, our security tools and our authentication systems. A key part of the new mindset of reachability must be better accounting for and mitigating the risks of people. This means simplifying their decision trees, either through better systems or better design, and by helping them become a better gray-matter perimeter.

Due to the atomization of our IT and application environment, and the resulting miniaturization of our security fortresses, we have created a far more granular surface. In some cases, such as through segmentation, we have enhanced security and reduced risk; in other cases, we have added complexity. Without a doubt, we now have many more points of contact and a broader attack surface exposed to the world. A critical consideration when assessing reachability is how far each of those points can be exploited, and to what breadth and depth—looking at horizontal traversal and immediate compromise of critical systems or subverting supply chain and other pivotal subsystems for subsequent attacks. The miniature fortress mentality is a useful framework for thinking through the potential attack surface and putting in place the right controls, at the right places, to address reachability.

Risk Scoring and Reachability

So what is the interplay between risk scoring and reachability? We cannot say no to new technologies that introduce more complexity, and potentially more risk, if our end users are demanding them. We need to better manage how risk scores are created and applied to help people perform better in this environment of greater complexity.

Today, the responsibility for understanding the complete security picture and applying the proper context to decision-making and judgments has grown more widespread. Everyone who is deploying, using, building or designing applications and technology environments must be constantly thinking about the security implications of whatever project or code they are working on. The lens for those implications is not only at the application or system or infrastructure level but for the general well-being of the business. This means a broader understanding of risk scoring driven by reachability and criticality and taking into better account the implications of reachability and attackability.

In cybersecurity, there is no shortage of risk measurements. Injecting reachability into the scoring process is a way of framing vulnerabilities to prioritize which of these issues are likely to be an actual risk. It is possible to have something that comprises a major business risk but have a very low reachability context, which may lead a security team to treat that vulnerability differently.

Broadly speaking, however, it is insufficient to make decisions solely based on programmatically generated risk scores. Any risk score is a reflection of the system generating the score, and how that system was tuned, either by the vendor or by your team. It is important to always remember that any risk score involves some subjective judgments. This is necessary, but injects opportunities for drift and bias that may skew scores in ways that leave you less secure.

To be clear, risk scores serve a crucial purpose in application security. Risk scores help you reduce the noise and find signals amidst the flood of CVEs and vulnerabilities and alerts. In fact, there is today so much noise around security that any effective team relies on multiple layers of noise reduction in their decision chain and prioritization and policy efforts. Where risk scores fail is with the inability to map accurately to business risk or by inaccurately reflecting the reachability of any CVE or other reported issue. This requires security teams to build their own internal and mental models of risk, modeling threats against the most critical systems and creating context that can be used to verify decisions by the risk scoring systems. Knowing how data flows through applications and infrastructure is crucial if a human agent can quickly judge the reachability and attackability of any given vulnerability presented or recognize when the automated risk scoring engines may not accurately reflect business risks.

In effect, there are two different processes at work: Calculating risk and contemplating risk. Calculating risk is programmatic and automated; human-guided, initially, but driven by models applied against the mass of vulnerabilities surfaced by security testing tools like software composition analysis, static application security testing, fuzzing and linting. Contemplating risk is a more holistic exercise that leverages the human capacity for intuition and innate knowledge of systems driven by experience and hard-won insights.

Contemplation is what will help security teams identify undetected biases, guide threat modeling to better consider drift and highlight areas where human breakdowns might have augmented risk in ways that a risk scoring system could never capture. Above all, contemplation is about context. The more context a human actor has, the better they can understand the big picture. The greater the understanding of the big picture, the better decisions people make. This must also flow through and inform the models and the risk scoring engines so that the value of contemplation is captured in the calculation.

Using Risk Scoring Tools to Minimize Reachability and Attackability

A security team must constantly measure the efficiency and efficacy of all security tooling. The general rule is that security controls lose efficacy over time as vendors lose focus or fail to account properly for emerging classes of threats. When you have a security tool that generates risk scores in tune with your reachability guidelines and proves accurate, you should elevate that system to a higher level of trust and give more weight to its findings. That said, even tools that generate risks you think are accurate and perform well must be constantly evaluated and cross-checked.

Unfortunately, it only takes a small amount of drift to open up an organization to potentially catastrophic attacks. A critical part of this calculation is ensuring that humans stay on top of how the risk scores are created and continue to influence the scores to fight drift. Drift is natural but it is not inevitable. Your code and environment are constantly changing, This may result in changes in reachability which can materially impact your overall risk posture and should alter the way you approach different systems and risks. An automated risk scoring system may not understand that a new type of supply chain attack might result in subsequent related attacks on similar components in other operating systems, to name one example. Human intelligence and contemplation, really, is the secret ingredient that makes risk scoring tools far more valuable and reliable.

Takeaways: Atomized Systems, Reachability and Risk and Humans

The environment and application landscape will continue to atomize further as we move into new infrastructure paradigms like edge computing and all systems on the network gain capacity. For this landscape, security teams and the developers they work with on shifting left should:
- Recognize that we have shifted from a global perimeter to an atomized constellation of mini-perimeters and shift their mindset to consider security and building fortresses at the micro level as a key way to reduce reachability and risk
- Developers and security teams need to assess the systems they build and protect on the basis of reachability and risk to properly prioritize. Risk scoring tools are useful elements for reducing noise but cannot supplant human intelligence to gut-check risk decisions
- All along the way, humans have to continue to guide and inform these risk scoring systems and ensure that risk scores do not become blind proxies for criticality and importance. Allowing automated risk scoring to dominate the decision process can make security drift more dangerous and allow human error to go unrecognized, leaving applications and infrastructure both reachable and at risk.
This article was co-authored by Bryan Smith, CTO, RiskLens and Rob Lundy, director of product marketing, ShiftLeft.

About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay
March 21, 2022
Secure Software Summit: Applying Chaos Engineering to Software Security
Today’s software systems are, essentially, controlled chaos—and lightly controlled chaos, at that. This makes it exceptionally challenging to model the behavior of those systems. Our systems are quickly becoming larger and larger, with more and more moving parts. It is not uncommon for enterprises to have over 1,000 microservices and millions of containers running thousands of applications containing many thousands of open source libraries.

Why are things growing more complex? Because from microservices to CI/CD pipelines, from service meshes to blue/green deployments, all these new practices and new technologies help us make our applications and infrastructure more resilient and scalable. The downside is the ever-increasing complexity, which is actually accelerating.

In addition, software has officially taken over every layer of the networking and hardware stack (which sounds like an oxymoron, but it’s containers and virtualization turtles all the way down until you hit bare metal). Software is, by definition, easy to rewrite. Thus, the ease of changing software has further increased the number of changes we make to systems. Which adds still more complexity. In fact, software only ever increases in complexity; never decreases in complexity. Software is always trying to do more and encompass more. For that reason, even when you try to simplify software, you just move the complexity around rather than eliminating it.

Introducing Turbulence

Here is how Netflix, the organization that pioneered chaos engineering, defines the practice: “Chaos engineering is the discipline of experimenting on a distributed system to build confidence in the system’s ability to withstand turbulent conditions.” Turbulence is the opposite of order, but it’s where all systems end up.

In the beginning of a system’s life cycle, there is always an orderly plan for deployment and operations. You always have a clean diagram of your resources, code repositories, containers and always a nice 3D diagram of AWS instances you will deploy to and the network topology.

In reality, our systems are never this simple. After the system goes live, that is when we start learning about how it actually operates through unforeseen events like when marketing asks to refactor the pricing module or an outage to the payments API forces you to hardcode a token or a DNS outage—well, you know about DNS outages. In the span of a year or two, you go from 300 to 800 microservices and you have to add Kubernetes clusters and a container network interface (CNI) to properly secure the networking of those clusters. And so on and so on. The bottom line? Almost all production systems evolve into a state that we do not understand and cannot easily map back to our original systems diagram.

Chaos engineering is a mechanism for dealing with this growing level of complexity. The basic idea behind chaos engineering is asking the computer or application or network (or any other environmental element) a series of questions. Consider a legacy system like a mainframe. How did our legacy systems become so stable? They are well-documented. Engineers feel confident in how the system operates. The question is, was the system always that way? Is the system the same one that they originally designed? The answer is “No.”

Legacy systems actually went through a trial-by-fire; a constant parade of unforeseen surprises. These surprises were, effectively, questions. Through these surprises, engineers learned how the system would actually operate (or wouldn’t operate) in the face of chaotic and unforeseen events. This process created iterative feedback that helped the engineers learn how to make the system continuously more stable.

The point of chaos engineering is to generate this same type of feedback loop without the pain, stress and repercussions of unforeseen (and generally unwanted) events. Thus, chaos engineering is a proactive way of introducing adverse conditions to ensure that the system can still do what it’s supposed to do when unforeseen events transpire. Because everything today is software and change is constant, the questioning must also be constant.

For example, chances are that in the time since we originally designed any given system, it has changed a lot. This means some or all the mitigation and resilience we programmed in—retry logic, failovers, circuit breakers—are old news, designed for an older system. When triggers for those conditions actually occur, those mechanisms of resilience we designed may not perform as we had intended.

Chaos engineers proactively ask the question, “Under the conditions in which you are designed to perform, do you do what you are supposed to?” We introduce the conditions themselves in a controlled manner with a limited blast radius to put the system to the test and verify that our assumptions line up with the intended reality. The practice of chaos engineering is intended to be performed on live, running systems without the risk of taking down critical production assets or ambushing unsuspecting applications and their associated DevOps and development teams. This helps engineering teams reduce uncertainty by building confidence in their understanding of their systems by providing them with enhanced context. It turns out that, when engineers are presented with better context, it considerably improves their ability to make smarter, well-informed decisions where resilience is concerned.

Chaos Engineering for Cybersecurity

Cybersecurity is a context-dependent discipline, dependent on how well you understand the systems you are protecting. Software engineers are constantly changing the system. And they must have the flexibility to change it because their primary task is to constantly move towards business value. Yet to effectively apply cybersecurity practices, we must know what we are trying to secure. From that knowledge, we then can know what we most need to secure and how to do it on a given system in its most current state.

Before an engineer can understand how to build robust security measures that provide effective cybersecurity, the engineer must first understand the system. This reliance on the “system first” is a foundational driver for why our security systems and controls—even the immutable and ephemeral ones—experience considerable engineering drift over time. We are continuously burdened by the dependence on a stateful understanding of the system.

Without having some sort of testing or instrumentation feedback loop in our post-deployment world, we don’t know there is a problem with our security until there is an actual problem. Finding out that the room is on fire after the house has been burning for hours is too late. The result is the self-defeating series of reactive incident fire drills, war rooms and outages that we find ourselves in today. If we do not change this reactive crisis loop of chasing the tiger’s tail, it inevitably leaves us taking two steps back for every step we take forward with engineering teams.

When this drift goes undetected, we are left with our wounds exposed and attackers salivating at the meal to come. Engineers always prefer to find evidence and sources of drift through some internal engineering failure or controlled test rather than experiencing a successful real-world attack. Over time, there is a constant drift between the system and security; engineers are constantly changing the system. As a result, there is frequently a lack of recalibration between systems and the security designed to protect them. The way to achieve this recalibration is through continuous testing or experimentation.

Testing is not chaos engineering. Testing is a verification and validation of something you know as either true or false. This could be vulnerability to a specific CVE or exposure to an attack pattern. In experimentation, you are trying to derive new information and explore the unknown unknowns. From this exploration, you seek to create a better understanding of how a system reacts and then share that with the people who are designing the system.

Live security incidents are not effective measures of detection because, by then, it is already too late in the life cycle to effectively address the problem. Worse, incidents cause stress, fear and irrational behaviors. People freak out. Not surprisingly, a security incident is not a good learning environment. Yet security incidents serve the same purpose: Asking questions about a system or systems and helping teams see how to improve them. Chaos engineering offers a better way to create these questions under conditions where it is easier to learn and observe system behaviors, providing clues to improving security. In other words, the purpose of security chaos engineering is to help us proactively understand our systems and where the security gaps are before an adversary has a chance to take advantage of them.

There are multiple use cases for security chaos engineering including:
- Incident response: Testing efficacy of run books, response plans, and determining whether data collection and alerting systems work as intended
- Security control validation: Introducing conditions that should induce behavior from one or more controls and observing behavior
- Security observability: Introducing unexpected attacks or breaches to see if your systems report and observe the incident accurately
- Compliance monitoring: Applicable across all chaos engineering events by documenting that systems do what they say they are supposed to do
A Practical Example

A team I headed wrote an open source application, ChaoSlingr, designed to generate chaos engineering events for security experiments. The application consisted of three serverless functions to generate the experiment, apply it to infrastructure and then monitor the resulting impact on the system.

One example of how we used ChaoSlingr was an exercise where we induced an open port. Enterprises and engineers inadvertently open ports in firewalls all the time. Sometimes this is a result of misconfiguration. It may also be the result of a software upgrade after which the developer or AppSec engineer forgot to reset the port to the proper security settings. In theory, firewalls should automatically detect an unauthorized open port.

For the experiment, we selected a set of resources that had opted-in to our experiments, selected a security group and opened the port. At the time we were relatively new to Amazon Web Services. We were surprised to find that the firewall only detected an unauthorized port opening 60% of the time. We discovered clear evidence of configuration and security tool drift.

We also noticed that a cloud-native configuration and management tool we had deployed was identifying and alerting the unauthorized port opening almost 100% of the time. We then realized that correlating and responding to incident events of an unauthorized port opening was complicated; it was hard to map the incident back to the log data describing how our systems responded to the event. We added a metadata pointer to each alert, allowing us to more easily correlate the information and save valuable time finding the affected security group and port.

In short, the exercise gave us excellent context on where our systems were drifting and failing to keep up, which of our systems were the most reliable and whether we could quickly and easily respond to these incidents by chasing down the right log files.

Security Chaos Engineering Programs Build Trust

In any environment where we are managing complex systems, trust is critical. You must trust that your systems will handle unforeseen events gracefully and not cost your company money or reputational currency. You must trust that the people designing and building your systems are constantly trying to improve them. You must trust that the security teams protecting your systems are up to the task. As systems grow more and more complex, the number of unknown unknowns multiplies. As humans, with our limited processing capacity, we cannot solve for unknown unknowns at scale, continuously. This is the task of security chaos engineering and the processes around it: To create a programmatic way to experiment against and find answers to unknown unknowns. Your systems, developers, customers and your CEO will thank you for it.

About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay
March 18, 2022
A Closer Look at Azure DevOps
Microsoft’s Azure DevOps Server is emerging as one of the most popular cloud application development environments. Like most continuous integration/continuous delivery (CI/CD) platforms, it provides features such as version control, reporting, project management, automated builds, and testing. It covers all aspects of the application life cycle and enables DevOps. Here’s a closer look at this cloud computing environment.

Azure DevOps provides a number of different developer services that allow DevOps teams to plan work, collaborate on code development and build and deploy applications. It is highly collaborative and brings together developers, project managers and individual contributors to develop software. It allows organizations to create and improve products much faster than with traditional software development approaches.

Services or Server?

Teams can work in the cloud using Azure DevOps Services or on-premises using Azure DevOps Server.

The cloud offering, Azure DevOps Services, is a scalable, reliable and globally available hosted service. It’s backed by a 99.9% SLA, is monitored by Microsoft’s 24/7 operations team and is available in local data centers around the world.

The on-premises offering, Azure DevOps Server, is built on a SQL Server back end. The on-premises version is preferable when organizations need their data to stay within their network or when they want access to SQL Server reporting services that integrate with Azure DevOps Server data and tools.

Although both offerings provide the same essential services, Azure DevOps Services offers some added benefits:
- Simplified server management.
- Immediate access to the new features and updates.
- Improved connectivity with remote sites.
- A transition from capital expenditures (servers, etc.) to operational expenditures (subscriptions).
There are also some other key differences depending on your organization’s or your team’s needs. These can also be helpful if you’re migrating from on-premises to the cloud:
Differences in Specific Feature Areas

Although Azure DevOps Services is a hosted version of Azure DevOps Server, there are some feature differences. Some Server features aren’t supported in Services; for example, Azure DevOps Services doesn’t integrate with SQL Server Analysis Services to support reporting. Process customization and reporting also differ in their support.

Integrated Features

Azure DevOps provides integrated features that can be accessed through a web browser or an IDE client. One or more of the following standalone services can be used based on your organization’s business needs:
- Azure Repos: Azure repos provides Git repositories or Team Foundation Version Control (TFVC) for source control.
- Azure Pipelines: This service provides build and release services to support continuous integration and continuous delivery of applications.
- Azure Boards: This suite of Agile tools supports planning and tracking work, code defects and issues using Kanban and Scrum methods.
- Azure Test Plans: Azure Test Plans provides several tools to test applications, including manual/exploratory testing and continuous testing.
- Azure Artifacts: These allow teams to share packages such as Maven, npm, NuGet and more from both public and private sources and integrate package sharing into pipelines.
Use of the following collaboration tools is also supported:
- Customizable team dashboards with configurable widgets to share information, progress and trends.
- Built-in wikis for sharing information.
- Configurable notifications.
Azure DevOps supports adding extensions and integrating with other popular services, such as Campfire, Slack, Trello, UserVoice and more. In addition, DevOps teams can also develop their own custom extensions.

Azure DevOps Services supports integration with GitHub and GitHub Enterprise Server code repositories. Azure DevOps Server supports integration with GitHub Enterprise Server repositories.

Getting Started

When you sign up for Azure DevOps, you must first register an organization, project and version control type. DevOps teams can create multiple projects and differentiate them with separate names and access permissions. Projects can be set to public or private. Teams can also use Azure Boards for collaboration and project management, and can also configure use of Agile practices. After setting up your team, you can use the platform to begin deploying your applications to Azure.

In addition to a collaborative environment, Azure DevOps provides access to an Azure Pipeline service that supports multiple languages and platforms. The Azure DevOps pipeline is a set of services that are grouped into individual projects.

Additionally, Azure Test Plans feature facilitates automated testing. It is also possible to set up deployment pipelines with different components for multiple applications.

Azure Pipeline can track work status, manage backlogs and create custom reports. With the help of the Azure DevOps pipeline, developers can create, manage and deploy their projects in a single environment. With this solution, DevOps teams will have a single point of contact and can communicate with each other.
March 18, 2022
CSS-Tricks Sells Out | Peloton Spins Out | ARM Axes 1,000

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: CSS-Tricks gets bought by DigitalOcean, Peloton pedals while its business burns, and Arm lays off up to 15% of staff. (more…)

March 17, 2022
Log Management Challenges Grow With Observability Adoption

It appears that, in many IT environments, observability is becoming too much of a good thing. A global survey of more than 315 IT executives, cloud application architects, DevOps professionals and site reliability engineers (SRE) conducted by Dimensional Research on behalf of log data management platform provider Era Software, found 96% are witnessing an explosion of log data.

The survey also found 79% of respondents reported the overall cost of observability and log management will skyrocket in 2022 if current tools don’t evolve.

More than three-quarters of respondents (78%) also noted attempts to manage volumes of log data have had mixed or unwanted results such as an inability to access data. Nearly two-thirds (65%) are also evaluating their observability options while the other 41% are considering it.

Stela Udovicic, senior vice president of marketing for Era Software, said the survey makes it clear that as organizations rely more on logs to analyze IT events, the cost of storing all that log data is becoming a significant challenge. Overall, the survey finds usage of observability tools and platforms has jumped by 180% as IT teams struggle to manage increasingly complex IT environments.

On the plus side, more business users are benefiting from log data. A full 83% of respondents report that business stakeholders outside of IT use insights from log data, with 96% reporting log data is being used to solve business problems.

Observability has always been a core tenet of DevOps best practices, but achieving it has always been a challenge. Monitoring tools are designed to consume predefined metrics to identify when a specific platform or application is performing within expectations. The metrics tracked generally focus on, for example, resource utilization. However, whenever there is an issue, it can still take days, sometimes weeks, to discover the root cause of an issue through what amounts to a process of elimination.

In contrast, observability combines metrics, logs and traces—a specialized form of logging—to instrument applications in a way that makes it simpler to troubleshot issues without relying solely on a limited set of metrics that have been predefined to monitor a specific process or function. DevOps teams can employ queries to interrogate data in a way that makes it easier to discover the root cause of an issue. Observability platforms correlate events in a way that makes it easier for analytics tools to identify anomalous behavior indicative of an IT issue.

The issue that arises is that IT teams need to find a way to store all the data generated by the logs they need to analyze. Each IT team can, of course, could reduce the amount of log data they retain. However, there is always a concern that in the event of an incident critical log data won’t be available.

Udovicic said most IT organizations have historically viewed storing log data as a necessary evil. The primary issue most of them encounter is the simple fact that log data is not easy to work with, she noted.

Regardless of how organizations view log data, the amount of it that needs to be managed will only increase as DevOps workflows become more mature. The issue IT teams need to come to terms with now is finding a way to minimize the cost of storing it all.

March 17, 2022
Secure Software Summit: Measuring and Mitigating OSS Risks

Measuring and mitigating the security risks in open source software is becoming a major issue in the software development community. Attacks on open source software (OSS) are on the rise; open source supply chain attacks grew 650% in 2021—a staggering number.

2021 and early 2022 saw significant attacks which impacted more than 1,000 organizations. The most prominent recent OSS attack was the Log4Shell vulnerability disclosed in December 2021. Attacks on the OSS supply chain and on OSS impacted nearly every single software organization on the planet. Everybody is scrambling to continuously patch systems as new vulnerabilities continue pouring in.

To properly navigate this environment and keep our systems safe, we as a community need to understand the security risks of OSS and be better prepared to handle them, collectively and individually.

Open Source Risks

There are a number of risks that make OSS more vulnerable without proper security measures.

OSS is Free; OSS Security is Expensive

To start, we must recognize that open source software may be free to download and require no payment, but it comes with a massive security cost.

A significant portion of open source software is developed by volunteers in their free time. Most of the prominent open source volunteers actually have full-time jobs, and the code they contribute to projects like the Log4j logging libraries are written on nights and weekends; on their own time and not on the company dime. Not surprisingly, security isn’t high on their priority list. They don’t have the resources to invest in a secure development life cycle or have the time and bandwidth to maintain their code. This may mean that security vulnerabilities remain unaddressed for long periods of time, sometimes indefinitely.

All Bugs are Shallow, but There Aren’t Enough Eyeballs

In software development, Linus’s law states “…given enough eyeballs, all bugs are shallow.” However, this is only true if those eyes are looking in the right place. In the case of open source, all code is open but there aren’t enough eyes paying attention to it to spot all the bugs. In addition, organizations may not consider the security consequences of their decision to consume a particular piece of open source software and adding it as a dependency. Dependency trees become more and more complex; harder and harder to monitor and secure, as you keep adding more third-party components to a core codebase. Each new dependency might bring additional nested dependencies—perhaps hundreds or thousands, depending on the size of the library added or the role it plays in the application architecture. For example, the Kubernetes project—one of the most successful and widely adopted OSS projects—has several thousand dependencies, many of which have additional secondary and tertiary downstream dependencies.

So Much Code. So Many Choices. Not all are Good.

The beauty of the open source ecosystem is that there is a solution for everything. The problem with the open source ecosystem is that there’s not just one solution, but several dozen. For example, if you need a JSON parser for your application, there are several to choose from. If your application is running a critical service or infrastructure, then picking the right JSON parser—one that is secure and comes from a responsible project or maintainer—becomes very important. A JSON parser will serve as a critical component for your application, so it will impact the security of your entire system. If you chose the wrong JSON parser, you might pay a big price down the road in security risk and subsequent breaches.

Proposal: An Entry Gate Security Risk Assessment

One potential solution to enhance OSS code security is the creation and implementation of a security risk assessment gate as part of your OSS ingestion pipeline. Putting this process in place will help you understand whether consuming a certain piece of software is high risk and avoid software that does not meet your risk criteria.

Here is an example of how this might work. Consider five different JSON parser libraries in GitHub, all of which have a high number of stars and appear to be highly rated. How can we assess the risk of these libraries and quickly and easily eliminate ones that are too risky? One of these JSON libraries is hosted under a user account. Most of the commits are done by a single developer without reviews. This alone indicates that using this library in a critical system presents a high risk. This lone developer could commit malicious code without review. The maintainer’s token access could be leaked, allowing attackers to insert malicious code. The developer might also turn over committer control of their library to a malicious actor without notifying users (who might not pay attention to the notification anyway). Identifying this risk required no more than a few minutes of review of the repository and cost little in time or effort.

A Broader View of Security Posture for OSS

More broadly, we need to understand the full security posture of an organization or developer and their open source code prior to adding it to critical systems. Because OSS is a complex ecosystem and many pieces of OSS are complex, the risks they generate are similarly complex. Our traditional, myopic view of these risks is that we just need to find all the security vulnerabilities in our codebase and fix them. This does not resolve all risks and may lull us into a false sense of security.

We need to understand the full comprehensive security posture of an open source project. That involves understanding a broad range of risks beyond code vulnerabilities. Here are some of the key risks to consider beyond ensuring that your codebase is well-secured and patched.

Supply Chain Attacks on the Build Pipeline

Attacks like SolarWinds and CodeCove taught us that supply chain risk is a very big deal and that tools are just as much at risk as code. Attackers have found it much easier and much more reliable to attack build pipelines, rather than finding vulnerabilities in code.

Security Risks Revealed by Continuous Security Testing

Continuous security testing is critical to maintaining the security of a codebase. There might not be code vulnerabilities today, but as developers continuously modify code and iterate on applications, vulnerabilities may be introduced. This can mean testing code as part of the CI/CD process with every new code push. For example, we have found numerous critical bugs in OpenSSL libraries. All were caught on a master or main branch and never made it to a production release.

Project Maintenance and Capabilities

One of the really big risks with open source is the quality and frequency of project maintenance. Issues raised may not be addressed in a timely fashion. Maintainers of a project may have a long backlog of issues, indicating that they are not actively maintaining a project. It is not uncommon for critical open source projects used in millions of applications and comprising a significant dependency to the global software supply chain to not have been updated in a year or more. In addition, maintainers of OSS projects may not know how to respond to security incidents and fix security vulnerabilities. This is particularly true if the maintainers do not work at a large organization, do not have cybersecurity or secure development experience or have never suffered through the toil and trauma of addressing a major vulnerability in their code on short notice.

Tools for Measuring and Addressing Supply Chain Risk

A number of efforts are underway to create free tools to help developers and application security teams better measure and address supply chain risk programmatically and analytically.

SLSA—Supply Chain Level for Software Artifacts

As a tool to help organizations better address software supply chain risks, Google created a novel framework that we call Supply Chain Level for Software Artifacts (SLSA). The project has been underway at Google for more than a decade and has been open sourced as part of the OSSF. Contributors include VMware, Citibank and Datadog, among others. Google hopes to standardize SLSA and make it an industry benchmark for evaluating supply chain risk. Right now, SLSA is focused primarily on build process risks but the hope is to expand it in the near future to other areas, such as vulnerability and maintainer risks.

OSS Security Scorecards Project

In order to scale the process of assessing, identifying and mitigating risks in OSS, we need robust automation. In 2020, as part of the OSSF, Google released a new project called Open Source Security Scorecard. The project aimed to automate the analysis and trust aspect when evaluating the security posture of open source code. Further, the Scorecards can provide specific security recommendations. The Scorecard process can incorporate many other automated tools, including checking against vulnerabilities using the Open Source Vulnerability (OSV) database and running dependency checks using open source dependency trackers like Dependabot. The scorecards project also makes it easier to report new vulnerabilities and get those to the right people in a project.

Conclusion: What is not Measured is not Mitigated

No one denies that OSS has become crucial to the vast majority of applications running the world’s technology foundation. The span of OSS continues to grow, as more and more languages and frameworks emerge and specialization increases. No one wants to reinvent the wheel and rewrite fresh code to perform the same tasks with every new application. The model of reuse, dependency and distributed software development works and speeds innovation. That said, the open source community and the supply chain used to build OSS need a more rigorous approach to reducing risk. Given the scope of the problem, automation and standardization are helping to scale toward the stated goal. We are in the early stages of creating the tools and infrastructure to automate the risk identification, measurement, assessment and mitigation processes. Security frameworks like SLSA and new processes like entry gating new OSS code with security scorecards will direct organizations down that path and move us toward a more secure open source future.

About Secure Software Summit
The Secure Software Summit brought together the world’s leading innovators, practitioners and academics of secure software development to share and teach the latest methods and breakthroughs on secure coding and deployment practices. If you are about developing, releasing and securing software, delivering new features fast and building things right from the start, click on the link below to get access to all of the sessions from this summit.

https://go.shiftleft.io/secure-software-summit-2022-replay

March 17, 2022
Managing Endpoint Compliance-as-Code
The rapid pace of modern software delivery and increasing fleet sizes have transformed how organizations see and handle security strategies. Compliance-as-code is required for today’s organizations that need security as a fundamental part of business processes. It is no longer possible to manually manage compliance with dedicated security teams.

Balancing security with growing infrastructure needs means IT security and compliance are non-negotiable. There is no room for more uncomfortable trade-offs between risk and an organization’s ability to deliver market-ready solutions quickly and efficiently.

Complying with growing security and compliance regulations in today’s world of rapid innovation is a constant challenge that affects all organizations, large and small. Organizations are implementing automated solutions to eliminate reliance on traditional, slow, error-prone manual processes because of tighter regulations in the industry and greater risks associated with security attacks and compliance violations.

What is Managing Endpoint Compliance-as-Code?

Compliance-as-code is the codification of compliance controls to automate their adherence, application and remediation. It includes the tools and practices that enable DevOps and developer teams to incorporate the three key compliance activities:

Detect: Discovering non-compliance through automated estate scanning and notifying stakeholders when offending infrastructure is discovered.

Remediate: Correcting non-compliance by implementing immediate infrastructure changes to ensure the highest level of compliance at scale.

Automate: Avoid non-compliance by automatically verifying that planned changes comply.

There are many important use cases for managing the compliance of endpoint state as-code including confidential data protection, detecting shadow IT resources, network security, data exposed to public access and code licensing compliance.

The Importance of Compliance-As-Code

Most organizations still struggle to stay secure and comply with regulatory standards. Many organizations lack visibility across heterogeneous infrastructure and applications. They also possess inconsistent language for communicating requirements between development, security and operations (dev, sec and ops) teams and are unable to remediate findings.

Compliance-as-code enforces a comprehensive compliance strategy that involves a robust set of controls; for example, managing data storage locations and access control management.
Ensuring that these are followed at scale is critical to the success of a digital business as more organizations shift to the cloud, there are more possibilities of non-compliance. Reports suggest that a regulatory compliance violation can cost businesses $15 million on average.

This violation is avoidable if DevOps, development and DevSecOps teams can automate compliance by adopting a different mindset and writing understandable code. The adoption of the “as-code” strategy led to infrastructure-as-code (IaC) which is one of the success factors in DevOps teams. Audits are streamlined and allow individuals to focus on higher-value activities.

The Challenges of Managing a Fleet Through Compliance-as-Code

Managing a fleet through compliance-as-code is not without its challenges. First, teams must bridge the talent gap as endpoint compliance is typically managed by IT admins, who are not inherently comfortable with an as-code approach.

Typically, compliance requirements are dense and difficult to comprehend by other stakeholders. Understanding the compliance requirements of the organization and collaborating with other stakeholders to convert them to as-code is the biggest hurdle to getting started.

Developer teams must find the right set of tools and integrate them into DevOps and DevSecOps workflows. This is not just a process change but a culture change. A boil the ocean approach will not just be painful but success will be close to impossible. Taking small steps and continuously improving is the mindset one has to adopt.

Visibility Throughout the Ecosystem

Whether you are talking about end-user devices like desktops, laptops or mobile devices or non-server edge devices like point of sale (POS) or IoT devices, endpoint devices are extremely vulnerable to exploitation from bad actors. Thus, it is necessary for organizations to bring them under the organization’s security policy and governance requirements just like server-side devices.

There are a number of advantages to adopting an endpoint compliance-as-code approach:

Users can easily understand what is going on across the organization in terms of compliance status at any time.

The organization can keep an audit log of everything that occurred throughout the development life cycle and calculate an appropriate risk acknowledgment rating.

Organizations can evaluate any deviations from fundamental compliance requirements.

The Continued Prioritization of Compliance-as-Code
Managing diverse IT infrastructure fleets through compliance-as-code ensures that endpoint devices meet a variety of security standards. The IT team is empowered to create rules that can enforce compliance and establish security baseline standards within the organization. Compliance checks are validated and more visible at each stage of the software development life cycle.

This guarantees detailed visibility and thorough compliance standards throughout the IT ecosystem.
March 17, 2022