devopsy.kinsta.cloud

Tag: application security

DevOpsSec: Survival is Not Mandatory

Deming, the patron saint of DevOps once advised, “It is not necessary to change. Survival is not mandatory.”

To survive, application development teams are constantly pressured to deliver software even faster. But fast is not enough. The best organizations realize that security, quality and integrity at velocity are mandatory for survival. Hence, DevOpsSec

My aim here is to leave you uncomfortably amazed. While the pace of development has changed significantly, our processes to secure our applications has not changed enough. It is as if a wildfire has raged uncontained for years now but we have failed to take notice.

Maybe that sounds a bit crazy, but once I share a little background on what is happening, you’ll understand why innovation in this arena is critical for survival (when it comes to your applications). If you are aiming to ramp up your own Rugged DevOps or DevOpsSec practices, this is critical reading material.

Skyrocketing Usage, Plummeting Visibility

Open source component use in development is skyrocketing, and for good reason. Over 17 billion open source and third-party components — across the major development languages — were downloaded last year helping development teams accelerate release timelines and deliver more innovative solutions to market. To better imagine the impact of this download volume, you need to understand that there are an estimated 11 million developers responsible for the billions of downloads.

While the magnitude of usage is amazing it has obscured a vast majority of the risks. Of the billions of downloads recorded last year, 1 in 16 components downloaded had known security vulnerabilities.

While we have seen 30x growth of download requests over the past seven years, we have also witnessed huge growth in the number of organizations improving the speed of consumption and efficiency of their downloads using repository managers.

In the past 18 months since I joined Sonatype, we have seen active instances of repository managers like Nexus, Artifactory, and Archiva grow from 40,000 to over 70,000 installations, with users in the millions — also for good reason. Development teams want to ensure faster, more reliable builds. They also need a private and safe place to house and share their own proprietary components as well as assembled applications, images and other binary outputs. The repository manager has established itself as a parts warehouse for software development.

All is Not Moonlight and Roses

1 in 16 may not sound like a lot until you recognize that many organizations are downloading over 250,000 components every year…some of the largest organizations consume millions of them. If you have not calculated this in your head yet: 250,000 / 16 = 15,625.

Couple this fact with the remarks in the 2015 Verizon Data Breach and Investigations Report stating that applications were the most often exploited attack vector by hackers. As a developer community we are electively sourcing known vulnerable components for use in our applications and those applications are now more vulnerable to attack.

Your Repository Manager Should Serve and Protect

If bad components are getting in, why not just stop them at the front door?

Easier said than done. Current approaches to preventing such behavior are ineffective. For some, “golden repositories” are used to house approved components — but components approved once are rarely vetted again for newly discovered vulnerabilities. For other organizations, OSS Review Boards mandate reviews and approval for all new components — but these organizations are poorly staffed to match the volume and velocity of consumption leading to workarounds by development teams. And while developers themselves would much prefer to use the best quality, highest integrity components when designing an applications, they are never allocated sufficient time to investigate the vulnerability status of every component required for their latest build.

Sparking Innovation

When current approaches fail, inspiration often sparks innovation.

Enter, a new invention: a repository firewall. Think of it as a repository manager with a guard at the front door. Every component that gets downloaded by a proxy repository is automatically evaluated against the parameters development, governance and security teams establish. Does it have an AGPL license, include a known security vulnerability, or is it incredibly outdated? The repository firewall allows organizations to download “good” components while it blocks and quarantines “bad” component downloads. Using a repository firewall can keep your repository manager and development lifecycle safe and secure – in an instant, automatically.

The first repository firewall of its kind is called Nexus Firewall. It couples software supply chain intelligence about component quality, security, and risks with automatic evaluations against personalized policies for approving or rejecting new downloads. Evaluations can also be applied later in the development lifecycle where staging repositories are in use.

Imagine the result: 16 out of 16 downloads, or 250,000 out of 250,000 downloads are now of the best quality. Everything flowing into your application development lifecycle contains the highest quality components. You are instantly compliant with established policies. Your applications are less vulnerable to attacks. You are automatically reducing risk.

Next up in this series, I will share insights about Repository Health Check reports (free to use). Used by over 15,000 organizations, they can offer organizations the first clue as to whether or not your team could benefit from a repository firewall.

December 8, 2015
DevOpsSec: 1 in 16 Chances

The quantitative research summarized below, covering over 7,000 repositories across nearly 100 countries, highlights some of the challenges with quality at modern development velocities, especially important for DevOpsSec practices. By leveraging automation in your repository manager, you can improve application quality and reduce unplanned work while lowering exposure to risk. While this practice supports DevOpsSec initiatives, at its core, it is simply a focus on “building quality in”.

(Written as a collaboration between me and Mike Hansen, Head of Products at Sonatype; Mike wrote the better parts.)

Innovation and Risk

Repository managers like Nexus, Artifactory and Archiva have been serving software components for development teams and their tooling for years now. We can measure over 70,000 active repository manager installations across the globe today. Use of a repository manager improves build performance and reliability while also providing a secure, private location to host open source, third party, and proprietary components needed across continuous delivery and DevOps tool chains.

Industry averages show software development organizations electively downloading over 240,000 components a year from public internet repositories. While these components accelerate development and increase innovation, the vast majority of components go through little to no vetting process to determine if known security vulnerabilities, risky license types, or outdated/unsupported are being used.

Earlier this year, researchers found that 1 in 16 downloads includes a known security vulnerability. That accounts for over 15,000 elective downloads of components with known security vulnerabilities per organization, per year.

Why the Focus on Repository Managers?

Repository managers like Archiva, Artifactory, and Nexus have become a critical piece of the DevOps toolchains and are commonly used across the entire application lifecycle. It is effectively a binary parts warehouse for your applications. Given how integral this warehouse has become, software supply chain intelligence and automation can be used to significantly reduce unplanned work and eliminate avoidable risk.

To understand this opportunity, let’s take a quantitative look at the world’s software development ecosystem.

Global Visibility

Not long ago Sonatype introduced a free feature in its Nexus Repository Manager that is providing new data on the health of software supply chains. This features, called Nexus RHC (Repository Health Check) – offers a simple way to get basic visibility into what OSS is flowing into development via your repository manager. RHC provides a summary of components in the repository that have known security vulnerabilities and risky license types. Usage of this service is now significant, with over 30 million components across over 15,000 Nexus instances being analyzed every day across the globe.

There are also around a million developers using those repositories. With the law of big numbers working for us, we can gain a statistically valid view into the behaviors of the modern software development ecosystem. The findings are quite telling.

The Data

We analyzed over 7,000 repositories containing 500 or more software components using the RHC service during the past 90 days. These repositories are representative of those used by development teams in medium to large organizations. The question was straightforward: To what extent are bad things flowing in, causing downstream rework and creating avoidable risk scenarios?

In three months time, the average number of new vulnerabilities that flowed into the repositories analyzed was 69. That is an average of 23 vulnerabilities per month, which is a little more than one every weekday. If you only include higher risk vulnerabilities – those with a CVSS score of 5 or greater – the average number was 48, or about 16 per month. That is a little less than one every weekday. For a sense of the kind of vulnerabilities in this higher risk category, the Heartbleed bug had a score of 5, the lowest risk in this grouping.

These are not isolated instances or outliers. In the set of repositories analyzed, 98.3% consumed at least 1 vulnerable component and 97.7% consumed at least 1 higher risk (CVSS >= 5) component. In a period of just three months, nearly every repository consumed something the organization running it would rather not have, and that is only with respect to known vulnerabilities. There are plenty of other attributes that are worthy of consideration given availability of the corresponding data.

Whether or not security is of particular concern, the real point is that a lack of visibility and control combined with an abundance of supply has led to inferior quality parts routinely being used by the world’s development organizations, adding avoidable risk and slowing us down.

Going (Way) Beyond Security

The consumption of vulnerable components is a quantifiable problem that we used to illustrate the current challenges that every organization has with open source component selection. However, security vulnerabilities are only one of the many dimensions.

Others include architectural aspects such as rules covering component age and popularity, intellectual property coverage with rules related to open source licensing and technical debt controls associated with technology stack selection. Introducing more formalized vetting processes coupled with software supply chain intelligence and automation will yield significant overall quality and efficiency benefits for an organization, allowing them to operate with a lower risk profile.

Yes, A Firewall for Repository Managers

In November, Sonatype became the first company to introduce a “firewall” for use with repository managers. Called Nexus Firewall, the solution offers perimeter quality control for software development. Similar to a network firewall, it leverages rules you define that automatically shield an organization from unacceptable software components entering and another set for stopping them from exiting your application development. The basic concept looks like this:

In order to build quality in from the beginning, this solution represents a “shift (far) left” innovation. The Nexus Firewall improves speed and reduce risk through the quarantine of components with known vulnerabilities. With a repository firewall in place, organizations can shield your application development from waste and risk by automatically blocking unacceptable software components inbound and preventing release of applications containing such components outbound. 1 in 16 downloads with known security vulnerabilities could be reduced to zero in 16.

The firewall also goes beyond blocking, providing organizations with the visibility and data needed to make ideal decisions for open source component selection early, significantly reducing waste related to rework and eliminating avoidable risk.

For DevOpsSec, Shift (Far) Left to Build Quality In

The consumption of these unacceptable or, at a minimum, inferior components is unnecessary. For the vast majority of these vulnerable components, there are non-vulnerable alternatives ready to be used in their place. The challenge is that it has been historically difficult for developers to understand these risks and even harder to avoid them. Few developers have countless hours to commit to researching vulnerabilities in the components they use and fewer more want to rely on colleagues in security, legal, or audit teams to spend hours or weeks to return with “go” or “no go” vetting decisions.

The volumes and velocity of component consumption we see today validates that any form of manual review process or automated workflows outside of the development teams will be quickly outpaced. DevOps and Continuous Delivery processes need to rely on automatic intelligence to help developers select the highest quality components and avoid those with known risks. When known security vulnerabilities and other risks are identified, software supply chain intelligence will be required to guide developers through the identification and selection of safer, higher quality components.

An example of the kind of information made available is shown below.

The information is divided into two areas. On the left side is component data, which includes details related to the component itself. To the right, there’s a graphical display of any security or license issues, as well as popularity data for each version of the component displayed. Selecting different versions updates this information accordingly, including whether or not a particular version passes the established criteria.

A Must for DevOpsSec: Quality at Velocity

Simply by enabling a repository firewall, organizations can immediately improve quality and reduce waste and exposure to risk. None of this comes at the expense of development speed.

The repository firewall automatically quarantines components that do not pass your rules preventing quality issues from entering the software being developed, immediately reducing risk and avoiding wasteful rework at some later point. Quality is improved, efficiencies are gained and risk is reduced, all through automation and all at the speed of modern development.

One of my favorite reference books is Continuous Delivery by Jez Humble and Dave Farley. Let me share a few of their words to help remind us of the importance of innovations and practices that allow us to bring pain forward. Jez and Dave remark:

“The techniques that we describe in this book, such as continuous integration, comprehensive automated testing, and automated deployment, are designed to catch defects as early in the delivery process as possible (an application of the principle “Bring Pain Forward”). The next step is to fix them A fire alarm is useless if everyone ignores it. Delivery teams must be disciplined about fixing defects are soon as they are found.”

Well said Dave and Jez. Well said.

December 2, 2015
The Software BOM Squad
In my previous post, “When Good Code Goes Bad“, I shared new research showing the average large development organization consumes over 15,000 known vulnerable and defective components annually. While we can’t stop software from going bad, there are practices from traditional manufacturers that we can use to improve our ability to recall and fix the “bad” software components.

The Software BOM

A Bill of Materials (BOM) is used in traditional manufacturing supply chains to list the suppliers and parts used in a product, a “software bill of materials“ (BOM) is an inventory of the third party and open source components used to build an application.

A sample software bill of materials lists all open source and third party components, including their version, age, license type, and any known security vulnerabilities.

As noted in Wikipedia, “The concept of a BOM is well-established in traditional manufacturing as part of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. If defects are later found in a specific part, the BOM makes it easy to locate affected products.

“A software BOM is useful both to the development organization (manufacturer) and the buyer (customer) of a software product. Builders often leverage available open source and third-party software components to create a product; a software BOM allows the builder to make sure those components are up to date and to respond quickly to new vulnerabilities. Buyers can use a software BOM to perform vulnerability or license analysis, both of which can be used to evaluate risk in a product. Understanding the supply chain of software, obtaining a software BOM, and using it to analyze known vulnerabilities are crucial in managing risk.”

A software bill of materials not only inventories what is used, but in some cases it also syncs with real-time component defect data to indicate which components have known vulnerabilities or license risks. Various software supply chain automation tools expand the bill of software much further, alerting stakeholders automatically when a defect alert occurs. In advanced tools, the entire software lifecycle is automated to ensure that defective components are avoided and continuous monitoring instantly identifies newly announced vulnerabilities as soon as they are discovered.

Breaking Bad

As mentioned in my previous post, When Good Code Goes Bad, there is no way to prevent software from “going bad”. No one is immune from these issues. Therefore, we need to focus on our responses to these issues.

To paraphrase Vince Lombardi, “It’s not how you get knocked down, it’s how you get up”. Following the OpenSSL vulnerability announcement in 2014 (or pick any other new security vulnerability with or without an assigned logo), every development and IT operations team on the planet was asking two questions:
1. Did we ever use that open source component?
2. if yes, where is it?
For those organizations tracking their use of components with a software bill of materials, the answer was easy to find. For other organizations, weeks and often months of research were required. To get a sense of the lack of maturity around traceability practices, take a look at this image from Sonatype research showing announcements of OpenSSL vulnerabilities in products following the initial vulnerability disclosure in April 2014.

Automating Traceability Across the Software Supply Chain

Software supply chain complexities, coupled with the volume and velocity of open source and third-party components flowing through them, are forcing us to think differently about our current practices across development and operations. We are not the first to face these challenges and we can learn a great deal from the mature practices of other high-velocity manufacturing organizations and industries.

One thing is certainly clear. Manual practices of monitoring or occasional checkpoints cannot keep pace with volume and velocity of components being consumed. And if you extend these considerations beyond just open source components and into containers, microservices, or other images used in development and operations, the problem is further out of reach for manual processes to keep pace. The volume and velocity within our software supply chains will not diminish—and without a new approach, the volume of unchecked quality and integrity of parts being consumed will continue to build up as technical debt.

But the answer is here today: automation. Automation can unleash the potential of an organization’s development capacity. Rarely is there such an opportunity to simultaneously increase speed, efficiency, and quality. Solutions that facilitate comprehensive software supply chain automation are poised to usher in the next wave in development productivity—with gains on par or even greater than possible with agile, Lean, and DevOps.

More information on the current state of software supply chain practices as well as examples of best practices being employed by software development teams, can be found in the 2015 State of the Software Supply Chain Report.
July 22, 2015
When Good Code Goes Bad

Milk spoils. Iron rusts. And in software, good code goes bad. Yet the difference is, with the first two, you know the change has occurred. With software, those changes are not always obvious.

Your 5,100 Binaries Went Bad

There is no way to prevent software from “going bad”. As with all products, bugs and defects,are bound to happen at some point. No one and no code is immune from these issues. But who’s looking for the 5,100 software components in your organization that went bad last year (meaning new security vulnerabilities were discovered in them)? It’s all too likely, no one.

Earlier this year, I took a deep dive into the analysis of software supply chains that fuel high velocity development practices and IT operations. The analysis revealed that some of the largest development organizations were consuming an average of 240,000 open source components to expedite development, accelerate innovation, and meet time to release deadlines. While reaping huge benefits from this consumption, these same organizations were unintentionally growing their technical and security debt. How you ask?

Here is a summary of the their average debt:

Source: 2015 State of the Software Supply Chain Report

The analysis showed that over 15,000 (7.5%) of the open source components being consumed by these organizations in 2014 had known security vulnerabilities. Of those 15,000 components, an average of 66% (9,900) had known vulnerabilities dated 2013 or older. That means, they were known vulnerable components (“bad”) before they were downloaded.

The remaining 34% (approx. 5,100) might have actually been “good” components at the time they were downloaded by development teams from public open source repositories, but at some time during 2014 a new security vulnerability was discovered and a CVE identifier was assigned.

Even with the best controls in place to protect developers from downloading open source components with known vulnerabilities, the fact that new vulnerabilities are continuously discovered — about 50 new CVEs are announced daily — means that continuous integration and delivery practices also need to be complemented by continuous governance and visibility practices.

We’re Not Alone

Software development is not the only industry facing such issues of things good things going bad. I bet all of us have received notification from our car’s manufacturer that a certain part is being recalled. While the part was known to be safe when the manufacturer sourced it from suppliers, placed it on the car during assembly and shipped it, discoveries of defects are often found years after the vehicle has been on the road. The same scenario has been played over and over in the food, pharmaceutical, defense, toy, electronics, and other industries.

The difference is those other industries are known to have more mature supply chain practices where they can track and trace every part or ingredient ever used in every product they have shipped. And when a part goes bad, they can recall and repair that defective component.

In software, track and trace practices are still in the earliest stages of maturity. In a 2014 survey of 3,300 development professionals, 63% stated that their organizations have incomplete or no records of the components used in their applications. This lack of traceability means that even when good components go bad, development and IT operations teams have no way to quickly find the defective component and replace it with a better, safer version — dramatically extending mean times to repair, or in the case of no action, growing their technical and security debt. And when applications with those defective components are in the hands of customers, the ability to “recall” the impacted software or distribute newer versions of the application are hindered.

Did he recommend a “Bomb”?

A Bill of Materials (BOM; pronounced “bomb”) is used in traditional manufacturing supply chains to list the suppliers and parts used in a product. According to Wikipedia, “The concept of a BOM is well-established in traditional manufacturing as part of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. If defects are later found in a specific part, the BOM makes it easy to locate affected products.”

In Part 2 of this blog, “The Software BOM Squad“, I will introduce the importance of automating the creation of a “software bill of materials“ (Software BOM) to take inventory of the third party and open source components used to build an application. When software components age more like milk than wine, a software bill of materials can help you quickly identify where your 5,100 “bad” component were used.

July 22, 2015
The Cost to DevOps: 27 Mufflers

Imagine

Imagine that you are designing the 2016 Range Rover line of sport utility vehicles. Like all gas powered vehicles, each one needs an exhaust muffler. Range Rover likely has narrowed in on a preferred provider of mufflers. But imagine what would happen if the designers and factory line workers could pick from any one of 27 past versions of that muffler from their preferred provider for the new model year. Yup, choose any one — even if it is outdated, lower performance, does not meet current emission standards, or has a known defect. Order it, place it on the vehicle, then ship it.

We all realize this scenario would never happen. No one would select a known defective part for the product they are manufacturing. And who would knowingly choose the 10 year old version of a muffler without the latest advancements to support air quality? No organization would give a each of their designers and line workers free reign on procuring any one of 27 different mufflers for a single model year. Imagine the operational cost of testing the different mufflers. Imagine the maintenance cost of having different versions of the same part shipping. Worse yet, imagine the liability of shipping parts with known defects.

Land Rover is not taking this approach, but software development teams are.

27 Versions

My latest research report, the 2015 State of the Software Supply Chain, reveals that large software development organizations sourced an average of 27 different versions of open source components that they used in development last year. To be more specific, out of the top 100 open source components these companies downloaded from the internet in 2014, they averaged 27 versions of each component.

While auto manufacturers like Land Rover have preferred suppliers of vetted parts, the procurement of open source components is often more of a free-for-all. If an organization has 300 developers, they effectively have a procurement department of 300 for sourcing externally developed software components. If you have 4,000 developers, you have 4,000 performing procurement. Even as we all know only the latest version of a components is the most feature rich and of the highest quality and integrity.

Not all organizations allow free-for-all sourcing practices for components. Fifty-seven percent (57%) have open source policies in place as a method to guide developers toward higher quality parts with the fewest known defects or security vulnerabilities. Organizations like Google mandate the use of no more than two versions of a given open source component across their development teams.

Good Cost, Bad Cost

What does this mean for a software development organization — especially one focused on improving it’s DevOps practices? First the good news: developers are using open source components to speed time to release and improve innovation. Both good things.

The bad news: Organizations are building mountains of technical debt through poor software supply chain practices by sourcing an average of 27 versions of the same component, impacting quality, complexity, maintainability, supportability, etc. They are doing this across the top 100 components being downloaded in a single year which equates to managing over 2700 different versions (when an optimized practice would only require the 100 best).

Organizations are sourcing component “parts” for use in their applications that have up to 26 known better versions available (more features, fewer bugs, improved performance, etc.).

Organizations are downloading and using parts that are sometimes 9 years old, or even older (simple math: open source projects release 3 – 4 new versions of their components each year). They are electively using outdated parts.

Organizations are also downloading some versions with known, documented security vulnerabilities. They are electively using known vulnerable parts.

Net Innovation Through Improved Sourcing

While we often hear people tout a DevOps mantra of “even faster”, there is a consequence to pursuing speed at any cost: technical and security debt.

Open source components are enablers of speed. But free-for-all sourcing and use practices, add to technical and security debt that reduces “net innovation” and adds to the cost of operations. As technical debt grows, net innovation and business value drops.

Image source: http://joapen.com/blog/2014/06/17/how-to-monetize-application-technical-debt/

If reducing the use 2,700 versions of 100 components was not motivating enough, consider this: the research also revealed that large development teams were consuming software components from over 7,600 external open source projects (suppliers). Including all versions of those components, they average organization consumed nearly 19,000 unique components in 2014.

Improved sourcing practices are just part of the overall story for improving net innovation and increasing business value. Improved sourcing practices can also enable more efficient operations, reduce mean times to remediation, and eliminate unplanned work for many DevOps practices. Improved sourcing practices have been employed across many industries resulting in sustained competitive advantages. Improved sourcing practices are part of a broader software supply chain management strategy.

Continuous Learning

There are so many lessons we can learn from other industries who have automated and optimized their supply chains — like Land Rover and other auto manufacturers. I’ll be sharing more of those lessons and perspectives from the 2015 State of the Software Supply Chain here this summer as well as expanding the discussion around “net innovation”. Stay tuned.

July 20, 2015
The Myth of DevOps as a Catalyst to improve Security?

Few topics within DevOps discussions elicit more controversy than the relationship between DevOps and information security. Do DevOps practices help to improve security, or do most organizations simply automate and bring their existing bad internal processes along into DevOps? With this topic in mind, I recently had an engaging conversion with Adam Muntner on this and more, and thought it would also be of interest to devopsy.kinsta.cloud readers.

Adam Muntner is a Security engineer at Mozilla and the creator of FuzzDB, an open-content licensed attack and discovery pattern database of test cases for software security testing and research. He has 20 years of experience at security governance and management, technical assessment, research, and consulting with a focus on secure software development. The opinions expressed are his, not his employer’s.

Here is our recent conversation on DevOps and security, edited for brevity and clarity.

devopsy.kinsta.cloud You’ve taken issue with some of the claims that DevOps can help to improve security. For example, my writing, “several years into enterprise DevOps deployments, what we are actually learning is that unifying development and operations teams through DevOps efforts does not create the wild west environments feared,” in my post DevOps Security Comes of Age. Another was from this Q&A with Gene Kim where he said that high performers are able to conduct more frequent code deployments complete their deployments from code committed to running in production more quickly, that they enjoyed twice the change success rate, and that outages were fixed 12 times more quickly.

I had used these findings to support DevOps as helping to improve security outcomes. You took issue with these statements. Why?

Muntner: I’m not opposed to DevOps or packaged deployment, rather I’m for taking a holistic view of benefit and risk from an informed perspective. The thing to keep in mind is that the security of an application environment is inherited – it’s the aggregate result of all its component pieces.

When system architectures are so complex that using opaque, componentized, pre-built pieces in an application environment becomes the norm, how do you know what’s in them? How much attackable surface area is being inherited?

Gene Kim is a smart guy and chose his words carefully with regard to security. Gene said,

“The first one is the high performers were massively outperforming their non-high performing peers. Thirty times more frequent code deployments. They were able to complete the deployments from code committed to running a production 8,000 times faster. In other words they could do a deployment measured in minutes or hours. Whereas low performers it took them weeks, months, or quarters. Furthermore, they actually had better outcomes. Twice the change success rate and when they caused outages they could fix it 12 times faster. There were 4,200 that completely responded and it just showed that you could be more agile and reliable at the same time. I think if we just connect some dots, they’re also going to be more secure, they’re probably going to have a faster find fix cycle time. We didn’t actually test that but, if that were true it would mirror what we found in the previous benchmarking work I had done.”

The old engineering maxim of “good, cheap, fast” has not been obsoleted by DevOps. Deploying code 8,000 times more quickly is not a measure of risk reduction. It might help get fixes out faster, but that doesn’t tell the whole story.

Gene said, when they caused outages they could fix it 12 times faster,” but how many of the outages are the result of faster deployment cycles? What does it mean to fix an outage 12 times faster than if the outage had never occurred due to a different development practice? Fixing outages quickly is good, that’s what is seen. It doesn’t take into account the unseen – newly introduced weaknesses that aren’t being tested or measured for.

How much of what was measured resulted from the ex-Devops security practices of organizations measured and resultant outcomes? What does “better outcome” mean, and is that even objectively quantifiable across different organizations or even situations in the same organization? Are we confusing correlation and causation? Every organization is different. The metrics are being held up as quantitative proof but it’s not certain that they are measuring what they purport to.

Consider this scenario: Operations staff are incentivized through the salary review/bonus process to maintain uptime. Despite working in the test environment, the latest code push to production broke your application. DevOps reinstalls the Docker and the problem is still there. What now? Go back to the old version that ran without crashing but (possibly unknown to your operations team) contains a known vulnerability in some library that is static-compiled into a binary?

devopsy.kinsta.cloud: What about the argument that DevOps helps to reduce code bloat from legacy environments goes a long way to reducing both technical and security debt?

Muntner: Reducing bloat might be one of a number of goal of a devops team, but devops practices are just as likely to increase code bloat, opacity, and attackable surface. Security is not an inherent property of DevOps practices. For example, the perception that software which can only be installed through prebuilt packages increases simplicity only creates the dangerous illusion of simplicity.

Relying on these opaque components increases code bloat in unpredictable and unknown ways, potentially decreasing security posture. If there even is a threat model (usually not, they’re hard and time-consuming to produce, thus expensive) then a popular software package like Hadoop probably won’t more than a box in a threat model, when in reality it’s an aggregation of many components and n-th order subcomponents, each with their own attackable surface area, depending on configuration. If it’s being installed as part of a prebuilt environment, there are many possible security consequences and none of these are addressed or measured in any of the types of tests or metrics that have been described.

Performing thousands of tests sounds good, but what if tens of thousands of tests are necessary? Or a completely different testing methodology and toolset? Where are will the false negatives be based on the limitations of the existing set of tests? And most important, do the metrics support the claim?

There are many categories of attacks that these tests aren’t able to identify: Logic flaws are one. Different components of a system understanding the same piece of (malicious or spurious) data to mean different things, another. Sometimes the problem isn’t a bug, it’s architectural deficiency. If it’s deeply layered inside a component of your system that uses its own non standard build system, you probably won’t find it. Software tests are for known, expected code execution paths and interactions, it’s much harder to identify all possible execution paths and orders of operations.

DevOps: But all of those flaws can be found later in production, when they exist. It is not as if those manually tests are being done away with.

Muntner: Typically, security audits and assessments are time-boxed. It’s often hard to assess an entire large application inside the time box, let alone all the many opaque components. And then there is the question of pen testers relying on commercial automated tools where even the tests the tools are performing and the included test cases are opaque. Ultimately, they are a snapshot of the security posture at a point in time, given the limitations of the tester(s), tools, methodology, time, and ability of the organization to efficiently consume and triage testing results and lessons learned. Given the rapid release cycles implied by Devops and Agile development, the results can go stale fast.

Adding unknown layers of dependencies and libraries stretches those already-thin review hours even thinner. Security resources are scarce and expensive. Organizations with mature application security assurance programs only apply those resources to the most critical applications. Outside consultants reviewing one application could cost $20,000 or more.

Automation is very critical but automated tests have big limitations and often miss configuration management details. Most important, not all vulnerabilities get found. If the automated tools don’t test for it and the pentester didn’t find it within their testing window, it’s still there. Good penetration testers are always finding bugs that weren’t found during previously time-boxed assessments. And so are malicious actors.

devopsy.kinsta.cloud: All of these possible conditions exist in enterprises now. And certainly, automating bad practices could more rapidly propagate a mess, or thinking it through and automating the right things could improve real outcomes. So why can’t this prove to produce more resilient code and architecture than current methods?

Muntner: Thinking security testing through and automating as much as possible will yield results, but that can happen with or without devops. I’m not saying devops is invalid, rather that it alone is not responsible for good outcomes. Thinking that an approach delivers more than it really does is only a false sense of security, arguably worse than awareness of insufficient security.

Secure systems and software development practices like command-safe APIs, network-layer features in TLS, HTTP layer features like CSP, improvements in application and protocol layer firewalls, developers learning to do proper encoding for the appropriate output context, automated testing with tools like OWASP ZAP or commercial equivalents as appropriate for the type of application are all high-impact but have nothing to do with devops.

devopsy.kinsta.cloud: Security should be part of the flow, an integral  part of QA, security and functional testing.                

Muntner: Security isn’t a state, it’s a process. It’s a verb, not a noun. Security ‘what’ should be part of the workflow? Security activities and tests, personnel, all of the above? Should a security organization report to the business management and governance side of management, or the technical side? And why is DevOps better for security maturity than separation of duties?
DevOps is the approach we take at Mozilla, but the IT and product organizations at Mozilla are full of exceptionally talented people who would do a good job with security using any development process.
devopsy.kinsta.cloud: Depends on the organization, I’m sure, but if an organization is going to move to DevOps than DevOps needs to be part of the processes.

Muntner: DevOps can be good or bad but DevOps doesn’t have much to do with security in and of itself, except for some of its popularly misunderstood practices, and then maybe not always in the ways people expect. Standardizing is good unless you standardize on terrible things.

devopsy.kinsta.cloud: That’s certainly a risk. For many companies they are implementing DevOps regardless, so it’s a matter for security working toward the best way to integrate themselves and their processes.

Muntner: IT is a business support function. Security is a business risk analysis function. If you standardize and integrate things without understanding threat, risk and security posture, what have you done? Ultimately the decisions are business decisions. Unfortunately, they are frequently made from the perspective of insufficient knowledge.

There is nothing inherent to DevOps that solves these problems and DevOps comes with its own potential pitfalls.  The problem is external to what DevOps can deliver.         For organizations that want to better understand the maturity of security in their Software Development Lifecycle, OWASP OpenSAMM is a good place to start. It helps organizations understand security maturity means for them specifically and incrementally advance towards their desired posture.

July 16, 2015
Managing Open source software components
Building any software need lots of efforts including resources, time, money, etc. It is really a great pleasure when it goes live or gets released. In parallel, there is always a chance that bugs may put the release in a difficult condition even after multiple rounds of testing. Teams can fix bugs related to software features or functionality, but the ones which can hit badly are
- Security vulnerability.
- Licensing risk of open source component.
- Outdated open source component.
The world is moving towards open source in a very fast pace and its growing use in software as components inside the application. Open source components are available free for us, however there could be more risk while using them. A product which is paid and commercial is bound to fix issues and provide adequate support for any vulnerability or security issue and Licensing, But open source software may have a little more of a high risk and its very important to be updated about new releases and available bug fixes.

If open source components are not monitored properly and keep up to date, that may lead the software to become vulnerable.

Above checks needs to be done during the development phase and ruled out any discrepancy related to security or legal. This check can be included as part of build process to make sure it does not pass the fist barrier and caught even before hitting the testing phase. Eventual this can save development and testing effort and their time.

There are tools available which can take care of some/all of verification of open source components in various phases.

Artifactory

This tool can verify the license of the open source component and check its compatibility with various definitions. You can even add your own license definition as per your requirement.

Sonatype Nexus and WhiteSource

These tools will verify the license and keep the component information up to date. They will check for any updates, bug fixes for any vulnerability issues and notify the user. These tools can do component security vulnerability and license analysis with the latest available information.

Security issues would be the last thing any software developer wants. Even small legal or security problems can put your system/software in a dangerous situation. Better to secure software first then fixing it later.

Reference ->

http://www.scmtechblog.net/2015/06/managing-open-source-component.html

http://www.scmtechblog.net/2015/03/maven-repository-tools-comparison.html
June 30, 2015
Chevy and DevOps: What the Wi-Fi?
In DevOps: Rapid Feedback, Rapid Repair

I’m sure you have seen it too. Chevy cars now come with wi-fi. How cool. I want that (and so would my kids). I can only imagine the possibilities.

But, this is not all about my needs. Chevy and every other vehicle maker wants this too. And not for the reasons that you might first consider. Quickly, let me introduce you to the recalls of today:
The glitches were related to cars losing power, erroneously deploying airbags, and other defects. This is just a small glimpse of the software recalls that have happened over the past year.

But more troubling is this form of software “recall” from BMW showing a security flaw that could allow cyber criminals to unlock doors on 2.1M vehicles. Because every one of the 2.1M vehicles has the same defective software part, criminals effectively had access to a skeleton key to every one of them.

In DevOps: Rule Your Supply Chain

The reality is, that your new car has more software in it than you can image — about 100 million lines of code. And, as we know, all software has flaws. But who wants to go to a dealer to get a software update? If my car can connect to the internet and update itself, that would save me an unscheduled trip to the dealer for a repair.

While software updates to cars models in the past were would take 5 years to feed themselves onto the production line, that is now all changing. For decades, large automakers like Toyota have been applying excellent management process for their entire supply chain. And with the introduction of wi-fi in vehicles, they are expanding their highly-tuned practices to include their software supply chains. They know where every piece and version of their software is and who owns it. If a flaw is detected (be it quality, safety, security, or another attribute), they will now be able to automatically update that vehicle. Not only can they remove defects, automakers like Telsa are even making their cars faster.

While many of us outside the auto industry think of these large industrialists as slow moving organizations, the introduction of wi-fi in cars is allowing them to now apply a core principle from the agile manifesto: “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”.

In DevOps: Version Everything

For those of you in the software industry, this kind of evolution is at the heart of many DevOps and continuous delivery efforts. Automate everything you can. Version everything you can. Deliver fast, improve visibility, and change fast. In many ways, the addition of 4G LTE Wi-Fi in Chevy Trucks means that DevOps has entered into our auto and software supply chains.

If you imagine an even broader perspective of this wi-fi feature and software supply chain management, the auto makers are now leading the way to expectations of quality improvements in the Internet of Things (IoT). There is not much difference between an internet enabled truck, thermostat or fitness bracelet when it comes to rapidly improving and updating quality, features, performance, etc. (I’ll cover this topic more in future posts.)

In DevOps: Optimize Your Software Supply Chain

What have the auto makers learned from the software industry? And perhaps more importantly — when it comes to software supply chain management and continuous delivery — what can we learn from them?

A final note on software and auto safety: I am the Cavalry (@iamthecavalry) is encouraging the automotive industry to commit to cyber safety. The movement recently published a Five Star Automotive Cyber Safety Framework (PDF download) and calls for Automotive Industry adoption. Media outlets from across the US and Europe praised the effort and joined in calling for automakers and security researchers to work together to ensure a safe future.
April 28, 2015