devopsy.kinsta.cloud

Tag: rugged devops

DevOps Connect: Rugged DevOps @ RSA Conference
Once again this year devopsy.kinsta.cloud along with Sonatype, The Nexus Community, Gene Kim, Josh Corman and Mark Miller are producing DevOps Connect: Rugged DevOps @ RSA Conference.

The full day event takes place at the Moscone Center on the Monday of RSA Week, February 29 (it only comes once every 4 years), 2016. After the day packed full of sessions there will be a cocktail reception* as well. *Cocktail reception requires separate registration.

Last years event had over 700 people in attendance and featured a great lineup of speakers. This year’s lineup is even better we think. It features leaders from both the DevOps and security communities with some great practitioner accounts of how DevOps helps security and how security helps DevOps.

Thanks to the generosity of our friends at RSA Conference, we have a larger room this year to accommodate more people. Anyone with any type of RSA pass is welcome.

What’s that, you don’t have an RSA pass? No problem. Use code 1U6DEVXPO when you register and save $100 off of an RSA full Expo pass. This should make your pass free if you register right away or only $25 if you register later.

With your RSA Expo pass not only do you have a ticket to to DevOps Connect: Rugged DevOps event, but you have a ticket to a full week of attending the RSA Expo floors. Sounds good to me. You can head over to REGISTER here. But don’t wait, there are less than 30 days to RSA!

Also you can get a lot more info about the speakers, abstracts, backgrounds etc. on the DevOps Connect: Rugged DevOps website.

So don’t wait. Register today as seating is limited and on a first come, first serve basis. Looking forward to seeing you at DevOps Connect: Rugged DevOps!

Here is a quick look at the full agenda:
- 8:30 AM – 8:50 AM |Doors Open
- 8:50 AM – 9:00 AM |Welcome & Overview | Alan Shimel, Mark Miller
- 9:00 AM – 9:20 AM |Chocolate and Peanut Butter (or some such) | Josh Corman, John Willis
- 9:20 AM – 9:40 AM |What’s Hiding Under Your DevOps Rocks? | Jez Humble
- 9:40 AM – 10:00 AM |The State of DevOps Data Science | Nicole Forsgren
- 10:00 AM – 10:40 AM |The Darker Side of DevOps and Security | Bruce Potter
- 10:20 AM – 10:40 AM |Break
- 10:40 AM – 11:00 AM |2016 in Review: Failures in Public Safety and Privacy | Kim Zetter
- 11:00 AM – 11:20 AM |Seven Habits of Rugged DevOps | Amy DeMartine
- 11:20 AM – 11:40 AM |Applying DevOps principles to address dynamic changes in Cybersecurity | Aaron Volkman, Hasan Yasar
- 11:40 AM – 12:10 PM |Panel: “DevOps Engagement: Politics, People and Process” | Goerlich/Thrasher/Corrierre/Wang
- 12:10 PM – 1:20 PM |Lunch Break
- 1:20 PM – 1:40 PM |The Journey to DevSecOps | Shannon Lietz
- 1:40 PM – 2:00 PM |Rugged DevOps: Compliance at Velocity | Justin Arbuckle
- 2:00 PM – 2:20 PM |Building Security Controls Around Attack Models, Not Gut Feel | Stephan Chenette
- 2:20 PM – 2:40 PM |Release Engineering’s Role in Rugged DevOps | J. Paul Reed
- 2:40 PM – 3:00 PM |Silver Linings for Miles: DevOps for Building Secure Solutions | Andrew Becherer, Zane Lackey
- 3:00 PM – 3:20 PM |Break
- 3:20 PM – 3:40 PM |Security Wargames | Sam Guckenhiemer
- 3:40 PM – 4:00 PM |Bridging Security and DevOps using an Attack Driven Approach | Earnest Mueller, James Wickett
- 4:00 PM – 4:20 PM |Architectures, Design Patterns, and Coding for Rugged DevOps at Scale | Rich Mogull
- 4:20 PM – 4:50 PM |The R.O.A.D. to Rugged DevOps for a Major Airline | Dan Glass
- 4:50 PM – 5:00 PM |Final Thoughts | Alan Shimel, Mark Miller
Also a special thank you to our sponsors:

HPE, Illumio, Cloud Passage, Evident.io, CA Technologies, TwistLock and RSA Conference!
February 2, 2016
Rugged DevOps: Good, Bad & Ugly

The Good: DevOps has been a success for Dev.

With higher expectations, better processes, more management focus and new tools like advanced PaaS and containers, developers are churning out more and more software faster than ever. Success! Or is it?

The Bad: DevOps has been a success…for Dev.

For Dev teams, this may result in pats on the back and high fives. But for Ops, this can look like one thankless tidal wave of work after another. The same software velocity that is praised in Dev is currently overwhelming their counterparts in Operations. And frankly, many DevOps tools released in recent years happen to benefit Dev much more than their counterparts in Ops. (My pet theory is that developers at IT automation firms are acting and thinking like…developers – focusing their creative energies on solving Dev problems.) A similar problem holds true for IT Security teams who want to build quality in by “shifting left” – see https://devopsy.kinsta.cloud/2015/12/10/getting-rugged-devops-right/ . From the vantage point of IT Operations and Security teams, they are doing more work than ever but without any more human or IT automation resources. To add insult to injury amid their increased workloads, many IT Ops teams have to invest countless hours (away from family and personal time) in training courses and migrations to reorient their processes to fit into shiny, new Dev-centric “DevOps” tools and paradigms.

The Ugly.

Work. Overwork. Glitch. Search. Find. Rework. Repeat.

A quick glance at IsItDownRightNow.com will show you which IT Operations teams are in a living hell at the moment. The other thing you will notice is that these IT Ops teams are from some of the most reputable companies on the planet. They are talented teams but stuck in rework cycles of web performance problems, security vulnerabilities and outages.

These problems get the attention of everyone. Operations…and Dev…and Security…and customers…and Finance…and auditors and the press.

No one likes rework. And no one likes the panicked fire drills that cause panicked rework. So what causes those rework cycles?

Drama-free IT. Visibility Cures Chaos.

Much of this chaos can be tied to miscommunication. Boring? Maybe. Simple? No. Part of miscommunication is cultural. (Hint: OrcaConfig’s “How to Screw Up DevOps” white paper covers the cultural aspects of miscommunication). But miscommunication is also cured by automation solutions that promote transparency and visibility.

If we want drama-free IT we need IT automation solutions that eliminate miscommunication and missed assumptions between Dev, Ops and Security.

What better way to do this than with full transparency and visibility?

Bird’s Eye Views:

Many IT automation tools cater to SysAdmins who are interested in managing configurations of individual servers. That’s great if your focus is on individual severs. But IT Operations, eCommerce teams, Application Owners, Compliance & Security teams and IT Managers often care more about the health and performance of their application ecosystem (including related applications, middleware, databases and configurations that tie them together). So there is often a mismatch between the tools that SysAdmins and Developers have versus what IT Ops needs.

Operating at Enterprise Scale Requires at-a-glance Visibility:

IT Ops teams need bird’s-eye views of applications, databases, middleware and ecosystems to promote ease of ongoing operation. They need to be able to see, at a glance that a configuration compliance problem exists and they need to know right away the location and nature of the error. Large scale IT Ops teams do not have the luxury of toggling back and forth between multiple reports and multiple screens searching for that non-compliant, broken needle in an ecosystem haystack. If they are faced with a glitch or an outage, they need to find and fix it right away. One size does not fit all. Dev, Ops and Security each have their own interests and their own need to view their application through a lens that is most meaningful for them. Multiple ecosystem views eliminate guesswork between Dev, Ops and Security and save precious time in the find-fix cycle.

Forrester Research spells it out well in their 7 Habits of Rugged DevOps white paper, Habit 1: Increase Trust And Transparency Between Dev, Sec, And Ops:

“I&O pros pride themselves on understanding the interconnectedness of the hardware and software environment, and they are unnerved by complexities that poorly governed delivery practices create because the firm rates I&O pros on their ability to improve the performance and stability of the production environment. To maintain the environment, they must reduce the chances of an outage or interruption of service, and they must reduce the MTTR when an issue does occur.“

Understanding that interconnectedness depends greatly on how much intuitive ecosystem visibility we provide to our colleagues in IT Operations. But it’s not just IT Ops that benefits from visibility, Dev and Security do too.

Developers are able to write more secure code when they get early, in-context feedback. As David Mortman, distinguished engineer and Dell chief security architect, said, “Security issues are product quality issues, and no one wants to write buggy code.”

Even better than efficiently finding and fixing errors is of course to prevent them in the first place and to proactively eliminate the damage they cause. Here too, the answer is better visibility. With true application ecosystem visibility, Dev and Ops teams would be able to model relationships between applications, middleware and their configurations so users will know that “changing this, affects that”.

My teenage son has better visibility via his smartphone as to his driving route and weather conditions than I have for my multibillion dollar IT operation.

Visibility into the Future:

Ecosystem visibility is a key to communicating between Dev, Ops and Security and running complex application environments in the present. But what about the future?

Release automation tools and custom scripts are excellent at deploying changes across large environments. Unfortunately they can be just as efficient at deploying typos and other mistakes across large environments, causing the chaos that we discussed earlier. Here again Operations teams would benefit from improved visibility – the ability to preview and pre-validate proposed changes across the entire ecosystem well before those changes are deployed. This is the real promise and savings of ecosystem visibility. Better visibility, with more use-case lenses cuts out the miscommunication that causes too many performance glitches, security vulnerabilities and outages. This chaos takes a toll on our IT Operations teams, our employers and our customers.

What Next?

We are soon entering a new season with new budgets. The temptation to “buy something” will be great. In the spirit of Rugged DevOps my recommendation is to take a full, bird’s eye view of your SDLC and to shift Ops and Security leftwards, building their critical considerations into your processes and into your IT automation tool selection.

December 22, 2015
Making the Case for Secure, Defect-Tested Software Development

Creating a software security initiative in any organization is no easy feat. Often times, organizational culture or politics can provide development managers with a strong counterargument for implementing software security concepts. Unfortunately, building software without a consideration for security has become a less viable option given the increase in compliance pressures and widely publicized data breaches leading software consumers to expect more security from developers.

There is no way around the fact that attacks have become increasingly more sophisticated and applications have become the central avenue of these attacks. As a result, organizations are increasingly confronted with the task of assuring secure, defect free, software development. Leaders within these organizations are finally coming to realize that vulnerable and defect rampant software undermines the productivity, privacy and security of their businesses, employees and consumers alike.

As organizations are beginning to accept the fact that there is a need to test for best practices and defects throughout the development lifecycle, it is important that they are provided with the resources to make this happen. To change the way large organizations build software, an enterprise-wide initiative is required. At its core, an initiative to change the way an organization tests and validates its development processes and practices are a fundamental business process improvement effort.

Typically, large organizations with unique operational requirements find themselves building custom software systems to address their specific needs. But building custom software on time, on budget, and without bugs is difficult. Building software that also complies with an organization’s software security policies presents an even more difficult challenge. Although the tools and practices to build secure software are maturing, most organizations find internal hurdles to be more daunting. Organizational impediments including culture, differing software development approaches, and short-term business drivers make it more difficult to effect meaningful change. In order for a software security initiative to be successful, organizations must take a phased approach that considers organizational culture at each step.

Sure, there is the option of treating the symptoms by deploying web application firewalls or running an automated scanner against applications, but this does not get to the root cause and solve the problem for most organizations. It highlights that there must be deeper process improvements involving the systems development life cycle, particularly for higher level business logic or authorization vulnerabilities.

Despite the hurdles that exist when approaching the issue of secure development, most organizations do realize that there is a problem that needs to be addressed. Organizations focus on technical means to write more secure code and strategies for putting controls around the software. The next step is to then educate executives on the process of leading a software security initiative as these initiatives are most likely to fail due to organizational issues, not technical issues. This means taking a disciplined approach by characterizing the landscape, securing champions, defining standards and strategy, executing, and then sustain the effort. These steps, tailored to the way an organization operates, will help ensure that corporate-wide efforts to secure applications are as productive as possible.

So in conclusion, it is not an impossible feat to address security at the root cause, software development, as part of standard operating procedure. It just takes a little forward thinking to demonstrate its undeniable value and get the corporate buy-in needed to change the development culture. That being said, once the culture is changed don’t rest on your laurels. Always observe what’s occurring within your organization to determine whether there are new technology risk areas emerging. If you simply focus only on well trodden areas like web applications, you may find yourself behind the 8-ball once again when it comes to new advances in software development, like mobile and the Internet of Things.

December 16, 2015
Getting Rugged DevOps Right
Two Perspectives

Jack, an accomplished application security pro, tells me, “The developers won’t talk to us. It’s like we speak a different language. They are releasing new builds so fast, how could they check each one for security vulnerabilities? We can’t move as fast as they do.”

Then in the next moment, Diane, a DevOps pro, let’s me know, “Our current security team’s tools and practices can’t keep up with the pace we have now established. If it doesn’t fit, it doesn’t get done here.”

I have heard this little ditty ‘bout Jack & Diane so often in the past two years, it could be a hit record. I imagine you have heard it too.

The Slow Lane Won’t Lead to Rugged DevOps

Let’s visit Jack’s world, first.

When visiting a well-known bank last month, they shared insight on their current application security practice. They have 10 full-time people employed (offshore where personnel costs are lower) to analyze a set of 100 applications. The bank has over 2000 applications, but the security practice can’t scale to access their entire portfolio. In 2.5 years, they had only been able to cover 5% of the application portfolio.

The application security team employs an application security tool originally built for waterfall-centric development timelines. As much as they want it and need it to move faster, it is not really for “continuous” velocity.

Each scan of an application can require anywhere from four hours to two days to get an assessment report. The reports average about 30,000 potential defects which then require further analysis. That analysis may take up to two weeks in order to sift through thousands of false-positives and false-negatives. While this security practice is critical to the bank, it cannot keep pace with development practices that are aiming to go much faster in order for the bank to remain competitive in their industry.

The Continuous Gap

Sound familiar? It should. This scenario is playing out all across the DevOps and Continuous Delivery landscape. It is reflective of what I call “The Continuous Gap”.

Leaders in Rugged DevOps practices aim to minimize the “Continuous Gap”.

In Diane’s world, DevOps teams pursue “shift left” strategies in an effort to build quality in from the beginning. Shifting left enables them to bake in the right code, components, configurations, and practices from the beginning. When done right, velocity increases dramatically, while operating costs are minimized, and unplanned/unscheduled work is reduced. Investments can then shift more in favor of innovation over upkeep and maintenance.

The Old-School Rub

Jack’s world of traditional application security practices often wait to check for security vulnerabilities at testing or release stages of the SDLC. There are two reasons for this: (1) some security tests like static or dynamic analysis can take hours or days to perform and (2) security teams cannot embed themselves earlier in the SDLC (due to political, organizational, or operational obstacles) without dramatically slowing development in a high velocity world.

If you are using old-school application security tools, you will continue to be blocked from getting more deeply entrenched in development. New velocity requirements call for new approaches.

The New-School Shifts Left

A number of newer application security solutions and practices been introduced recently that are development and developer-centric. For example, imagine if build managers could analyze every open source component used in every build for known security vulnerabilities.

What if, developers could access information inside their IDE that would inform them if a component they were planning to use in an application was known to be vulnerable? What if developers had access to spell-check like alerts of security issues as they were creating their code?

Furthermore, imagine if information was available that not only told the developer of a known security defect, but directed them to an alternative safer version of that component to use. Components would be automatically vetted against corporate security policies at the instant they were chosen by a developer. When 90% of an application is composed of open source and third party components today, automatic analysis early in the lifecycle save millions of dollars. This how Rugged DevOps rolls. This is Diane’s world.

The evolution of Rugged DevOps is forcing application security practices to shift left.

But Diane’s world (and your’s for that matter) are not just about increasing velocity. All organizations need to consider operational costs to both detect and remediate security vulnerabilities in their applications. We all realize that the further along we are in the application development lifecycle the more costly fixes become. Feel free to apply your own cost figures to the chart below, but you should also note that these calculations do not include breach cost, help desk calls, maintenance, time-to-market, reputation or stock price.

Cost considerations for Rugged DevOps practices compared to waterfall approaches.

Rugged DevOps: Shifting Gears…Quickly

Shifting from old-school to new school approaches doesn’t take long. On a recent visit to one of the largest insurance companies in North America, they shared their tale of transition. Of five major applications that run their business, one in particular had 40,000 different files that needed to be analyzed. Their old-school approach involved a lot of manual code investigation and required about two-and-a-half weeks to complete the analysis of that large application. The new approach that used automated analysis of security vulnerabilities took two minutes — and produced identical results.

A government tax office wanting to shift security analysis of applications from hours (outside of development – Jack’s world) to seconds (inside of development – Diane’s world) was able to complete the first pass analysis of their existing portfolio of 600 applications within two working days. The security team was not only faster at identifying known security vulnerabilities in applications, but could also quickly identify alternative open source and third-party software components that were safer to use. The security team was not simply discovering issues, but the new class of Rugged DevOps solutions were aiding in guided remediation.

Questions to Ask About That Rugged DevOps Solution

There are several companies and open source projects that now offer application security solutions that work at extremely high velocity. When exploring solutions to match the velocity of your Rugged DevOps practices, consider asking the following questions:
1. How long does it take to complete the analysis of a small, medium, and large application?
2. Do developers use these tools, or are they only meant for security personnel? Do you have members of development teams that act as positive references for your tool?
3. Does the analysis require further investigation of the findings, or does it pinpoint the root cause of the issue?
4. Does the solution only identify vulnerabilities or does it help guide remediation of issues?
5. What is the balance of false positives to positive alerts generated by the solution?
6. How many people are employed to support the solution in environments similar to ours?
December 10, 2015
DevOpsSec: Survival is Not Mandatory

Deming, the patron saint of DevOps once advised, “It is not necessary to change. Survival is not mandatory.”

To survive, application development teams are constantly pressured to deliver software even faster. But fast is not enough. The best organizations realize that security, quality and integrity at velocity are mandatory for survival. Hence, DevOpsSec

My aim here is to leave you uncomfortably amazed. While the pace of development has changed significantly, our processes to secure our applications has not changed enough. It is as if a wildfire has raged uncontained for years now but we have failed to take notice.

Maybe that sounds a bit crazy, but once I share a little background on what is happening, you’ll understand why innovation in this arena is critical for survival (when it comes to your applications). If you are aiming to ramp up your own Rugged DevOps or DevOpsSec practices, this is critical reading material.

Skyrocketing Usage, Plummeting Visibility

Open source component use in development is skyrocketing, and for good reason. Over 17 billion open source and third-party components — across the major development languages — were downloaded last year helping development teams accelerate release timelines and deliver more innovative solutions to market. To better imagine the impact of this download volume, you need to understand that there are an estimated 11 million developers responsible for the billions of downloads.

While the magnitude of usage is amazing it has obscured a vast majority of the risks. Of the billions of downloads recorded last year, 1 in 16 components downloaded had known security vulnerabilities.

While we have seen 30x growth of download requests over the past seven years, we have also witnessed huge growth in the number of organizations improving the speed of consumption and efficiency of their downloads using repository managers.

In the past 18 months since I joined Sonatype, we have seen active instances of repository managers like Nexus, Artifactory, and Archiva grow from 40,000 to over 70,000 installations, with users in the millions — also for good reason. Development teams want to ensure faster, more reliable builds. They also need a private and safe place to house and share their own proprietary components as well as assembled applications, images and other binary outputs. The repository manager has established itself as a parts warehouse for software development.

All is Not Moonlight and Roses

1 in 16 may not sound like a lot until you recognize that many organizations are downloading over 250,000 components every year…some of the largest organizations consume millions of them. If you have not calculated this in your head yet: 250,000 / 16 = 15,625.

Couple this fact with the remarks in the 2015 Verizon Data Breach and Investigations Report stating that applications were the most often exploited attack vector by hackers. As a developer community we are electively sourcing known vulnerable components for use in our applications and those applications are now more vulnerable to attack.

Your Repository Manager Should Serve and Protect

If bad components are getting in, why not just stop them at the front door?

Easier said than done. Current approaches to preventing such behavior are ineffective. For some, “golden repositories” are used to house approved components — but components approved once are rarely vetted again for newly discovered vulnerabilities. For other organizations, OSS Review Boards mandate reviews and approval for all new components — but these organizations are poorly staffed to match the volume and velocity of consumption leading to workarounds by development teams. And while developers themselves would much prefer to use the best quality, highest integrity components when designing an applications, they are never allocated sufficient time to investigate the vulnerability status of every component required for their latest build.

Sparking Innovation

When current approaches fail, inspiration often sparks innovation.

Enter, a new invention: a repository firewall. Think of it as a repository manager with a guard at the front door. Every component that gets downloaded by a proxy repository is automatically evaluated against the parameters development, governance and security teams establish. Does it have an AGPL license, include a known security vulnerability, or is it incredibly outdated? The repository firewall allows organizations to download “good” components while it blocks and quarantines “bad” component downloads. Using a repository firewall can keep your repository manager and development lifecycle safe and secure – in an instant, automatically.

The first repository firewall of its kind is called Nexus Firewall. It couples software supply chain intelligence about component quality, security, and risks with automatic evaluations against personalized policies for approving or rejecting new downloads. Evaluations can also be applied later in the development lifecycle where staging repositories are in use.

Imagine the result: 16 out of 16 downloads, or 250,000 out of 250,000 downloads are now of the best quality. Everything flowing into your application development lifecycle contains the highest quality components. You are instantly compliant with established policies. Your applications are less vulnerable to attacks. You are automatically reducing risk.

Next up in this series, I will share insights about Repository Health Check reports (free to use). Used by over 15,000 organizations, they can offer organizations the first clue as to whether or not your team could benefit from a repository firewall.

December 8, 2015
All Systems Fail*

Vince Lombardi once said, “It’s not whether you get knocked down, it’s whether you get up.” But in today’s world of high velocity development and release practices, it’s also about how fast you can get up.

Today, we are seeing more news about sites from major online retailers and service providers like HP, Target, PayPal and Neiman Marcus going down. High site traffic, poor configurations, scalable architectures, and well planned recovery practices all come into play.

But the best organizations have been failing repeatedly for some time now…on purpose. Organizations like Amazon, Google and Netflix practice regularly for application, system and site outages. In fact, Netflix has baked continuous failures into its operations through an internally developed Simian Army. While it might seem unconventional at first, failing all the time allows them to succeed.

Our CTO here at Sonatype, Josh Corman, often reminds us that “All Systems Fail*”. For those that take this to heart and prepare for the inevitable, they can be better prepared to recover. In fact, for the best prepared, recovery can be so seamless, it is a continuous part of their operations.

December 4, 2015
DevOpsSec: 1 in 16 Chances

The quantitative research summarized below, covering over 7,000 repositories across nearly 100 countries, highlights some of the challenges with quality at modern development velocities, especially important for DevOpsSec practices. By leveraging automation in your repository manager, you can improve application quality and reduce unplanned work while lowering exposure to risk. While this practice supports DevOpsSec initiatives, at its core, it is simply a focus on “building quality in”.

(Written as a collaboration between me and Mike Hansen, Head of Products at Sonatype; Mike wrote the better parts.)

Innovation and Risk

Repository managers like Nexus, Artifactory and Archiva have been serving software components for development teams and their tooling for years now. We can measure over 70,000 active repository manager installations across the globe today. Use of a repository manager improves build performance and reliability while also providing a secure, private location to host open source, third party, and proprietary components needed across continuous delivery and DevOps tool chains.

Industry averages show software development organizations electively downloading over 240,000 components a year from public internet repositories. While these components accelerate development and increase innovation, the vast majority of components go through little to no vetting process to determine if known security vulnerabilities, risky license types, or outdated/unsupported are being used.

Earlier this year, researchers found that 1 in 16 downloads includes a known security vulnerability. That accounts for over 15,000 elective downloads of components with known security vulnerabilities per organization, per year.

Why the Focus on Repository Managers?

Repository managers like Archiva, Artifactory, and Nexus have become a critical piece of the DevOps toolchains and are commonly used across the entire application lifecycle. It is effectively a binary parts warehouse for your applications. Given how integral this warehouse has become, software supply chain intelligence and automation can be used to significantly reduce unplanned work and eliminate avoidable risk.

To understand this opportunity, let’s take a quantitative look at the world’s software development ecosystem.

Global Visibility

Not long ago Sonatype introduced a free feature in its Nexus Repository Manager that is providing new data on the health of software supply chains. This features, called Nexus RHC (Repository Health Check) – offers a simple way to get basic visibility into what OSS is flowing into development via your repository manager. RHC provides a summary of components in the repository that have known security vulnerabilities and risky license types. Usage of this service is now significant, with over 30 million components across over 15,000 Nexus instances being analyzed every day across the globe.

There are also around a million developers using those repositories. With the law of big numbers working for us, we can gain a statistically valid view into the behaviors of the modern software development ecosystem. The findings are quite telling.

The Data

We analyzed over 7,000 repositories containing 500 or more software components using the RHC service during the past 90 days. These repositories are representative of those used by development teams in medium to large organizations. The question was straightforward: To what extent are bad things flowing in, causing downstream rework and creating avoidable risk scenarios?

In three months time, the average number of new vulnerabilities that flowed into the repositories analyzed was 69. That is an average of 23 vulnerabilities per month, which is a little more than one every weekday. If you only include higher risk vulnerabilities – those with a CVSS score of 5 or greater – the average number was 48, or about 16 per month. That is a little less than one every weekday. For a sense of the kind of vulnerabilities in this higher risk category, the Heartbleed bug had a score of 5, the lowest risk in this grouping.

These are not isolated instances or outliers. In the set of repositories analyzed, 98.3% consumed at least 1 vulnerable component and 97.7% consumed at least 1 higher risk (CVSS >= 5) component. In a period of just three months, nearly every repository consumed something the organization running it would rather not have, and that is only with respect to known vulnerabilities. There are plenty of other attributes that are worthy of consideration given availability of the corresponding data.

Whether or not security is of particular concern, the real point is that a lack of visibility and control combined with an abundance of supply has led to inferior quality parts routinely being used by the world’s development organizations, adding avoidable risk and slowing us down.

Going (Way) Beyond Security

The consumption of vulnerable components is a quantifiable problem that we used to illustrate the current challenges that every organization has with open source component selection. However, security vulnerabilities are only one of the many dimensions.

Others include architectural aspects such as rules covering component age and popularity, intellectual property coverage with rules related to open source licensing and technical debt controls associated with technology stack selection. Introducing more formalized vetting processes coupled with software supply chain intelligence and automation will yield significant overall quality and efficiency benefits for an organization, allowing them to operate with a lower risk profile.

Yes, A Firewall for Repository Managers

In November, Sonatype became the first company to introduce a “firewall” for use with repository managers. Called Nexus Firewall, the solution offers perimeter quality control for software development. Similar to a network firewall, it leverages rules you define that automatically shield an organization from unacceptable software components entering and another set for stopping them from exiting your application development. The basic concept looks like this:

In order to build quality in from the beginning, this solution represents a “shift (far) left” innovation. The Nexus Firewall improves speed and reduce risk through the quarantine of components with known vulnerabilities. With a repository firewall in place, organizations can shield your application development from waste and risk by automatically blocking unacceptable software components inbound and preventing release of applications containing such components outbound. 1 in 16 downloads with known security vulnerabilities could be reduced to zero in 16.

The firewall also goes beyond blocking, providing organizations with the visibility and data needed to make ideal decisions for open source component selection early, significantly reducing waste related to rework and eliminating avoidable risk.

For DevOpsSec, Shift (Far) Left to Build Quality In

The consumption of these unacceptable or, at a minimum, inferior components is unnecessary. For the vast majority of these vulnerable components, there are non-vulnerable alternatives ready to be used in their place. The challenge is that it has been historically difficult for developers to understand these risks and even harder to avoid them. Few developers have countless hours to commit to researching vulnerabilities in the components they use and fewer more want to rely on colleagues in security, legal, or audit teams to spend hours or weeks to return with “go” or “no go” vetting decisions.

The volumes and velocity of component consumption we see today validates that any form of manual review process or automated workflows outside of the development teams will be quickly outpaced. DevOps and Continuous Delivery processes need to rely on automatic intelligence to help developers select the highest quality components and avoid those with known risks. When known security vulnerabilities and other risks are identified, software supply chain intelligence will be required to guide developers through the identification and selection of safer, higher quality components.

An example of the kind of information made available is shown below.

The information is divided into two areas. On the left side is component data, which includes details related to the component itself. To the right, there’s a graphical display of any security or license issues, as well as popularity data for each version of the component displayed. Selecting different versions updates this information accordingly, including whether or not a particular version passes the established criteria.

A Must for DevOpsSec: Quality at Velocity

Simply by enabling a repository firewall, organizations can immediately improve quality and reduce waste and exposure to risk. None of this comes at the expense of development speed.

The repository firewall automatically quarantines components that do not pass your rules preventing quality issues from entering the software being developed, immediately reducing risk and avoiding wasteful rework at some later point. Quality is improved, efficiencies are gained and risk is reduced, all through automation and all at the speed of modern development.

One of my favorite reference books is Continuous Delivery by Jez Humble and Dave Farley. Let me share a few of their words to help remind us of the importance of innovations and practices that allow us to bring pain forward. Jez and Dave remark:

“The techniques that we describe in this book, such as continuous integration, comprehensive automated testing, and automated deployment, are designed to catch defects as early in the delivery process as possible (an application of the principle “Bring Pain Forward”). The next step is to fix them A fire alarm is useless if everyone ignores it. Delivery teams must be disciplined about fixing defects are soon as they are found.”

Well said Dave and Jez. Well said.

December 2, 2015
The Software BOM Squad
In my previous post, “When Good Code Goes Bad“, I shared new research showing the average large development organization consumes over 15,000 known vulnerable and defective components annually. While we can’t stop software from going bad, there are practices from traditional manufacturers that we can use to improve our ability to recall and fix the “bad” software components.

The Software BOM

A Bill of Materials (BOM) is used in traditional manufacturing supply chains to list the suppliers and parts used in a product, a “software bill of materials“ (BOM) is an inventory of the third party and open source components used to build an application.

A sample software bill of materials lists all open source and third party components, including their version, age, license type, and any known security vulnerabilities.

As noted in Wikipedia, “The concept of a BOM is well-established in traditional manufacturing as part of supply chain management. A manufacturer uses a BOM to track the parts it uses to create a product. If defects are later found in a specific part, the BOM makes it easy to locate affected products.

“A software BOM is useful both to the development organization (manufacturer) and the buyer (customer) of a software product. Builders often leverage available open source and third-party software components to create a product; a software BOM allows the builder to make sure those components are up to date and to respond quickly to new vulnerabilities. Buyers can use a software BOM to perform vulnerability or license analysis, both of which can be used to evaluate risk in a product. Understanding the supply chain of software, obtaining a software BOM, and using it to analyze known vulnerabilities are crucial in managing risk.”

A software bill of materials not only inventories what is used, but in some cases it also syncs with real-time component defect data to indicate which components have known vulnerabilities or license risks. Various software supply chain automation tools expand the bill of software much further, alerting stakeholders automatically when a defect alert occurs. In advanced tools, the entire software lifecycle is automated to ensure that defective components are avoided and continuous monitoring instantly identifies newly announced vulnerabilities as soon as they are discovered.

Breaking Bad

As mentioned in my previous post, When Good Code Goes Bad, there is no way to prevent software from “going bad”. No one is immune from these issues. Therefore, we need to focus on our responses to these issues.

To paraphrase Vince Lombardi, “It’s not how you get knocked down, it’s how you get up”. Following the OpenSSL vulnerability announcement in 2014 (or pick any other new security vulnerability with or without an assigned logo), every development and IT operations team on the planet was asking two questions:
1. Did we ever use that open source component?
2. if yes, where is it?
For those organizations tracking their use of components with a software bill of materials, the answer was easy to find. For other organizations, weeks and often months of research were required. To get a sense of the lack of maturity around traceability practices, take a look at this image from Sonatype research showing announcements of OpenSSL vulnerabilities in products following the initial vulnerability disclosure in April 2014.

Automating Traceability Across the Software Supply Chain

Software supply chain complexities, coupled with the volume and velocity of open source and third-party components flowing through them, are forcing us to think differently about our current practices across development and operations. We are not the first to face these challenges and we can learn a great deal from the mature practices of other high-velocity manufacturing organizations and industries.

One thing is certainly clear. Manual practices of monitoring or occasional checkpoints cannot keep pace with volume and velocity of components being consumed. And if you extend these considerations beyond just open source components and into containers, microservices, or other images used in development and operations, the problem is further out of reach for manual processes to keep pace. The volume and velocity within our software supply chains will not diminish—and without a new approach, the volume of unchecked quality and integrity of parts being consumed will continue to build up as technical debt.

But the answer is here today: automation. Automation can unleash the potential of an organization’s development capacity. Rarely is there such an opportunity to simultaneously increase speed, efficiency, and quality. Solutions that facilitate comprehensive software supply chain automation are poised to usher in the next wave in development productivity—with gains on par or even greater than possible with agile, Lean, and DevOps.

More information on the current state of software supply chain practices as well as examples of best practices being employed by software development teams, can be found in the 2015 State of the Software Supply Chain Report.
July 22, 2015
The Cost to DevOps: 27 Mufflers

Imagine

Imagine that you are designing the 2016 Range Rover line of sport utility vehicles. Like all gas powered vehicles, each one needs an exhaust muffler. Range Rover likely has narrowed in on a preferred provider of mufflers. But imagine what would happen if the designers and factory line workers could pick from any one of 27 past versions of that muffler from their preferred provider for the new model year. Yup, choose any one — even if it is outdated, lower performance, does not meet current emission standards, or has a known defect. Order it, place it on the vehicle, then ship it.

We all realize this scenario would never happen. No one would select a known defective part for the product they are manufacturing. And who would knowingly choose the 10 year old version of a muffler without the latest advancements to support air quality? No organization would give a each of their designers and line workers free reign on procuring any one of 27 different mufflers for a single model year. Imagine the operational cost of testing the different mufflers. Imagine the maintenance cost of having different versions of the same part shipping. Worse yet, imagine the liability of shipping parts with known defects.

Land Rover is not taking this approach, but software development teams are.

27 Versions

My latest research report, the 2015 State of the Software Supply Chain, reveals that large software development organizations sourced an average of 27 different versions of open source components that they used in development last year. To be more specific, out of the top 100 open source components these companies downloaded from the internet in 2014, they averaged 27 versions of each component.

While auto manufacturers like Land Rover have preferred suppliers of vetted parts, the procurement of open source components is often more of a free-for-all. If an organization has 300 developers, they effectively have a procurement department of 300 for sourcing externally developed software components. If you have 4,000 developers, you have 4,000 performing procurement. Even as we all know only the latest version of a components is the most feature rich and of the highest quality and integrity.

Not all organizations allow free-for-all sourcing practices for components. Fifty-seven percent (57%) have open source policies in place as a method to guide developers toward higher quality parts with the fewest known defects or security vulnerabilities. Organizations like Google mandate the use of no more than two versions of a given open source component across their development teams.

Good Cost, Bad Cost

What does this mean for a software development organization — especially one focused on improving it’s DevOps practices? First the good news: developers are using open source components to speed time to release and improve innovation. Both good things.

The bad news: Organizations are building mountains of technical debt through poor software supply chain practices by sourcing an average of 27 versions of the same component, impacting quality, complexity, maintainability, supportability, etc. They are doing this across the top 100 components being downloaded in a single year which equates to managing over 2700 different versions (when an optimized practice would only require the 100 best).

Organizations are sourcing component “parts” for use in their applications that have up to 26 known better versions available (more features, fewer bugs, improved performance, etc.).

Organizations are downloading and using parts that are sometimes 9 years old, or even older (simple math: open source projects release 3 – 4 new versions of their components each year). They are electively using outdated parts.

Organizations are also downloading some versions with known, documented security vulnerabilities. They are electively using known vulnerable parts.

Net Innovation Through Improved Sourcing

While we often hear people tout a DevOps mantra of “even faster”, there is a consequence to pursuing speed at any cost: technical and security debt.

Open source components are enablers of speed. But free-for-all sourcing and use practices, add to technical and security debt that reduces “net innovation” and adds to the cost of operations. As technical debt grows, net innovation and business value drops.

Image source: http://joapen.com/blog/2014/06/17/how-to-monetize-application-technical-debt/

If reducing the use 2,700 versions of 100 components was not motivating enough, consider this: the research also revealed that large development teams were consuming software components from over 7,600 external open source projects (suppliers). Including all versions of those components, they average organization consumed nearly 19,000 unique components in 2014.

Improved sourcing practices are just part of the overall story for improving net innovation and increasing business value. Improved sourcing practices can also enable more efficient operations, reduce mean times to remediation, and eliminate unplanned work for many DevOps practices. Improved sourcing practices have been employed across many industries resulting in sustained competitive advantages. Improved sourcing practices are part of a broader software supply chain management strategy.

Continuous Learning

There are so many lessons we can learn from other industries who have automated and optimized their supply chains — like Land Rover and other auto manufacturers. I’ll be sharing more of those lessons and perspectives from the 2015 State of the Software Supply Chain here this summer as well as expanding the discussion around “net innovation”. Stay tuned.

July 20, 2015
Bring Your Own Exploit

Here’s a simple question: for each tool that your organization uses, have you (or someone else on your DevOps team) changed the default passwords? That should be a no-brainer, shouldn’t it? Of course you changed them! Or at least you changed them for the tools that looked like they would have default passwords. Well, OK, at least the ones that looked like they might present an obvious security risk — and where you noticed that there was a default password. Would those other tools even provide an intruder with a useable exploit?

(more…)

July 14, 2015