devopsy.kinsta.cloud

Tag: Software Supply Chain

Rework is Choking Software

Rework is Hell

“Software may be eating the world, but rework is choking software”, tweeted John Jeremiah (@j_jeremiah). To shed more light on what is choking software, new data was released last week in the 2015 State of the Software Supply Chain Report.

In its discussion of application quality and integrity, the report revealed that the average application includes 106 open source components. It is clear that the use of these components has benefited development tremendously in helping to speed time to market and improve innovation. While the benefits are undeniable, development teams are also delivering applications that are “insecure by design”. Of the 106 components per application, the report’s analysis revealed an average of 24 (i.e., 23%) have known critical or severe security vulnerabilities. Those same apps also showed an average of 9 restrictive license types (e.g., GPL, AGPL, LGPL).

By electively sourcing components with known vulnerabilities and potential license risks, software development teams are building up higher levels of technical and security debt. And in order to reduce some of that debt, unplanned and unscheduled rework is often required.

No Developer Intentionally Uses Bad Parts

I have never met a developer that intentionally wanted to use “bad” components in their applications. That said, I have also never met a developer that wanted to spend four hours researching the latest versions, open source licenses and known security vulnerabilities for components (including dependencies) they wanted to use in their applications. While 57% of development organizations have internal policies in place that direct “thou shall not use components with poor quality and integrity”, many have stated that the policies are not followed or are simply not enforced.

The problem is one of volume and velocity, while people clutch to old processes that simply can’t keep pace. The analysis in the 2015 State of the Software Supply Chain report revealed that billions of open source downloads occurred in 2014, of which 6.2% had known vulnerabilities (1 in 16 downloads). For some large companies, the volume of downloads exceeded 240,000 — where 15,000 (7.5%) had known security vulnerabilities. At these volumes, manual processes and paper-based policies to prevent poor quality or risky components from getting into your software have no possibility of keeping pace. While software development practices aim for higher velocity, through component-based development as well as Agile, continuous and DevOps practices, the processes to keep up with it have not changed enough.

The only way to keep pace with this velocity is through automation. And if we want to avoid unplanned rework associated with remediating known security vulnerabilities or selecting alternative components with approved license types, automation has to become the norm rather than the exception within development.

At Speed

“If I look at the mass, I will never act. If I look at the one, I will.” — Mother Teresa

Looking at the massive volume and velocity of open source and other artifact consumption can be discouraging and overwhelming. However, the volume and velocity within our software supply chains will not diminish—and without a new approach, the volume of unchecked quality and integrity of parts being consumed will continue to build up as technical debt.

Automation in areas of testing, build, and deployment has provided significant performance benefits. Likewise, investments in software supply chain automation have shown markedly improved efficiency and controlled risk, as the best practices in the 2015 Report illustrate. Automation can unleash the potential of an organization’s development capacity. Rarely is there such an opportunity to simultaneously increase speed, efficiency, and quality.

In order to improve the quality and integrity of components used in our applications, we need to release our clutch on old practices. We can further embrace automation across our software supply chains — and improve the quality of our applications — by considering these three steps:

1. Start with creating a software bill of materials for one application. Visibility into one application can help you better understand your current component usage. A number of free and paid services are available to help you create a software bill of materials within a few minutes. The bill of materials will help you to identify the unique component parts used within your application and the suppliers who contributed them. These reports list all components used, and several services also identify component age, popularity, version numbers, licenses, and known vulnerabilities.

2. Design approval processes to be frictionless, scalable, and automated. Manual reviews of components and suppliers cannot keep pace with the current volume and velocity of consumption. Your organization must not only define your policies for supplier and part selection but also find practical ways to enforce them without slowing down development or inadvertently encouraging workarounds. Policies must be agile enough to keep pace with modern development. Strive to automate policy enforcement and minimize drag on developers.

3. Enable developer decision support. Developers are primary consumers of components in software supply chains. They initiate every component request. Help developers by automating the availability of information on component versions, age, popularity, licenses, and known vulnerabilities within their existing development tools so it is easy to pick the best components from the start. By selecting the highest-quality components from the highest-quality suppliers early, you will improve developer productivity and reduce costs.

Here is the complete 2015 State of the Software Supply Chain Report. I will also be discussing more of the Report findings and best practices on a webinar scheduled for Wednesday, June 24, 12pm ET (register or view it on-demand).

The previous blog in this series was, Fewer and Better Suppliers. Next up, I will be reviewing Visibility and Traceability practices across the software supply chain with more commentary from the 2015 State of the Software Supply Chain Report.

June 24, 2015
2015 State of the Software Supply Chain Report
In April of this year, I embarked on a six-week journey diving deep into an analysis of the world’s software supply chains. I evaluated the practices of 106,000 organizations, the 100,000+ suppliers they relied on, and the billions of software components that fueled their agile, continuous delivery and DevOps practices.

The facts I discovered and share in the 2015 State of the Software Supply Chain Report: Hidden Speed Bumps on the Road to Continuous, (pre-register for the full release: Tuesday, June 16th) fundamentally changed the way I thought about software (and about DevOps).

The volume and velocity of consumption, the variety of parts and suppliers, and the impact on innovation and quality astounded me. Early reviewers of the report including Gene Kim (co-author of the Phoenix project), Gareth Rushgrove (Puppet Labs and DevOps Weekly newsletter), Nick Galbreath (Signal Sciences), and Nigel Simpson (Fortune 100 Entertainment and Media company) agreed.

My aim for this research is not simply to present facts about the global software ecosystem. I’m aiming to point a spotlight on software supply chain best practices within across a variety of industries that could be used as new benchmarks for software supply chain automation. Similar to manufacturing of auto, pharmaceutical, healthcare, or defense systems, the effective management of supply chains will create winners and losers. I’ll share evidence that the best, high-performance software development organizations are benefiting from:
- Working with fewer and better suppliers
- Relying on the highest quality supplies from those suppliers
- Maintaining traceability and visibility throughout the software supply chain for prompt and agile recall
Key Points from the Study

In the best organizations, the research revealed developer net productivity increasing by up to 40%. Just imagine applying that time to more innovation, rather than to rework and maintenance efforts.

At the same time, the report touches on inefficiencies and complexities that are creating a huge drag on the velocity software development teams are aiming to achieve. A lack of discipline, focus, and visibility around the software supply chain has resulted in mountains of technical debt, unnecessarily context switching, and outdated sourcing methods that wasted over 3.3 million build days last year alone.

Hightlight: Automation Across the Software Supply Chain

The other key insight from this research is the clear need for further automation across software supply chains. With individual organizations consuming hundreds of thousands and sometimes millions of software components annually, it became obvious that waterfall-centric approaches to identifying the most functional parts, checking quality, validating appropriate licenses, or evaluating security vulnerabilities could not keep pace. Sourcing practices that regularly go unchecked have also resulted in the use of severely outdated software components, even numerous versions of the same component part.

Download the Full Report

Over the next few weeks, I will publish excerpts of findings and best practices identified in the 2015 State of the Software Supply Chain Report and invite you to read along. Or you can access the full report now through this link.
June 12, 2015
DevOps Leadership Series 2015
We recently hosted the 2nd annual “DevOps: Wine-ing, Not Whining” event in San Francisco to bring together a number of thought leaders and leading practitioners in the DevOps field. We traded stories from the trenches, formed new insights, and offered visions of the future.

This week, I’ll be sharing videos captured from the event that highlight key themes from our conversations. We call it the DevOps Leadership Series.

You’ll get to hear:
- Jez Humble (Chef), Julie Tsai (Retail Giant) and Nick Galbreath (Signal Sciences) discuss on importance of software supply chains
- Chris Corriere (Autotrader) and Mitchell Ashley (CableLabs) will share perspectives the state of DevOps and security
In the next few weeks, I’ll be on the west coast at the Velocity Conference and on the east coast covering DevOps Days DC. I’ll be capturing more insight from DevOps thought leaders at these events. If you are planning to attend one of these events, let’s capture your DevOps perspective for the series. To participate, please leave a comment below or catch me on @weekstweets.

Today, we’ll start the series with Gene Kim on important themes for DevOps in 2015:

[youtube http://www.youtube.com/watch?v=0IXrnhcRNUk]

PART 2 of the series is now available at:

https://devopsy.kinsta.cloud/2015/05/19/devops-leadership-series-software-supply-chains/
May 18, 2015
My picks for opening day of DevOpsDays Austin

As we wrote earlier, we are happy to be attending and sponsoring DevOpsDays Austin 2015. The opening day morning is chocked full of great sessions with the afternoon dedicated to openspaces as is the norm. While everything from breakfast to Damon Edwards keynote through JJ Ashgar’s session on Chef before lunch look tasty. Here are three sessions that I will be attending that I am really looking forward to:

The Coming Donkey Apocalypse by Michael Cote’ – Cote’ formally of 451 Research and now with Pivotal is going to talk about the “state of the union” of Devops. From the abstract:

DevOps is growing in popularity and even usage in “the real world.” It seems like we’re slowly getting past unicorns only and seeing many “horses” do DevOps. Soon we’ll see the mainstream market – the “donkeys” – start to pick it up and bend DevOps to its will. This talk will go over “the state of the union” of DevOps and provide some guidance for how to prepare for the donkey apocalypse.

I don’t know what to make of this title but it peaked my curiosity. I don’t think DevOps adoption moving mainstream is equal to moving from horses to donkeys and so it is not a very flattering title. On top of this not only don’t I think main stream adoption of DevOps means it is Donkeys adopting it, I do not think it means an apocalypse for DevOps. In fact I think it is a good thing. On top of this, I think the idea of calling mainstream adoption, adoption of the donkeys and them causing an apocalypse smacks of elitism and is against what DevOps is about. I don’t think it is a bad thing that more organizations will get involved and that they will “bend DevOps”. I don’t know Michael well and have only spoken to him once briefly. Everything I hear though is that he is a very bright guy and dynamic speaker and writer. So i am interested to hear what Michael has to say in this session and whether he thinks it is a good thing, bad thing or how we can help move it along. You can’t judge a book by its cover and you sometimes you can’t judge a session by its title and abstract.

Fixing your Organization by Continually Breaking It by J Paul Reed – Reed is a consultant and has over a decade of experience in the trenches as a build/release and tools engineer, working with such organizations as VMware, Mozilla, and Symantec. In 2012, he founded Release Engineering Approaches, a consultancy incorporating a host of tools and techniques to help organizations “Simply Ship. Every time.” From the abstract:

When we discuss DevOps, we often find ourselves mired in the “tools vs. culture” debate. A point often forgotten is the “amplify feedback loops” of DevOps “3 Ways” and how to build that into our cultures. In this talk, we’ll look at the importance of continually breaking your organization with experiments, some surprising examples, and how to do it in such a way that you aren’t left with a broken organization.

I love this concept going back to Netflix and the Simian Army and have heard similar talks before on it. Looking forward to what Reed has to say and learning. Amplifying feedback loops is something I would like to see more written about as well. I am looking forward to this one!

Continuous Acceleration: Why Continuous Everything Needs a Supply Chain Approach by Josh Corman – Josh is of course the CTO of Sonatype and a former 451 Research analyst too. Besides all that Josh is a friend from the security world that I know a long time. This talk sounds like a continuation of what Josh and Gene Kim presented at DevOps Connect at RSA Conference a few weeks ago. Also Ericka Chickowski wrote a good article on her interview with Josh on this last week. Here is the abstract:

Today we write less code & consume more re-usable open source code. Innovation accelerates & complexity grows. Complexity is the enemy of quality. Poor quality creates break-fixes & slows dev. It’s a continuous loop.
Learn about software supply chain management strategies to accelerate delivery: scrutinize your ‘suppliers,’ minimize risk & bloat, and improve traceability, visibility & response times.

Josh is a great presenter. I have seen him present many topics over the years at several events, I am sure this one won’t disappoint.

There are other good presentations including some stuff on Dockers and containers (what would a conference be without them?). Additionally the I am sure the open spaces will spawn some good stuff as well.

I will try to update throughout the day and give those who can’t attend a sense of what is going on.

I realize many of you can’t make Austin tomorrow, but if you are going and see me please introduce yourself and say hi.

May 4, 2015
Security, DevOps and the shift to a software supply chain

Last week at the DevOps Connect event held in conjunction with RSA Conference, the security and continuous delivery worlds united to deliver an overarching message that security teams that can embed themselves in DevOps patterns can not only secure IT at speed, but they can actually contribute to a more reliable, repeatable and higher quality release cycle.

“DevOps extended the value of Agile so that we went from just continuous integration to continuous integration and continuous deployment,” says Josh Corman, CTO of Sonatype. “If the three measures of any software manufacturing lifecycle is speed, efficiency and quality/risk, then I think this Rugged DevOps era ideally allows you to go even faster, be even more efficient because of fewer break fixes and faster mean time to repair and better quality and lower risk.”

To read more about security and DevOps, check out our eBook on Rugged DevOps.

Corman and Gene Kim, author of The Phoenix Project, developed on this idea in their talk, which pushed the thought that the ultimate Zen state of software delivery to strive for is a software supply chain.

“The way we can get more safety and security into digital infrastructure, is to move to the ultimate evolved posture of software development as a supply chain,” Corman explains. “This makes you even faster than DevOps–even more efficient and with higher quality and risk mitigation without tradeoffs.”

The idea of the software supply chain further builds on the lean manufacturing principles of W. Edwards Deming, who many in the Agile and DevOps worlds see as the spiritual grandfather of these movements.

“If you take Deming to his logical consequence, we’re going to start paying attention to the quality of our suppliers, the quality of our supply and the traceability and visibility of which parts went were, such that when there’s another Heartbleed, we can do a prompt and Agile response,” he says, explaining that that avoiding known bad parts, such as a bad version of struts in car manufacturing or a faulty SSL component like the one that caused Heartbleed, “will make you keep top speed, halve your blowouts, faster meantime repair, better traceability and make the auditors go away with less pain”

Security teams have traditionally been known as roadblocks rather than speed enhancers to development processes. But looking at it from the perspective of creating an effective software security supply chain, security actually plays a role in further speeding up production if it is helping organizations depend on higher quality components of production code and the code running infrastructure.

“If we get better at our component selection and our traceability of what we’re using where, we can reduce the number of break-fixes or reduce the number of unplanned unscheduled work,” Corman explains

He emphasizes that security debt tracks roughly with technical debt and CISOs should be having conversations with DevOps teams to that end. In the end security work pays down principle on technical debt, making it possible to spend more time writing code and less time on rework

“I’d love to give us a more defensible opportunity which embraces what devops already cares about and doesn’t articulate how we can make them safer, instead articulate how we can make it faster and more efficient,” he says.

As security pros start to do so, they might be surprised at their reception by the developers and operations folks. As Kim explained in their talk, in the end everyone does want the kind of quality that a cooperative security team can bring to the DevOps equation.

“When security imposes itself on day-to-day DevOps processes, dev and ops say ‘thank you,’” he says.

May 1, 2015
A True Story: DevOps(Sec) Manages Out Elective Risks
A True Story

There are over 2000 developers in Bill’s organization. He is a C-level executive for one of the largest insurance companies in North America. Bill boosted developer productivity by 15% last year after taking a closer look at the company’s software supply chain. And this approach isn’t unique to Bill’s organization. Many high performance IT and DevOps teams are adopting proven supply chain principles to accelerate software delivery.

Origins, Quality and Integrity

The concept was rather simple and straightforward. Bill’s team recognized that they were consuming hundreds of thousands of open source and third-party software components to build their applications. But at the same time, they did not apply much scrutiny to the origin, quality, age, or integrity of those components. Developers simply selected components at-will that met their functional requirements and helped them meet the next delivery deadline.

Secrets of the Free-For-All

Deeper analysis of their practice revealed that the free-for-all component sourcing methods were leading to quality, efficiency, and productivity issues. Here is what Bill found:
- The company had downloaded an average of 27 versions of over 100 different binary artifacts over a one year period. Bill knew this meant the quality of their applications was being impacted through the use of outdated software components.
- The company had also downloaded 13,203 software components with known security vulnerabilities — and 65% of those included alerts earlier than 2014. Without quality controls in place, Bill recognized they were sourcing in elective and avoidable risks that would impact the integrity of their application, grow their technical debt, and lead to more rework down the road to replace the flawed components.
Removing Complexity

Bill decided to take action. He knew that to accelerate innovation, he needed to remove complexity from the company’s software supply chain. He enacted three moves to eliminate waste and complexity within the open source and third-party components being used:
1. Automate the scrutiny of software suppliers to minimize risk and bloat.
2. Enable real-time traceability and visibility to components used across the development lifecycle, in order to improve response times when defects were discovered – cutting back from days/weeks to minutes!
3. Reducing complexity, rework and risk by standardizing on specific components types (e.g., web frameworks, logging frameworks, encryption modules), using fewer and more current artifact versions, and managing out elective security risks.
Want to hear the rest of the story?

The same principles that Bill applied to his business that boosted developer productivity by 15% are being discussed by Gene Kim (@RealGeneKim), author of The Phoenix Project, and Joshua Corman (@JoshCorman), CTO of Sonatype, on April 30th. Both Gene and Josh are huge fans of marrying DevOps practices with leading supply chain management principles that enable developers to maximize throughput of features from “code complete” to ‘in production,” without causing chaos.

If you’re looking to achieve similar goals, be sure to join Josh and Gene for this April 30th, 1pm ET, discussion.
April 23, 2015