devopsy.kinsta.cloud

Author: Derek E. Weeks

2015 State of the Software Supply Chain Report
In April of this year, I embarked on a six-week journey diving deep into an analysis of the world’s software supply chains. I evaluated the practices of 106,000 organizations, the 100,000+ suppliers they relied on, and the billions of software components that fueled their agile, continuous delivery and DevOps practices.

The facts I discovered and share in the 2015 State of the Software Supply Chain Report: Hidden Speed Bumps on the Road to Continuous, (pre-register for the full release: Tuesday, June 16th) fundamentally changed the way I thought about software (and about DevOps).

The volume and velocity of consumption, the variety of parts and suppliers, and the impact on innovation and quality astounded me. Early reviewers of the report including Gene Kim (co-author of the Phoenix project), Gareth Rushgrove (Puppet Labs and DevOps Weekly newsletter), Nick Galbreath (Signal Sciences), and Nigel Simpson (Fortune 100 Entertainment and Media company) agreed.

My aim for this research is not simply to present facts about the global software ecosystem. I’m aiming to point a spotlight on software supply chain best practices within across a variety of industries that could be used as new benchmarks for software supply chain automation. Similar to manufacturing of auto, pharmaceutical, healthcare, or defense systems, the effective management of supply chains will create winners and losers. I’ll share evidence that the best, high-performance software development organizations are benefiting from:
- Working with fewer and better suppliers
- Relying on the highest quality supplies from those suppliers
- Maintaining traceability and visibility throughout the software supply chain for prompt and agile recall
Key Points from the Study

In the best organizations, the research revealed developer net productivity increasing by up to 40%. Just imagine applying that time to more innovation, rather than to rework and maintenance efforts.

At the same time, the report touches on inefficiencies and complexities that are creating a huge drag on the velocity software development teams are aiming to achieve. A lack of discipline, focus, and visibility around the software supply chain has resulted in mountains of technical debt, unnecessarily context switching, and outdated sourcing methods that wasted over 3.3 million build days last year alone.

Hightlight: Automation Across the Software Supply Chain

The other key insight from this research is the clear need for further automation across software supply chains. With individual organizations consuming hundreds of thousands and sometimes millions of software components annually, it became obvious that waterfall-centric approaches to identifying the most functional parts, checking quality, validating appropriate licenses, or evaluating security vulnerabilities could not keep pace. Sourcing practices that regularly go unchecked have also resulted in the use of severely outdated software components, even numerous versions of the same component part.

Download the Full Report

Over the next few weeks, I will publish excerpts of findings and best practices identified in the 2015 State of the Software Supply Chain Report and invite you to read along. Or you can access the full report now through this link.
June 12, 2015
DevOps Leadership Series: Monitoring Containers and Microservices

Trevor Parsons (@trevparsons) is a Co-Founder and Chief Scientist at Logentries, a leading SaaS-delivered log management and analytics service. I caught up with Trevor at the Velocity Conference in Santa Clara and asked him what themes were resonating with attendees this year.

For this episode of the DevOps Leadership Series, Trevor briefly illustrates some current problems with state of microservices and container-based architectures, while sharing a positive outlook for the future.

https://www.youtube.com/watch?v=zHRE4011WVk

Trevor says there’s been a lot of development in how to log and monitor containerized systems and he sees better support from Docker for logging, showing an increasing focus on this topic in 2015 and beyond. He anticipates the next year to have huge improvements in managing and monitoring these new systems and architectures.

Next up in the DevOps Leadership Series, I headed to London for the DevOps Connect: Rugged DevOps event, where we’ll discuss more about Rugged DevOps practices. In London, I was able to catch up with Helen Beal (Ranger4), Justin Arbuckle (Chef), and Gareth Rushgrove (Puppet Labs).

If you have missed any of the other videos from this series, you can find them here. (We’re up to 15 so far).

June 11, 2015
DevOps Leadership Series: Compliance, Testing, and Rugged

This past week, I headed to London for the Rugged DevOps event, where I had the chance to catch up with a few more industry thought leaders.

First, I caught up with Gareth Rushgrove from Puppet Labs, who also runs the DevOps Weekly newsletter. In this episode of the DevOps Leadership Series, Gareth explains why the importance of security tests within DevOps practices is going to be a big topic for years to come. Gareth anticipates that “In five years time we will look back on this and ask ‘why were we not always doing this’?”

https://www.youtube.com/watch?v=ktpkmGv5gX4

I then caught up with Helen Beal, Head of DevOps at Ranger4, where she discussed the importance of DevOps and security. While the two ultimately need to work hand and hand, she voiced concerns about DevOps practices sometimes circumventing controls that are essential to a business’s safety. She also said that DevOps supports security in a number of ways: from making things consistent to relying on more automation.

https://www.youtube.com/watch?v=W4Zs1SSjGFs

Finally, I caught up with Justin Arbuckle, Vice President, EMEA & Chief Enterprise Architect at Chef. He discusses key trends he expects to see in over the next 9-12 months regarding compliance and security. Justin explains that compliance as a core driver of DevOps is something that will start to shape our conversations over the next year, whereas the two have previously been seen as contradictory. He believes that we will see that high velocity organizations are able to improve compliance continuously, and tells us to look out for the changing role of the security officer in 2015.

https://www.youtube.com/watch?v=ojEOqRJey2o

Next up in the series, I head to the United States Capitol, Washington DC, for DevOps Days DC.

NOTE: If you have missed any of the other videos from this series, you can find them here. (We’re up to 15 so far).

June 10, 2015
DevOps Leadership Series: Security at Velocity

If it does not fit, it does not get done. For many DevOps practices, application security falls into the “does not get done” bucket. That’s because for many DevOps-centric organizations, application security has historically been done somewhere else, by someone else, who is slow.

Go faster. Shift left. Remove complexity. Reduce rework. All mantras of DevOps practices. And while DevOps practices have changed dramatically in recent years, many experts will tell you that application security has not changed enough.

In this installment of the DevOps Leadership Series, you will hear Chris Corriere (DevOps Engineer, Autotrader) and Mitchell Ashley (VP Information Technology, CableLabs) share perspectives on the state of DevOps and security. It is about security at velocity.

But is going faster and automating resulting in better security? There are still security pros who are uncomfortable with that premise. Nevertheless, that is where this is headed. The security community has to come to the mountain, otherwise the mountain is coming to them.

First, listen to Chris’ Corriere of Autotrader tell us his perspective that security can move at DevOps speed, as long as you take a diversified approach:

https://youtu.be/8DdfHkAK1Fk

Then hear from Mitchell Ashley, VP of IT at CableLabs as he remarks on the evolution of security from a validation role (at the right) to an integrated, ingrained role that has shifted left:

https://www.youtube.com/watch?v=pg_m7dYqUPU

This is the third in our series of interviews from this event. Stay tuned as the actual sessions from DevOps Connect will be posted shortly.

NOTE: If you have missed any of the other videos from this series, you can find them here. (We’re up to 15 so far).

May 21, 2015
DevOps Leadership Series: Software Supply Chains
We kicked off this series on Monday with Gene Kim (@RealGeneKim) sharing his views on the big theme for DevOps in 2015: proving that DevOps is applicable for large organizations.

Another theme that arose often during our recent “DevOps: Wine-ing, Not Whining” event was the importance of software supply chains. Every software development organization has a software supply chain, and DevOps leaders are now applying principles from lean manufacturing, Deming, and Toyota supply chain management to improve their operations.

They are applying software supply chain management principles help:
- choose higher quality parts (e.g., software binaries, containers, microservices)
- select higher quality suppliers (e.g., fewer defects, faster MTTR records), and
- improve traceability (e.g.., identifying defects or known vulnerabilities to expedite break/fix)
Today, we will hear from Jez Humble (Chef), Julie Tsai (Retail Giant) and Nick Galbreath (Signal Sciences) as they discuss on importance of software supply chains and DevOps. Then after watching the videos, you can learn more about the sea change in software supply chain automation in these two blogs from Mike Hansen (Software Supply Chain Automation) and Ericka Chickowski (Security, DevOps and the shift to a software supply chain).

Here are today’s short videos from our 2015 DevOps Leadership Series:

[youtube http://www.youtube.com/watch?v=4L4nrNpdTM4]

[youtube http://www.youtube.com/watch?v=UEkVYJ1T2NY]
[youtube http://www.youtube.com/watch?v=nTv8Yx0CpfY]

NOTE: If you have missed any of the other videos from this series, you can find them here. (We’re up to 15 so far).
May 19, 2015
DevOps Leadership Series 2015
We recently hosted the 2nd annual “DevOps: Wine-ing, Not Whining” event in San Francisco to bring together a number of thought leaders and leading practitioners in the DevOps field. We traded stories from the trenches, formed new insights, and offered visions of the future.

This week, I’ll be sharing videos captured from the event that highlight key themes from our conversations. We call it the DevOps Leadership Series.

You’ll get to hear:
- Jez Humble (Chef), Julie Tsai (Retail Giant) and Nick Galbreath (Signal Sciences) discuss on importance of software supply chains
- Chris Corriere (Autotrader) and Mitchell Ashley (CableLabs) will share perspectives the state of DevOps and security
In the next few weeks, I’ll be on the west coast at the Velocity Conference and on the east coast covering DevOps Days DC. I’ll be capturing more insight from DevOps thought leaders at these events. If you are planning to attend one of these events, let’s capture your DevOps perspective for the series. To participate, please leave a comment below or catch me on @weekstweets.

Today, we’ll start the series with Gene Kim on important themes for DevOps in 2015:

[youtube http://www.youtube.com/watch?v=0IXrnhcRNUk]

PART 2 of the series is now available at:

https://devopsy.kinsta.cloud/2015/05/19/devops-leadership-series-software-supply-chains/
May 18, 2015
Chevy and DevOps: What the Wi-Fi?
In DevOps: Rapid Feedback, Rapid Repair

I’m sure you have seen it too. Chevy cars now come with wi-fi. How cool. I want that (and so would my kids). I can only imagine the possibilities.

But, this is not all about my needs. Chevy and every other vehicle maker wants this too. And not for the reasons that you might first consider. Quickly, let me introduce you to the recalls of today:
The glitches were related to cars losing power, erroneously deploying airbags, and other defects. This is just a small glimpse of the software recalls that have happened over the past year.

But more troubling is this form of software “recall” from BMW showing a security flaw that could allow cyber criminals to unlock doors on 2.1M vehicles. Because every one of the 2.1M vehicles has the same defective software part, criminals effectively had access to a skeleton key to every one of them.

In DevOps: Rule Your Supply Chain

The reality is, that your new car has more software in it than you can image — about 100 million lines of code. And, as we know, all software has flaws. But who wants to go to a dealer to get a software update? If my car can connect to the internet and update itself, that would save me an unscheduled trip to the dealer for a repair.

While software updates to cars models in the past were would take 5 years to feed themselves onto the production line, that is now all changing. For decades, large automakers like Toyota have been applying excellent management process for their entire supply chain. And with the introduction of wi-fi in vehicles, they are expanding their highly-tuned practices to include their software supply chains. They know where every piece and version of their software is and who owns it. If a flaw is detected (be it quality, safety, security, or another attribute), they will now be able to automatically update that vehicle. Not only can they remove defects, automakers like Telsa are even making their cars faster.

While many of us outside the auto industry think of these large industrialists as slow moving organizations, the introduction of wi-fi in cars is allowing them to now apply a core principle from the agile manifesto: “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”.

In DevOps: Version Everything

For those of you in the software industry, this kind of evolution is at the heart of many DevOps and continuous delivery efforts. Automate everything you can. Version everything you can. Deliver fast, improve visibility, and change fast. In many ways, the addition of 4G LTE Wi-Fi in Chevy Trucks means that DevOps has entered into our auto and software supply chains.

If you imagine an even broader perspective of this wi-fi feature and software supply chain management, the auto makers are now leading the way to expectations of quality improvements in the Internet of Things (IoT). There is not much difference between an internet enabled truck, thermostat or fitness bracelet when it comes to rapidly improving and updating quality, features, performance, etc. (I’ll cover this topic more in future posts.)

In DevOps: Optimize Your Software Supply Chain

What have the auto makers learned from the software industry? And perhaps more importantly — when it comes to software supply chain management and continuous delivery — what can we learn from them?

A final note on software and auto safety: I am the Cavalry (@iamthecavalry) is encouraging the automotive industry to commit to cyber safety. The movement recently published a Five Star Automotive Cyber Safety Framework (PDF download) and calls for Automotive Industry adoption. Media outlets from across the US and Europe praised the effort and joined in calling for automakers and security researchers to work together to ensure a safe future.
April 28, 2015
A True Story: DevOps(Sec) Manages Out Elective Risks
A True Story

There are over 2000 developers in Bill’s organization. He is a C-level executive for one of the largest insurance companies in North America. Bill boosted developer productivity by 15% last year after taking a closer look at the company’s software supply chain. And this approach isn’t unique to Bill’s organization. Many high performance IT and DevOps teams are adopting proven supply chain principles to accelerate software delivery.

Origins, Quality and Integrity

The concept was rather simple and straightforward. Bill’s team recognized that they were consuming hundreds of thousands of open source and third-party software components to build their applications. But at the same time, they did not apply much scrutiny to the origin, quality, age, or integrity of those components. Developers simply selected components at-will that met their functional requirements and helped them meet the next delivery deadline.

Secrets of the Free-For-All

Deeper analysis of their practice revealed that the free-for-all component sourcing methods were leading to quality, efficiency, and productivity issues. Here is what Bill found:
- The company had downloaded an average of 27 versions of over 100 different binary artifacts over a one year period. Bill knew this meant the quality of their applications was being impacted through the use of outdated software components.
- The company had also downloaded 13,203 software components with known security vulnerabilities — and 65% of those included alerts earlier than 2014. Without quality controls in place, Bill recognized they were sourcing in elective and avoidable risks that would impact the integrity of their application, grow their technical debt, and lead to more rework down the road to replace the flawed components.
Removing Complexity

Bill decided to take action. He knew that to accelerate innovation, he needed to remove complexity from the company’s software supply chain. He enacted three moves to eliminate waste and complexity within the open source and third-party components being used:
1. Automate the scrutiny of software suppliers to minimize risk and bloat.
2. Enable real-time traceability and visibility to components used across the development lifecycle, in order to improve response times when defects were discovered – cutting back from days/weeks to minutes!
3. Reducing complexity, rework and risk by standardizing on specific components types (e.g., web frameworks, logging frameworks, encryption modules), using fewer and more current artifact versions, and managing out elective security risks.
Want to hear the rest of the story?

The same principles that Bill applied to his business that boosted developer productivity by 15% are being discussed by Gene Kim (@RealGeneKim), author of The Phoenix Project, and Joshua Corman (@JoshCorman), CTO of Sonatype, on April 30th. Both Gene and Josh are huge fans of marrying DevOps practices with leading supply chain management principles that enable developers to maximize throughput of features from “code complete” to ‘in production,” without causing chaos.

If you’re looking to achieve similar goals, be sure to join Josh and Gene for this April 30th, 1pm ET, discussion.
April 23, 2015
31 Reference Architectures for DevOps and Continuous Delivery
At QCon London, David Farley (@davefarley77) told the audience that “continuous delivery changes the economics of software delivery”. I could not agree more.

If you have been drawn to the evangelists like David Farley, Jez Humble, and Gene Kim, you’ll know that high performance IT organizations are seeing a massive payoff in their continuous delivery investments. Industry leaders big and small are sharing noticeable results:
- 8x more frequent production deployments
- 50% lower change failure rates
- 12x faster service restoration times when something went wrong
5 Principles

How are they achieving these results? They are following a few key principles that David and Jez outlined in their seminal book, Continuous Delivery (highly recommended). The principles outlined in their book and in David’s QCon presentation are as follows:
- Deliver fast
- Automate almost everything
- Keep everything in version control
- Build quality In
- Empower the team
Reference Architectures for DevOps and Continuous Delivery

In their book, they also provide insight to continuous delivery tool chains and processes which highlight build tools, CI platforms, testing suites, artifact repositories, and many other components. There are numerous examples of reference architectures available, and each of them vary in levels of detail, tools highlighted, and processes followed. Yet, there is a constant theme among the tool sets: Jenkins, Maven, Nexus, Subversion, Git, Docker, Puppet/Chef, Rundeck, and Sonar seem to show up time and again.

To help you along your continuous delivery and DevOps journey, I have compiled a set of 31 reference architectures created by users across the continuous delivery and DevOps communities.

Each architecture is accompanied by a link to the original presentation or blog where the architecture was referenced to ensure you have access to the full context of the discussion. You can also freely download the presentation from SlideShare where we now have over 7,000 views.

Offer to help others: If you have a reference architecture to share, please consider posting a link to it in the comments section below. I am sure others would love to see it and learn from it.
April 22, 2015
DevOps: The Last Great Hope for Application Security?

Once upon a time, there was a great battle between speed and security. Development wanted to go fast. But, Security wanted to slow down and be safe.

“We must protect our gilded apps”, cried the application security team.

“Speed is cherished by our people”, declared the development team.

For years, they endured the pain of testing late in the lifecycle, sorting through reams of false positive reports, and dealing with the added cost of pushing bad software out the door. They knew there had to be a better way…

And then came, The DevOps Revolution. The DevOps team had an answer. “Let’s bring Application Security and Development closer together — and shift its focus further to the left”. The DevOps team knew that by introducing awareness of security vulnerabilities and policies early in and across the software development lifecycle — without creating a time-consuming tax on development — that both teams could win.

During an evening event named “Wining Not Whining” at the RSA Conference 2014, we gathered some of the top DevOps experts and influencers and asked them “Why is application security so important to the DevOps revolution?”

Listen to what they had to say:

March 31, 2014