devopsy.kinsta.cloud

Tag: open source

Survey: Most Open Source Software Maintainers Want to Get Paid

A survey of 339 maintainers of open source software projects found 60% of them classified themselves as unpaid hobbyists versus only 13% who said they earn most or all of their income from maintaining projects. Slightly less than a quarter (23%) described themselves as semi-professionals, earning some of their income from maintaining projects. Overall, more than three-quarters of unpaid maintainers (77%) would prefer to be paid for their efforts.

The survey was conducted by Tidelift, a provider of a platform for managing open source software use, and found there is a clear correlation between getting paid and the amount of time spent actually maintaining a project. A full 81% of professional maintainers spend more than 20 hours per week maintaining their projects, compared to 27% of semi-professional maintainers and only 7% of unpaid hobbyist maintainers, the survey finds.

In fact, paid maintainers are much more likely to have reproducible and verifiable build processes (77%), formal backward compatibility policy (71%), a security disclosure plan (69%), provide fixes and recommendations for vulnerabilities (69%) and have a defined dependency management process (57%).

Unfortunately, the survey also made it clear that more than half of maintainers of open source software projects (52%) are not even aware of emerging frameworks to better secure software.

On the plus side, however, the survey also found that among maintainers that are aware of those frameworks 43% have already begun work to align with them or plan to start within the next year. Alas, 39% said they have no plans to align to these industry standards, while another 19% are still on the fence. Well over one-third of maintainers (38%) who do not plan to align their projects with industry standards said they just don’t have the time, while 37% won’t do it because they are not being paid for the work.

Nearly half of maintainers (47%) want to be paid for undertaking the work needed to align their projects with the security frameworks, with 54% of maintainers noting they would appreciate help that would enable them to better understand these frameworks and how they apply to their project.

Tidelift CEO Donald Fischer said as more organizations are concerned about the security of the open source software used across software supply chains, the survey makes it clear that maintainers need some sort of financial compensation to make it worth their time and effort to address vulnerabilities in a timely manner. That’s especially critical, especially as legislation is pending that would hold software developers more liable for vulnerabilities in software. In the absence of being paid to more proactively address vulnerabilities, many open source software maintainers may abandon projects altogether because of liability concerns.

It’s not clear to how enterprise IT organizations are evaluating which open source projects developers can use based on the level of support made available. What is clear is that not all open source software is created and supported equally well. Of course, many of the organizations benefitting from open source software could be doing a lot more—not just in terms of financial support, but also helping with everything from identifying vulnerabilities and testing patches to simply writing better documentation.

May 2, 2023
Breaking Down Silos: Applying Open Source Practices in the Workplace
DevOps plays a prominent role in mainstream technology and companies’ ability to deliver results. While the definition of DevOps encompasses the complete range of characteristics it covers, in practice, current tech culture tends to focus solely on the tooling aspect, neglecting the equally essential focus on the people and process critical in delivering value to end users. While cultures are difficult to change and redefine, a transformation is needed to fully capitalize on the potential DevOps has to accelerate software development.

Conway’s law–the concept that organizations ship software based on their organizational structure–outlines the need for visibility. This law revolves around the idea that in order for a software product to successfully operate, its developers must have established open pathways of communication to ensure each component works as a whole rather than as individual pieces. Unfortunately, today, many organizations still run into problems with how their teams operate when individuals own limited pieces of development.

Take a retail platform, for example. In the development of the platform, there will likely be several sub-teams that focus on individual components of the design. Perhaps there’s a team for the shopping cart service, another for the catalog service, a team for the underlying data platform and one for user account services. The list goes on. Typically, those teams do not have visibility into each other’s work and operate in their own silo. As soon as teams start thinking about ‘their’ component or ‘their’ microservice, a dangerous tunnel-vision takes hold. In some cases, there may be standup meetings across teams for coordination efforts, but this takes time, energy and steals focus from the important task: Delivering value to end users.

Alternatively, think about your favorite open source project. The inner workings were likely designed asynchronously, openly, remotely and at a global scale. Given the remote nature of open source innovation, developer teams don’t need to be co-located in the same workspace. While physically operating separately, open source developers have mastered alternative methods to operate with transparency in such a way that external contributions aren’t just welcome, but encouraged.

This same idea applies to proprietary software development, or InnerSource, a modern engineering approach that applies open source practices inside your organization to proprietary code. While a technological shift in practice, InnerSourcing is also a cultural revolution toward collaboration.

Companies of all types can benefit from implementing InnerSource practices. Some enterprise organizations that operate using InnerSource methodologies today include 3M, Ford, KPMG, Mercedes-Benz, Nubank, Spotify and more.

A critical component in making this change is prioritizing team visibility. If other teams in your organization are unable to see what you’re working on, you’re inherently (intentionally or not) creating silos and limiting efficient collaboration. In practice, there will be scenarios where company-wide visibility may not be feasible due to external regulations, highly secure environments or sensitive projects; however, these moments of limited project visibility should be the exception rather than the default.

When approaching InnerSource through an open source mindset, you will find that many DevOps principles still hold true. Here are a few strategies for implementing a collaborative open source culture when developing proprietary code:
1. Use existing resources: When problem solving, using the existing materials available within your organization is a great place to start—it is possible someone has already tackled the issue you are addressing. If you find the right fit, you can reuse prior work and, if it has not been started, this is an opportunity to add value to your organization by creating a project that could help others in the future. When building upon someone’s work or starting a project, prioritizing visibility by providing an explanation of any technical requirements of the change will allow future project maintainers to determine whether it’s in the project scope, already a proposed change, on the backlog, or not a prioritized piece of work. This use of existing resources will help develop a collaborative culture over time.
2. Documentation of work: Documentation is critical. Keeping track of the expected standards of the project and which tools or dependencies are required to build the project for local testing will help others in your organization know what to contribute. When in the early stages of contributing code to a project, documenting proposed changes without directly impacting the production version will allow you to receive feedback early on from the project maintainer about the direction you should take as they will be able to see your steps clearly outlined. To contribute code to a project, you can propose changes as a pull request, which doesn’t impact the production version but means you can get valuable feedback on the direction the project should take. Handling contributions by a pull request also ensures that branch protection rules apply and continuous integration practices are required.
While implementing these practices may not result in immediate changes, over time they will result in a significant cultural shift. And while cultural transformations require continued conscious efforts, embracing modern software engineering practices–learned from the open source community–will enable teams to break barriers for organizational collaboration.
May 2, 2023
Open Source Communities Need More Than Funding From Big Tech

In recent years, major players like Microsoft, Amazon and Google have increased their involvement in open source projects, providing both financial and technical support. More consistent contributions from big tech—whether in the form of people or resources—is exactly what the open source community needs. However, underdeveloped collaboration guidelines can lead to friction between large corporations and open source developers, which ultimately harms both parties. Friction jeopardizes tech companies’ ability to rely on open source for operations while leaving the open source community without the support it needs. To make their relationship more sustainable, both parties need to cultivate a greater level of trust, develop clear guidelines for collaboration, and prioritize community.

Big Tech Contributions: Moving From Cash to Collaboration

For years, major tech companies have used open source solutions in their tech stacks. But many open source developers—who receive little to no money for their work—have voiced concerns about companies that take more from the open source community than they give.

In many ways, big tech has answered the call. For example, Amazon recently passed IBM as the fifth-largest contributor to open source projects. Companies like IBM and Amazon have a vested interest in supporting these projects—if open source projects are understaffed and underfunded, the quality of the solutions suffers.

But many tech companies are grappling with an unstable financial environment, which makes hefty donations more difficult to justify. As a result, support from big tech often involves donations of their developers’ time. However, this is also challenging as tech companies like Twitter and Google continue to lay off the developers who were keeping many open source projects afloat.

Yet, even before recent layoffs, collaborations between big tech and open source teams were strained. There’s a natural power imbalance between a corporation like Amazon and the open source community, which includes members who contribute to these projects in their free time, sometimes for little or no pay. Big tech developers must juggle the interests of the open source team they’re collaborating with and their own employer—with little guidance on how to do so.

A lack of transparency and communication from big tech also puts open source developers in a tricky situation. Many developers need financial and personnel support but may not want to relinquish control of their solutions to an outside corporation.

To alleviate this tension, big tech needs to establish clear, mutually beneficial guidelines for open source collaboration that prioritizes sustainability.

Reconciling the Fraught History Between Big Tech and Open Source

Past tensions between big tech and the open source community can affect collaboration if left unaddressed. For example, Microsoft has become one of the biggest contributors to the Linux kernel, but many open source teams still remember former Microsoft CEO Steve Ballmer calling Linux a “cancer” in 2001.

To overcome this tension, big tech companies need to clarify their intentions from the beginning of a project. They should disclose their reasons for selecting the project, the developers who will work on that project and how their contributions will look. Open source maintainers also have a role to play here. They must properly vet these companies and inquire about their prior open source experience and strategies before welcoming them into the fold.

One way to reduce the risk of power imbalances or other tension is by collaborating through a project like the Linux kernel. Google, IBM and many other corporations contribute toward Linux kernel, and this distribution of contributors creates a much more balanced playing field. Rather than one company assuming operational control, multiple companies play a small part in keeping the Linux kernel up and running.

The Benefits of an Open Source Program Office

While these guardrails certainly help, big tech companies also need to formalize their open source contribution processes. At Aiven, we’ve done this through our open source program office (OSPO), which helps us manage our relationships with the open source communities we rely on. OSPOs ensure organizations remain compliant with open source licensing and provide a foundation to expand their involvement with the community over time.

When developing an OSPO, tech companies should be intentional about how they measure success. The most obvious metrics relate to the work developers do—for example, the number of features delivered, the number of fixes and the number of projects they contributed to. But OSPOs should also measure how developers contribute to the community itself, not just the projects within it. For example, do developers review work from other developers in the community? Do they attend conferences, write blog posts and share knowledge with other developers?

Big tech leaders must place as much emphasis on community-building as they do the number of contributions developers make. If they don’t evaluate developers based on their engagement with the community, the developer is less likely to focus their efforts there, injecting toxicity into the communities they’re trying to support.

Open Source Thrives on Collaboration. Let’s Keep it That Way

The challenge of reducing the stress and burnout of open source developers can’t be solved solely through financial means. No amount of money will reduce the amount of work it takes to maintain an open source project.

To make collaborations between open source teams and big tech sustainable, both sides need to prioritize supporting the community above all else and develop guidelines that support this mission. More sustainable collaborations will ensure that the open source community gets the support it needs while big tech continues to reap the rewards of the community’s innovations.

May 2, 2023
Report: Most IT Teams Can’t Fix Open Source Software Security

Lineaje, a provider of a platform for securing software supply chains, today published an analysis of 41,989 open source components embedded in the top 44 popular projects managed by the Apache Software Foundation (ASF). That analysis found more than a quarter (26%) of vulnerabilities are not patchable by the application development team that deployed them.

In addition, the report found that 64% of the vulnerabilities analyzed have no patch as of yet. Overall, 68% of the vulnerabilities analyzed are due to dependencies created when an open source software project included a component or package developed by another open source project maintainer.

Lineaje CEO Javed Hasan said that means that 90% of open source dependencies are transitive, in the sense that they are created when maintainers of a project included a vulnerable open source component created by another entity. Only 10% of the vulnerabilities discovered by Lineaje are the result of a dependency that an application development team could actually address on their own, he added.

Overall, the report found a full 82% of components that are relied on to build open source software are inherently risky due to vulnerabilities, security issues, code quality or maintainability concerns. In addition, 5% of components failed a basic integrity check, with 3% having no known origin.

Open source software is not created equal no matter what consortium is overseeing its development, noted Hasan. Unfortunately, there is no universally accepted method for validating the integrity of open source software, so application development teams should proceed with caution whenever employing open source software, he said.

The challenge, of course, is that developers have been reusing open source software within applications for years now. It’s only in the recent wake of breaches involving open source software that organizations have been taking a harder look at how applications are developed.

The Open Source Security Foundation (OpenSSF), an arm of the Linux Foundation, is now at the forefront of an effort to better secure open source software by focusing on 10 streams of investment that, in total, will require more than $150 million in funding to drive greater adoption of DevSecOps best practices among maintainers of open source software projects. The issue is that many of those projects are maintained by a small number of programmers that voluntarily contribute their time and effort to build components that others are free to use. Like any other developer, the amount of security expertise those individuals have is limited. The onus for making sure open source software is secure when deployed in a production environment belongs to the organization that deploys it.

Hopefully, following a recent executive order issued by the Biden administration, the level of open source security will steadily improve in the months ahead—assuming, of course, the organizations that rely heavily on open source software make more substantive contributions to securing it. In the meantime, organizations can count on the fact that cybercriminals have taken note of the vulnerabilities in applications that can be traced back to flaws in a wide range of open source software.

April 17, 2023
Endor Labs Taps ChatGPT to Identify Secure Open Source Software

Endor Labs has launched DroidGPT, an extension of its software for assessing risks in open source code. DroidGPT integrates the ChatGPT generative artificial intelligence (AI) platform to make it simpler to discover the most secure version of an open source package.

That capability makes it possible for developers to launch a natural language query from within the Endor Labs platform that asks ChatGPT to identify, for example, the most secure logging modules of Java applications.

Endor Labs CEO Varun Badhwar said the goal is to make it simpler to apply guardrails to application development processes that today rely heavily on reusing open source packages. Developers often wind up using an older version of those packages that are not as secure because a known vulnerability has not been remediated.

Endor Labs’ Dependency Lifecycle Management platform applies graph analysis to identify the depth of dependencies that exist within an application. That capability makes it simpler to identify where vulnerable components have actually been employed within an application. DroidGPT extends that capability to identify the most secure versions of those components that developers should be using, said Badhwar.

Having a full understanding of their dependency graph also lets customers generate and analyze accurate software bills of materials (SBOMs) as applications are dynamically updated, noted Badhwar.

The level of dependency on open source software packages to create applications has risen sharply over the years. An analysis of nearly 2,000 software packages published by Endor Labs found 95% of all application vulnerabilities can be traced back to a transitive dependency created when a developer used an open source component. Functional dependencies are created whenever developers download a third-party component, so it is crucial to assess the risk levels created by those dependencies.

Fortunately, in the wake of a series of high-profile breaches, there has been increased focus on securing software supply chains. The ongoing challenge is most developers don’t have a lot of cybersecurity expertise, so as contributions are made to open source projects it’s relatively easy for mistakes to be made. The simple truth is many applications deployed in production environments are riddled with known vulnerabilities that have yet to be addressed. DroidGPT is designed to make it simpler to begin the process of remediating software vulnerabilities in applications as they are built and after they are deployed.

There is no doubt that it will take years before organizations are able to implement a set of truly mature DevSecOps best practices to teach developers how to build more secure applications. That journey, however, needs to begin with tools that enable them to address fundamental issues that dependencies inevitably create. After all, the best way to combat application vulnerabilities is to make sure they don’t manifest themselves in code in the first place. Making it easier to update applications using older versions of open source modules is nothing less than critical at a time when cybercriminals are getting more adept at exploiting them.

April 14, 2023
Creating a Growth Loop for Your Open Source Project

Every open source project hopes to achieve escape velocity, attracting enough users and contributors to become a successful, sustainable project that expands over time. But how do you attract developers and create the cycle of adoption that can lead to a thriving ecosystem?

It obviously starts with building a great product that developers love to use, and there’s an element of luck in finding this sweet spot and breaking through. But there’s also a lot that you can control. I was fortunate to have helped create an open source project, Streamlit, that expanded quickly and our company was acquired last year. I wanted to share what worked for me in the hope that it also can be helpful for you.

My guiding principle is to meet developers where they are and make it as simple as possible, at every step in the journey, to discover your project, learn how to use it and build great products. Depending on the type of open source project you’re creating, providing a simple way to share artifacts and use cases built from your code—in our case, this was applications—can also be a powerful way to raise awareness.

I think about this in terms of a “growth loop” with multiple points of entry where developers can get interested and get started. These entry points include developer forums, source repositories, your documentation page, learning assets, your website and more. The point is to eliminate friction at every point in this loop and make it as easy and fun as possible to start building and sharing with your code.

For organizations trying to get their open source project off the ground, here are some key learnings about what worked for my company in building this growth loop and some tips for driving continued adoption and innovation.

Be Visible in the Open Source Community

We made it a point to go where developers are already at, including forums like Reddit and Stack Overflow and conferences like ODSC and FOSDEM. Engaging with developers to understand what they really need from your project and how to reduce points of friction is key. We were always grateful when people gave us feedback and we reached out to any developer who engaged with our product, whether they had 10 followers or 10,000. I’ve found that developers really appreciate it when you “show up,” and you learn an incredible amount that can help your project succeed.

Give Developers What They Need

Don’t hold back features or functionality that developers will find useful. Developers should have access to everything they need and feel confident that they won’t run into a significant paywall that prevents them from scaling. Building a successful open source project is a bottom-up process, and that means eliminating barriers to entry and making your code as easy to use as possible.

Where developers go, enterprises typically follow. If you can build a tool that developers love so much they’re coding with it in their free time, they’re likely to take it into the workplace. If your goal is monetization, this is where you’re most likely to have an opportunity to recoup your investment.

Enterprises require particular features around security, access management, IT support, scaling, service level objectives (SLOs) and other areas—capabilities they are often willing to pay for. So, consider a model where you give developers everything they need to become super users and look for adjacent areas where you can charge for enterprise-type features or enterprise implementations that businesses need.

Make Sharing Incredibly Easy

If developers build great things with your code, it’s helpful if they can share those things with the community to show what’s possible and drive interest. You can think of this as a type of advertising. Most developers aren’t won over by traditional ads, but discovering cool things that other developers have done with your code can be a great way to attract new users and fuel the cycle of discovery and adoption.

So think about the natural output from your project that developers could share with the community. Can they put code on GitHub? Share an app or visualization they built? Post a speed comparison? What can they put on Twitter if they love your project and want to show what they’ve achieved with it?

Our project is for building data apps, so early on, we added a feature that allowed people to take a screen recording and share their work on Twitter so others could see it in action. Later on, we added a free cloud offering so they could directly share apps with the community. Think about what works for your project and how you could make sharing easy.

Create Amazing How-Tos and Tutorials.

These can be important entry points on your growth loop, and it’s best to provide them through developers’ preferred channels. YouTube is a go-to for tutorials of all types, and you can also leverage Coursera, Educative and other e-learning platforms. Someone on my team created a Coursera course for our project that has more than 20,000 enrollees, and one of our Code Camp videos has been viewed over a million times.

Get creative. If your project is starting to see traction among students, think about providing educational materials that professors can use in the classroom. Online interactive courses are not only a great way to learn programming, they make it easier for educators to manage coursework and see who’s keeping up. This is an excellent way to promote your project and capture new developers.

Documentation is another important entry point. Our documentation page is starting to look like an e-learning course itself, with guides for getting started, sample code and sample apps. Documentation pages can be listed by search engines, and we found it was worth the engineering time to make this happen.

Contribute to Other Open Source Projects

Giving back to the community is a good practice for anyone who cares about software, but it can also benefit your project: Contributing to open source projects helps ensure that your software will be compatible with the tools your target developers use. As well as ensuring interoperability, this can help raise awareness of your project in relevant developer communities.

Reaching critical mass with an open source project is about understanding what your target developers need, being smart with your resources and removing friction wherever you can. This concept of a growth loop worked for our project, and I hope it helps you to achieve success with yours.

April 5, 2023
npm is Scam-Spam Cesspool ¦ Google in Microsoft Antitrust Thrust

Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

This week: The npm registry suffers spam infestation, and Microsoft makes Google sad. (more…)

March 31, 2023
Oracle Makes Java 20 Platform Generally Available

At an Oracle DevLive Level Up event, Oracle today announced the availability of a Java 20 update that makes a range of innovations more accessible to application development teams.

Georges Saab, senior vice president of development for the Java Platform and chair of the OpenJDK Governing Board for Oracle, said the latest version of the venerable programming language builds on previous extensions made to the Java development kit (JDK) that exemplify how the pace of Java innovation is now increasing.

Specifically, seven JDK Enhancement Proposals (JEPs) are now supported in Java 20, including OpenJDK project Amber for record patterns and pattern matching for switch expressions and statements; enhancements from OpenJDK Project Panama to interconnect Java Virtual Machines (JVMs), features related to Project Loom that streamline the process of writing, maintaining and observing high-throughput, concurrent applications and a Function & Memory application programming interface (API) and Vector API.

Oracle also revealed Java 20 is supported by Java Management Service, an Oracle Cloud Infrastructure (OCI) native service that provides a single pane of glass to help organizations manage Java runtimes and applications on-premises or on any cloud.

Finally, Oracle is making available an Oracle Java Universal SE Subscription that provides access to a pay-as-you-go offering that includes entitlement to GraalVM Enterprise, the Java SE Subscription Enterprise Performance Pack, access to the advanced features of the Java Management Service and the flexibility to upgrade at any pace an organization prefers.

Oracle Java Standard Edition (SE), GraalVM Enterprise and the Java SE Subscription Enterprise Performance Pack are also now available free of charge on OCI.

Despite the number of programming language alternatives that exist, many enterprise IT organizations remain committed to Java. Rather than opting to retrain developers, it’s simply been more cost-effective to enable developers to continue to build applications using a programming language they already know.

Java itself, meanwhile, has been experiencing a renaissance after more organizations began contributing to projects once the core platform became available under an open source OpenJDK license. In fact, the Java community is now adopting many concepts pioneered in other programming languages to enable organizations to, for example, drive digital business transformation initiatives that are dependent on legacy applications that are being modernized. As part of those efforts, the rate at which Java applications are being deployed in the cloud has substantially increased.

As part of those efforts, many organizations are currently in the process of shifting to Java 17. It’s not clear how many organizations will skip Java 17 and jump to Java 20 from previous versions of Java, but DevOps teams should expect to be supporting a wide range of versions of Java for years to come.

In fact, most organizations will deploy Java applications everywhere from the network edge to the cloud. The one thing all these applications will have in common is they will be written in a programming language that has persevered through two decades and appears to have enough staying power to stick around well into the next.

March 21, 2023
Grafana Labs Acquires Pyroscope to Add Code Profiling Capability

Grafana Labs this week announced it acquired Pyroscope and plans to merge Pyroscope’s open source continuous profiling project with the open source continuous profiling database that Grafana Labs launched last year.

Richard Hartmann, director of community at Grafana Labs, said after logs, metrics and traces, continuous profiling is now the fourth pillar of observability that enables DevOps teams to identify how much infrastructure resources any given piece of code is consuming. That capability is critical for everything from controlling costs to improving performance to limiting the amount of carbon generated by an application both before and after an application is deployed, he added.

The resulting Grafana Pyroscope project will be natively integrated with Grafana Labs’ open source dashboard project to make it simpler to visualize profiling data and correlate it with metrics, logs, and traces. Grafana Labs is also committing to adding profiling capabilities to the fully managed Grafana Cloud observability platform.

Pyroscope currently makes a plug-in for Grafana available in addition to a cloud instance of Pyroscope. Integrating Pyroscope with the Grafana Labs portfolio will make it simpler for DevOps teams to visualize issues before there is a major disruption to an application environment, said Hartmann.

It’s still early days as far as the adoption of observability platforms is concerned, but it’s apparent that as application environments become more complex, the plethora of monitoring tools that DevOps teams rely on will need to be streamlined. Observability platforms promise to unify logs, metrics and traces in a way that makes it simpler to launch queries to identify the root cause of an issue. The rate at which DevOps teams will embrace observability will naturally vary, but the biggest obstacle might not be the platforms themselves. Instead, the issue is understanding what queries can help DevOps teams better understand the root cause of an IT issue. Visualization tools, in combination with code profiling tools such as Pyroscope, should make it simpler to surface those issues.

Grafana Labs is also enabling organizations to better address sustainability concerns that have emerged as more policies are enacted to encourage organizations to reduce the overall size of their carbon footprints. The European Union, especially, has been enacting policies that provide organizations with a range of incentives to encourage greater efficiency from IT environments. It won’t be long before more DevOps tools and platforms provide greater visibility and control over the amount of infrastructure application code is consuming to achieve sustainability goals.

In the meantime, overall observability should continue to improve as organizations move beyond continuous monitoring to gain insights into how application environments are behaving. As IT environments become more complex, tools that monitor a set of pre-defined metrics are no longer enough to ensure application resiliency. The challenge—and the opportunity—is to determine how to take observability to the next level before the next inexplicable application performance degradation.

March 17, 2023
How Bazel and GitHub Can Fix the Dependency Availability Problem

Recently, GitHub upgraded the internal version of Git they use to produce repository archives. You’ve probably used these archives before if you’ve downloaded a .zip or .tar.gz file from a repository at a particular version. GitHub produces those archives on demand using Git archive and caches them for a short time.

Upgrading Git regularly is a generally good idea, but this change regrettably broke a huge number of Bazel projects. Bazel is a widely used, free, open source tool for building and testing software, so this breaking change is a big deal. What happened? Most Bazel projects fetch at least some of their dependencies using rules in their WORKSPACE files like this:

http_archive(

name = “com_github_bazelbuild_buildtools”, sha256 =

“05eff86c1d444dde18d55ac890f766bce5e4db56c180ee86b5aacd6704a5feb9”, strip_prefix = “buildtools-6.0.0”,

urls =

[“https://github.com/bazelbuild/buildtools/archive/refs/tags/6.0.0.ta

r.gz”],

)

See that /archive/refs/tags/ part of the path? That’s the endpoint I’m talking about.

This is bare-bones dependency management: Bazel attempts to download an archive from the first URL in the list; it tries the next URL if the first is not available and so on. Bazel then checks the file’s SHA-256 sum against the known value and, if it’s correct, extracts the archive and proceeds with the build.

The Git upgrade caused a change in archives’ SHA-256 sums. I think there was a small change in zip compression, but it doesn’t really matter—any variation in file ordering, alignment or compression causes the archives’ SHA-256 sums to change even though the extracted contents are the same.

This is at least the third time Bazel builds have broken that I can remember. This has also been discussed extensively before. I’m writing this in the hope that we can make our systems more resilient and avoid these kinds of problems in the future.

Since GitHub made the change that triggered this, they naturally get the immediate blame from the community, though I think it’s mostly undeserved. Upgrading dependencies (especially Git and especially if you’re GitHub) is a reasonable thing to do. To my knowledge, GitHub has not documented a guarantee that files returned by the archive endpoints have stable SHA-256 sums. It’s a mistake for users to rely on a guarantee that was never made. It’s tempting, of course, because it’s easy, but it’s a mistake nonetheless.

This is a classic example of Hyrum’s Law:

“With a sufficient number of users of an API, it does not matter what you promise in the contract: All observable behaviors of your system will be depended on by somebody.”

Since these updates have broken Bazel (and presumably others) a few times now, I’d really like to see GitHub clarify in documentation whether users should or should not depend on stable archive SHA-256 sums. A GitHub engineer commented that this is not stable, but product managers and support engineers have commented at other times that is stable. I don’t really think discussion comments count since they’re not discoverable. Only official documentation is authoritative.

I haven’t actually found any documentation for these release archive URLs, so I’m not sure where this clarification should go. It’s not part of the REST API. Linking to releases is pretty close.

If archive SHA-256 sums are guaranteed to be stable (now or in the future), I think documenting and testing that would let us all sleep easier at night.

If archive SHA-256 sums are not guaranteed to be stable, it wouldn’t be a terrible idea to inject a little chaos to prevent people from depending on them. For example, in Go, the iteration order of elements in a map is undefined. To prevent developers from depending on iteration order (and tests from breaking when the hashing algorithm is tweaked), the Go runtime adds a random factor into the hashing algorithm, so the iteration order is different every time a program runs. Something similar could be done here with archive file order or alignment. I wouldn’t suggest gratuitously breaking this API, but if it needs to change anyway for some reason in the future, it would be a good idea to add something like this.

What Could the Bazel Community do Better?

Bazel developers should not rely on stable archive SHA-256 sums unless that stability is guaranteed and documented by GitHub. More importantly, developers should not rely on dependency artifacts being available on GitHub at all: A library author could delete their project at any time.

I’ll point to Go modules as a model of a great dependency management system designed to solve this exact problem. The Go team operates proxy.golang.org, a mirror for all publicly available Go modules. Internally, the proxy stores actual files for each module and does not need to regenerate them. The proxy protocol is open and easy to implement as an HTTP file server, so you can run your own proxy service for better availability. I’d love to see something like this happen for Bazel, especially if it’s operated by Google. It is not technically difficult to build a service like this, but there are a lot of thorny issues around handling abuse and legally distributing software with unrecognized licenses, and Google has already figured out those issues for Go.

Until such a service exists, developers can protect themselves by copying their dependencies to their own mirror. A GCS or S3 bucket works fine.

Library authors can and should protect their users by providing static release artifacts (not dynamically generated archives), and mirroring those. For example, check out the http_archive boilerplate for rules_go:

http_archive(

name = “io_bazel_rules_go”,

sha256 =

“dd926a88a564a9246713a9c00b35315f54cbd46b31a26d5d8fb264c07045f05d”, urls = [

“https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/d ownload/v0.38.1/rules_go-v0.38.1.zip”,

“https://github.com/bazelbuild/rules_go/releases/download/v0.38.1/rul es_go-v0.38.1.zip”,

],

)

The file rules_go-v0.38.1.zip is created by the rule authors and attached to the release; it’s not dynamically generated.

It’s also copied to mirror.bazel.build, which is a thin frontend on a GCS bucket shared by many rule authors in the Bazelbuild organization.

One other tip: If you’re feeling adventurous enough to use an experimental, undocumented feature (to make your build more stable! Really!), you can configure Bazel’s downloader to rewrite those GitHub URLs to point to your own mirror.

Aside: SHA-256 of archives or contents?

It’s unfortunate that a change to the Git archive that does not affect extracted contents of an archive can still change its SHA-256 sum. Bazel absolutely does the right thing by checking the sum of the downloaded file before extracting its contents.

This is the (delightfully named) Cryptographic Doom Principle. If Bazel only authenticated the contents of an archive, it might be possible for an attacker to exploit a vulnerability in Bazel’s zip parser before the archive is authenticated. Since Bazel authenticates the archive before extracting it, the pre-authentication attack surface is very small.

Closing Thoughts

When you’re designing software, think carefully about how it’s going to be used. If there’s a right way and a wrong way to do something, make sure the right way is easier and more obvious. Better yet, make the right way the only way.

I think this is a case where Bazel’s dependency management is too limited: To use http_archive safely, you need to set up an HTTP mirror with copies of your dependencies. That’s too much work for users, especially new users who aren’t aware of the hazards. A more complete dependency management system should include an artifact registry or a read-through caching system with at least one public implementation. I was hoping Bazel modules and the Bazel central registry would provide that, but the central registry only includes module metadata: Module content is separate, specified in URLs that still frequently refer to the unstable GitHub endpoint.

March 16, 2023