devopsy.kinsta.cloud

Tag: dependency

  • How Bazel and GitHub Can Fix the Dependency Availability Problem

    How Bazel and GitHub Can Fix the Dependency Availability Problem

    Recently, GitHub upgraded the internal version of Git they use to produce repository archives. You’ve probably used these archives before if you’ve downloaded a .zip or .tar.gz file from a repository at a particular version. GitHub produces those archives on demand using Git archive and caches them for a short time.

    Upgrading Git regularly is a generally good idea, but this change regrettably broke a huge number of Bazel projects. Bazel is a widely used, free, open source tool for building and testing software, so this breaking change is a big deal. What happened? Most Bazel projects fetch at least some of their dependencies using rules in their WORKSPACE files like this:

    http_archive(

    name = “com_github_bazelbuild_buildtools”, sha256 =

    “05eff86c1d444dde18d55ac890f766bce5e4db56c180ee86b5aacd6704a5feb9”, strip_prefix = “buildtools-6.0.0”,

    urls =

    [“https://github.com/bazelbuild/buildtools/archive/refs/tags/6.0.0.ta

    r.gz”],

    )

    See that /archive/refs/tags/ part of the path? That’s the endpoint I’m talking about.

    This is bare-bones dependency management: Bazel attempts to download an archive from the first URL in the list; it tries the next URL if the first is not available and so on. Bazel then checks the file’s SHA-256 sum against the known value and, if it’s correct, extracts the archive and proceeds with the build.

    The Git upgrade caused a change in archives’ SHA-256 sums. I think there was a small change in zip compression, but it doesn’t really matter—any variation in file ordering, alignment or compression causes the archives’ SHA-256 sums to change even though the extracted contents are the same.

    This is at least the third time Bazel builds have broken that I can remember. This has also been discussed extensively before. I’m writing this in the hope that we can make our systems more resilient and avoid these kinds of problems in the future.

    Since GitHub made the change that triggered this, they naturally get the immediate blame from the community, though I think it’s mostly undeserved. Upgrading dependencies (especially Git and especially if you’re GitHub) is a reasonable thing to do. To my knowledge, GitHub has not documented a guarantee that files returned by the archive endpoints have stable SHA-256 sums. It’s a mistake for users to rely on a guarantee that was never made. It’s tempting, of course, because it’s easy, but it’s a mistake nonetheless.

    This is a classic example of Hyrum’s Law:

    “With a sufficient number of users of an API, it does not matter what you promise in the contract: All observable behaviors of your system will be depended on by somebody.”

    Since these updates have broken Bazel (and presumably others) a few times now, I’d really like to see GitHub clarify in documentation whether users should or should not depend on stable archive SHA-256 sums. A GitHub engineer commented that this is not stable, but product managers and support engineers have commented at other times that is stable. I don’t really think discussion comments count since they’re not discoverable. Only official documentation is authoritative.

    I haven’t actually found any documentation for these release archive URLs, so I’m not sure where this clarification should go. It’s not part of the REST API. Linking to releases is pretty close.

    If archive SHA-256 sums are guaranteed to be stable (now or in the future), I think documenting and testing that would let us all sleep easier at night.

    If archive SHA-256 sums are not guaranteed to be stable, it wouldn’t be a terrible idea to inject a little chaos to prevent people from depending on them. For example, in Go, the iteration order of elements in a map is undefined. To prevent developers from depending on iteration order (and tests from breaking when the hashing algorithm is tweaked), the Go runtime adds a random factor into the hashing algorithm, so the iteration order is different every time a program runs. Something similar could be done here with archive file order or alignment. I wouldn’t suggest gratuitously breaking this API, but if it needs to change anyway for some reason in the future, it would be a good idea to add something like this.

    What Could the Bazel Community do Better?

    Bazel developers should not rely on stable archive SHA-256 sums unless that stability is guaranteed and documented by GitHub. More importantly, developers should not rely on dependency artifacts being available on GitHub at all: A library author could delete their project at any time.

    I’ll point to Go modules as a model of a great dependency management system designed to solve this exact problem. The Go team operates proxy.golang.org, a mirror for all publicly available Go modules. Internally, the proxy stores actual files for each module and does not need to regenerate them. The proxy protocol is open and easy to implement as an HTTP file server, so you can run your own proxy service for better availability. I’d love to see something like this happen for Bazel, especially if it’s operated by Google. It is not technically difficult to build a service like this, but there are a lot of thorny issues around handling abuse and legally distributing software with unrecognized licenses, and Google has already figured out those issues for Go.

    Until such a service exists, developers can protect themselves by copying their dependencies to their own mirror. A GCS or S3 bucket works fine.

    Library authors can and should protect their users by providing static release artifacts (not dynamically generated archives), and mirroring those. For example, check out the http_archive boilerplate for rules_go:

    http_archive(

    name = “io_bazel_rules_go”,

    sha256 =

    “dd926a88a564a9246713a9c00b35315f54cbd46b31a26d5d8fb264c07045f05d”, urls = [

    “https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/d ownload/v0.38.1/rules_go-v0.38.1.zip”,

    “https://github.com/bazelbuild/rules_go/releases/download/v0.38.1/rul es_go-v0.38.1.zip”,

    ],

    )

    The file rules_go-v0.38.1.zip is created by the rule authors and attached to the release; it’s not dynamically generated.

    It’s also copied to mirror.bazel.build, which is a thin frontend on a GCS bucket shared by many rule authors in the Bazelbuild organization.

    One other tip: If you’re feeling adventurous enough to use an experimental, undocumented feature (to make your build more stable! Really!), you can configure Bazel’s downloader to rewrite those GitHub URLs to point to your own mirror.

    Aside: SHA-256 of archives or contents?

    It’s unfortunate that a change to the Git archive that does not affect extracted contents of an archive can still change its SHA-256 sum. Bazel absolutely does the right thing by checking the sum of the downloaded file before extracting its contents.

    This is the (delightfully named) Cryptographic Doom Principle. If Bazel only authenticated the contents of an archive, it might be possible for an attacker to exploit a vulnerability in Bazel’s zip parser before the archive is authenticated. Since Bazel authenticates the archive before extracting it, the pre-authentication attack surface is very small.

    Closing Thoughts

    When you’re designing software, think carefully about how it’s going to be used. If there’s a right way and a wrong way to do something, make sure the right way is easier and more obvious. Better yet, make the right way the only way.

    I think this is a case where Bazel’s dependency management is too limited: To use http_archive safely, you need to set up an HTTP mirror with copies of your dependencies. That’s too much work for users, especially new users who aren’t aware of the hazards. A more complete dependency management system should include an artifact registry or a read-through caching system with at least one public implementation. I was hoping Bazel modules and the Bazel central registry would provide that, but the central registry only includes module metadata: Module content is separate, specified in URLs that still frequently refer to the unstable GitHub endpoint.

  • DevOps: Trust, Obligations and Promises

    DevOps: Trust, Obligations and Promises

    Mark Burgess (@markburgess_osl) is a theoretical physicist, but in his keynote at the 2016 All Day DevOps conference, he talked more about economics and human interactives than physics. What does either have to do as the keynote for a conference on DevOps?

    Well, for a little more background, Mark Burgess is also the founder and former CTO of CFEngine, a configuration management and automation framework, and is the author of “Promise Theory.” While at CFEngine, Mark worked to apply a theory of how the autonomous agents in software interact with each other. Promise Theory was born.

    Exploring Promise Theory

    Promise Theory sits counter to obligation theories. Obligation theories view human interactions and behavior under the assumption that people (or agents, in the case of software) choose behavior based on their obligation to follow the rules. Agents are obligated to take a certain action. Promise theory contends the desire to follow the rules is voluntary and that agents can only be responsible for their own behavior, not the behavior of other agents.

    In his presentation, Mark started out looking at human behavior and trends in our society. For instance, with technological advances, our need for one another is decreasing, even as our interdependence through communication is increasing. For example, our smartphones enable us to do almost anything without the direct interaction of someone else.

    Looking at interactions in society, trust is the basis for our exchanges, working communities, and function as a society. He contends, “The promise is the source of intent by which all promises and observers may calibrate their expectations.” Money is an example of this. Money does not have intrinsic value, but we use it because we trust the backer will stand behind its value and it is more convenient than bartering goods. He explains that in the old monetary economy, banks and governments were the provider of trust; in the new service/IT economy, the service provider is the basis of trust.

    How Agents Make Promises

    How do you model this trust? Enter Promise Theory, which describes how agents make promises with each other and how this leads to trust between them. Mark outlines the “ingredients” of Promise Theory:

    1. Agents and super agents make promises in a scaling hierarchy.
    2. An agent can never make a promise about another agent’s behavior (only its own).
    3. An agent’s promise need not be accepted or used by its intended or unintended recipient!
    4. An agent makes its own valuations: what you have is only worth what another is willing to give you for it.
    5. Dependency invalidates promises.

    At the core of understanding this, you have to understand that collaboration is made up of semantics, what words mean and dynamics: all how things work.  And that our ever-increasing service economy is driving us to an explosion of semantics, because words matter so much more than the mechanics. As an example, Mark sketched the hierarchy of promise dependencies in the old and new economies. Looking at the old economy, where we have more dynamic money, the hierarchy is:

    Retail→ Money→ Banks→ Central Banks→ Government.

    In the new economy, where we have more semantic services, the hierarchy is:

    Client→ Network Availability→ Provider→ Internet Authorities→ Company/Service Organization.

    The Need for Cooperation

    None of these systems could work without cooperation at the bottom of it all. So, production is a cognitive process between the producer and the consumer, but also between the agents inside those organizations that have to interact. Often we create pipelines to describe these interactions, but the interdependencies within the systems are often much more complex than the pipeline can model.

     

    This is all just the tip of the iceberg. Mark has three books that dive into it in detail: “Thinking in Promises,” “In Search of Certainty,” and, “Promise Theory.”

    In the end, Mark made the point that technology does an excellent job of both binding us together and pulling us apart. We need to be careful that we don’t take ourselves out of the systems we create, or we could scale ourselves out of existence.

    devopsy.kinsta.cloud Supports the All Day DevOps Conference

    If this piqued your interest, you can watch Mark’s full All Day DevOps conference session (just 40 minutes).  When you do, you can view the other 56 presentations from the 2016 conference online and free of charge.  Finally, be sure to register you and the rest of your team for the 2017 All Day DevOps conference here.  This year’s event will offer 96 practitioner-led sessions (no vendor pitches allowed).  It’s free and online on Oct. 24.  devopsy.kinsta.cloud is a media sponsor for All Day DevOps 2017.

    — Derek E. Weeks