Welcome to To Be Continuous, a show about continuous delivery and software development, hosted by Heavybit members Paul Biggar, founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly.
In this episode, Paul and Edith discuss the fallout after the widely covered “left-pad” incident, and while they agree that these sorts of moments move the entire open source community forward, they wonder: Have we forgotten how to code?
[soundcloud url=”https://api.soundcloud.com/tracks/264605904?secret_token=s-Kv9Ep” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]
Edith: Hey, Paul. So, somebody unpublished a module called left-pad and the internet basically exploded. And now it’s been a couple weeks since then, so there’s a lot of different topics to cover in terms of what this means for open source, whether it means that people have forgotten how to program.
Paul: So LaunchDarkly. Or, I’m sorry, the left-pad. I’m obviously looking at my own t-shirt.
Edith: You’re looking at your LaunchDarkly t-shirt?
Paul: The left-pad incident was pretty interesting and one of the things that we did when coming up to this was just, like, looking at all the Hacker News discussions and comments on it.
And there’s wildly varying positions on what this means for open source, what it means for whether people can program, what it means for, like, about the Node ecosystem. So to kick this off, why don’t you give us a recap of the entire incident?
Edith: Yeah. So, in short, a programmer had a module called Kik, K-I-K. Which is also [the name of] a very popular messaging app. I mean, even I’ve heard of it and I’m not a teenager.
Paul: The Kik lawyers contacted the dude. Said, “Please take down your module, change it.” And the programmer said, “No. I basically don’t believe in copyright laws.” So then, the Kik people contacted NPM. The NPM people took down the module called Kik. And the guy, in I’d say, kind of a fit of pique, said, “Okay, you took down my one module, I’m gonna take down all my modules.” Basically, I’m taking my marbles and going home.
So, he took down all of his modules. One of which turned out to be an incredibly popular module called left-pad, which did basically what the name implied. It sorted left pads. And without this in NPM, the quote I read was, “Builds all over the world started to fail very quickly.”
Paul: So as I understand it, left-pad was used by another module, which was used by another module, which was used by something core, or like, this incredibly popular module—
Edith: Yeah, it was basically turtles on top of turtles on top of turtles, and at the bottom turtle was this guy who—
Paul: Took his marbles and went home.
Edith: Yeah, or yank the bottom Jenga, basically.
Paul: Gotcha. And so all the builds started to fail. People’s deploy started to fail. And people got pissed.
Edith: People got pissed because they didn’t even realize that they had this dependency. So what happened was NPM basically republished it. They said that, even though you’re the original publisher—
Paul: So there was a lot of outcry on the internet and then they republished it.
Edith: Just basically said, “You don’t have the right to un-publish this.”
Paul: Gotcha. So, um, but they had an Unpublish button, as I understand it. So he did something that was built into NPM to remove a module.
Edith: Yeah, and since then they have tightened their rules a lot. They basically said that this should be something that’s used extremely rarely, if at all, and not something you can just do capriciously.
Paul: So one thing that’s interesting about the NPM ecosystem, and the Node ecosystem in general, is they’ve had a bunch of things like this. Things that are security-related, or that are drama-related.
And people have repeatedly come out and say, “Oh, no, JS is a tar pit, or ghetto,” or whatever it is that people use to describe poor ecosystems, that normal programmers don’t know how to code, and so on, so on.
And what it looks to me like is, you know, each of these incidences brings the Node ecosystem quite a bit forward. To the point that, I mean Node is relatively new, it hasn’t had 15 years before prime time in the way that Ruby or Python did, or 20 years before it got big, and it’s a real, kind of bottom-up thing.
I mean, if someone built it and there were some steroids and then someone else took over, and then NPM took over, and then. It’s this sort of weird language from the ecosystem perspective. It didn’t have the same advantages that, like, Java had, or that C or C++ had, in order to get to maturity. And so each of these incidences propels it closer to maturity.
Edith: Yeah, I compare it somewhat to when electricity first started coming out. And then you were just working through all the issues that could happen. Like, for example, when a power line went down or a transformer blew.
Like, you work through, Okay, what are our points of failure? Let’s not make that a point of failure again.
Paul: Hmm. So, NPM changed their policies on this basis, right?
Edith: Yep, so now you can’t have an individual contributor be able to unpublish, even if they originally published. Now you have to contact them, and it’s much more stringent.
Paul: Gotcha. The response, despite people being … Well, people were annoyed at their response.
Edith: It seems perfectly rational to me.
Paul: Oh, I think it seems perfectly rational. This is the thing about the in trend. There’s always some set of people who are annoyed at everything you do.
Edith: Well you know, the internet is full of billions of people. I mean, there’s gotta be some sort of standard deviation, or otherwise it would be a very boring internet.
Paul: Right, right, and this is why there’s always haters on Hacker News. No matter what you do. If you do it one way, there’ll be one group of haters, if you do it another way, there’ll be a different group of haters.
And so, with left-pad, there were people who were annoyed that NPM retroactively decided that they had ownership over their namespace, which previously, they had sort of gifted to people in the community, the first people to take the namespace.
Edith: You mean the original thing about Kik?
Paul: So there’s that, but there’s also the left-pad. They took away the sort of autonomy of the left-pad offer.
Edith: I read a good article that basically says part of open source is, if it truly is open source, do you maintain the rights to it? Or is it a community project?
Paul: Hmm. The traditional way of deciding that sort of ownership was the fork. That you could go, and you could fork, and when you forked, you weren’t necessarily entitled to the same name, you weren’t entitled to the same trademarks. When Debian forked Firefox, they called it Iceweasel.
When Jenkins was forked, or Hudson was forked, it became Jenkins. GCC got forked. They all got new names. But what was the fork here, it wasn’t even a fork. It was a republish of the same thing, the same software, in a namespace that the original author didn’t control.
So typically you’d own the URL, you know, if you have a software library, you’d own the URL. And they did own, in some sense, the URL within NPM, or the name within NPM’s space, and NPM, the sort of owners of that namespace, overrode it.
Edith: I think they did what was necessary for good order.
Paul: Sure. I completely agree, I think that was a good decision.
Edith: I mean, so to take this to the logical conclusion, what if you had a module named left-pad, and you decided that, like, as a prank you were going rewrite it to be right-pad? Like, sure, that’s … Is that within your rights? Kind of, but it’s a real jerk thing to do that you probably want to prevent.
Paul: This is kind of one of the dangers of the whole ecosystem around centralized package managers, of sorta trusted package managers, where there’s no actual trust involved. So, what I mean by that is what is published does not have any gatekeepers.
Edith: Yeah, exactly.
Paul: I mean, it’s great, it allows an ecosystem to grow much faster, but it has safety concerns because people generally don’t pin their packages to an exact version.
Edith: You’re basically pinning it to a floating value.
Paul: You’re pinning it to a floating value. And that floating value could change. There’s no guarantee of what a minor number means or what a major number means or what a patch number means or what a patch even means. There’s some sort of, like, general community guidelines in semantic versioning, but there’s no enforcement of that.
Edith: Yeah, and I mean this basically goes back to the whole Cathedral versus Bazaar thing.
Paul: Right, right. It’s very Bazaar. And intentionally so. It’s the thing that allows a community to grow, and is there a point at which open source communities need to, like, batten down the hatches to prevent that sort of behavior? Or prevent that sort of risk?
Edith: Yeah, and it’s really interesting. I think this goes back to something we’ve talked about with Nadia Eghbal and Sean Byrnes a couple episodes ago about when you buy from a corporation, you have some sort of guarantee of standards that they publish somewhere. You know, this is our guarantees versus with open source, you don’t normally have those same sort of guarantees.
Paul: Even when you do, it tends to be—
Edith: There’s no enforcement mechanism.
Paul: Right. So there’s no enforcement mechanism in the Bazaar.
Edith: Well, there’s no, um … Like, so with a company, if a company has said, “Okay, we’re gonna release twice a year and it’s gonna have this level of quality,” and they break that, you can walk with your feet or your money. If an open source project stops or does something that you don’t like, there’s very little enforcement end.
Paul: Theoretically, you could walk. You know, you could fork the thing, but when you end up in an ecosystem that has massive network effects, and in the way that NPM does, you can’t walk with it. Because the pends is all transitive. The left-pad was, you know, a turtle way at the bottom of the stack.
Edith: It was almost like a turtle hatchling.
Paul: So, to fork left-pad, you would have to fork the entire NPM ecosystem.
Edith: Yeah, and it’s not even. The fork is not the issue, though. The issue was that people were depending on it being in a certain place in a certain way. It wasn’t that it wasn’t forked, it was just like this entire structure at the bottom was gone.
Paul: I guess what I’m saying is that if you get pissed off with a certain set of developers or a certain set of practices within the NPM community, because they’re so interlinked, you don’t have the traditional open source option of forking.
Edith: Absolutely. Absolutely.
Paul: You could rename it. You could pick a different library, but because of the way—I’m not sure if they still do this, ’cause I read something about it, but in NPM, it used to be that the version of the software that your library took was the version that library got. Do you know what I mean by that, okay. So let’s say there’s some software, let’s call it left-pad.
Edith: Let’s call it right-pad.
Paul: Right-pad. So, right-pad version one is being used by library A. And right-pad version two is being used by library B. In the Java ecosystem, and many, many other ecosystems, only one version of right-pad would be used.
And I believe in NPM, both versions of right-pad would be used. So library A will use version one, and library B will use version two. Now that’s definitely the way that it used to be, I’m not sure if they’ve actually changed that.
Edith: Yeah, I don’t know, but that sounds very messy, very quickly.
Paul: So both options are terrible. And there isn’t a right way to do it.
So, really, what it depends on. So, the NPM way of doing it, every library get its own version of the dependencies, is fantastic. Because it lets that library do what it actually intended to do under the covers.
If, however, you start having an object that gets sent around, or that two modules are, let’s say it receives a, I think, an HTTP packet or something like that would be a good example. Let’s say you take an HTTP packet from one module and just hand it to another module, and they have different understandings of the shape of that object, or what functions you can run on that object, then that’s messy.
But, on the other hand, if you override the version of right-pad that some other library’s using, you end up with all sorts of problems. This is a major problem in the, sort of the Maven ecosystem. And at CircleCI, we used Clojure, and Clojure is built on top of the Maven ecosystem, and we would have bugs where the library that we used.
One library that we used had a dependency, and that was in the package file for another thing. And so, that’s the version of the packets that we used. So we would have to pick. And as a result of that, we have to pick the library that works. And, often, there isn’t a library that works for both, or there isn’t a, sort of subdependency that worked for both. So those are close to forked no matter which way you do it.
Edith: Yeah, I mean, we ran into the same issues with our STKs. I mean, ’cause we have dependencies on other libraries that sometimes conflict.
Paul: Right, disaster.
Edith: On the other hand, I mean, the bigger point, I think, was. So there was an interesting backlash, I think, that everybody kinda projected what they wanted on the NPM thing.
People said, Oh, people don’t know how to program anymore. You know, why do you even use what they called left-pad? You know, why don’t you just grind your own wheat and make your own bread from scratch and eat it at lunch also?
Paul: I have no sympathy at all for this view of the world.
Edith: Uh, that you should make everything from scratch? Or—
Paul: That, “Oh, we’ve forgotten how to program because we need left-pad.”
Edith: Yeah. The fact that we have modules available that make it easier and easier and easier to program is a good thing.
Paul: I think it’s kinda different to that. I think it’s more of a curmudgeon view of the world. It’s a get-off-my-lawn—
Edith: Wait, that you’re the curmudgeon?
Paul: No, the people who complain, “Have we forgotten to program anymore?” I think it’s a no true Scotsman kind of argument. Real programmers know how to write their own left-pad, real programmers write in statically typed languages, real programmers write in C, real programmers, and I’m going back in time here, real programmers write in Assembly, real programmers hand-code their. Tweak the needles on the.
Edith: They do their own garbage collection.
Paul: Oh, and it’s garbage collection, what is this? Garbage collection is for amateur coders, right? I mean, you could just as easily have written, have we forgotten how to manually allocate memory?
Edith: Or, you might as well say, like, nobody knows how to make their own transistors anymore. I mean, like, the entire computing revolution has been that you no longer have to assemble a computer by hand.
Paul: Right. And each layer of the ecosystem allows a vastly larger number of people to take part in it. So I think the thing that we’re seeing now. This has been happening for five or 10 years, but what’s getting really, really big now is people who only know how to front-type program.
People who only know Javascript and HTML/CSS, and they go to boot camps and they come out, they maybe have some, like, vague real understanding.
And they call themselves coders, and we, the real programmers, who know how to write a quick sort, and who know how to write left-pad, we don’t welcome them to our community. And frankly, I mean, I have felt this in the past, I have since changed my mind, but it’s a very, very common perspective.
This podcast is brought to you by Heavybit, a program that helps startups turn products loved by developers into successful businesses. Throughout the program, founders work on developer traction, product market fit and customer development. If you like what you hear, be sure to subscribe to Heavybit’s Podcast Network on iTunes.
