devopsy.kinsta.cloud

Tag: To Be Continuous

  • To Be Continuous Podcast: Category Creation featuring Martin Casado

    To Be Continuous Podcast: Category Creation featuring Martin Casado

    In this episode of “To Be Continuous,” Paul and Edith are joined by Martin Casado, General Partner at Andreessen Horowitz. He previously was co-founder and chief technology officer at Nicira, which was acquired by VMware in 2012. While at VMware, Martin served as senior vice president and general manager of the Networking and Security Business Unit.

    The group discusses the deeply complicated and difficult process of category creation, with a special focus on technology infrastructure products.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/283115632?secret_token=s-zX7AW” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Edith Harbaugh: What do you like about continuous delivery?

    Martin Casado: I think that there’s a broader movement around continuous delivery that is really interesting, which is, traditionally, infrastructure and operations came from an independent group. My first job out of undergraduate was working on simulations code in a national lab.

    I couldn’t control anything about infrastructure. I really didn’t have budget to buy a pencil. You get infrastructure, and you’d have some application to code, and you’d work on that application. We’d fiddle around with our MIG files or whatever, and that was pretty much it.

    The cool thing about the broad movement now, and I think CI/CD (continuous improvement/continuous delivery) is just part of this, is more and more applications, application developers, are actually getting responsibility for operations.

    This is provisioning, this is delivery, this is deployment. That’s a big huge change in not only what technologies get created but who’s consuming it and why they’re consuming it. I think it’s actually one of the more interesting trends right now.

    Edith: This would be a great time for you to introduce yourself.

    Martin: My name is Martin Casado, and I’m a general partner at Andreessen Horowitz.

    Edith: But you did stuff before you were at Andreessen.

    Martin: That’s right. The quick arc, as I mentioned, I used to be a research scientist at Lawrence Livermore National Labs. I spun out of that, did my PhD at Stanford. My research was in networking, so I was part of the team that started software-defined networking.

    I was there for four years, spun out of that, started a company called Nicira where we did software-defined networking. I did that for five years. That was acquired by VMware. I ran the networking security business unit at VMware for almost four years, and four months ago I decided to change and become a VC.

    Edith: So you basically created a category. What drove you to do that? What made you decide to start something knowing that it was so hard?

    Paul Biggar: You’re talking about software-defined networking? For the viewers at home, Martin roughly invented software-defined networking.

    Martin: There’s a whole bunch of people contributing, but I think a lot of the original momentum came from my thesis. You know, I’m going to be totally honest. If I had any idea how hard it was, I never would’ve done it.

    Edith: Really?

    Martin: I think so.

    Edith: I think you would’ve.

    Martin: Well, maybe. But it would’ve been a lot more daunting. I came out of Stanford pretty darn naive, like, “This is going to be easy!” Now, nine years and tons of gray hair later, I just realized that, man, it was a battle. I don’t think I had one-quarter that was easy. Everything was a battle.

    This is the way that I think about it. If you’re an entrepreneur, and you’re going to do category creation, you basically just have two jobs, in addition to product development. And job No. 1 is, there’s going to be some constituency out there you’re selling to, and that constituency, they wake up every morning and they think about all sorts of stuff, but they’re not thinking about your thing, because it hasn’t been created yet.

    That object in the mind space, that object doesn’t even exist in their brains yet. The first thing you have to do is create that object. It’s like Leonardo DiCaprio at “Inception.” You go in, you’re planting an idea in somebody’s head. It has to become a thing, something people think about, something people know and integrate.

    This is from nothing, and that’s hard. For us, it took five-plus years, and it touches every aspect of thought leadership such as analyst relations and technical conferences and broad marketing.

    Paul: So the activities that you’re doing in that category creation are largely marketing and getting people to hear about the activity and learn about it?

    Martin: I think so. I think maybe technologists will view marketing as this pretty shallow thing, but I think you’re exactly right. It’s marketing. But it is often very technical marketing, for example, in the guise of open source.

    In releasing open-source projects, you’re getting people to use it, you’re going to standards bodies, you’re going to conferences. It’s an incredibly technical thing, but in the end, it’s marketing in the sense that you’re actually creating awareness and a market.

    The first thing that you’ve got to do as an entrepreneur is, if you’re doing category creation, nobody’s thinking about your thing. You’ve got to get them to think about your thing, which is a lot of work. In my experience, five to seven years.

    Then the second thing that you have to do, which is just as much work, is then you have to attach a value to that thing. And I think we as technologists, there’s this kind of fallacy that we have where, “I’m going to create this new widget.” Let’s say I create the iPhone.

    Paul: Everyone will know how amazing it’s going to be.

    Martin: Exactly.

    Paul: Just build it and they will come.

    Martin: Yeah, there’s intrinsic value in my widgets. I take my widget, I put it on a sidewalk and someone picks it up. They’re like, “This is amazing. It’s worth millions of dollars!” And that’s totally not the reality.

    The reality is people only see value if they exchange money and they’ve used it.

    And if you’re doing category creation, how do people know what that value is? It’s a very difficult thing.

    Paul: And when you say attaching value, do you mean monetizing? Or is there more to it than that?

    Martin: I meant more the idea. Ultimately, this materializes monetization. Let’s say that you’re my constituency, and I’ve created this new widget. First I need to get you to think that this is actually a real thing. I’m going to call it a new phone. So, you need this new phone. First I need to convince you that you need the new phone.

    Then the second thing I need to convince you is it’s worth tens of thousands of dollars to you because you need to mentally actually attach that value to it. And it turns out introducing new concepts don’t necessarily have value if they’re new concepts.

    Normally, the value comes from, for example, an actual transaction. There’s a psychology around this. Or at least some experience with it, knowing that it’s going to, I don’t think you can just sit back to a couch experiment and be like, “Oh, I think this is going to save me money, therefore it’s worth something.” It really requires a lot of effort to make value.

    Paul: So the category that I see being created, or the one that’s closest to me at least, is this idea of containers and Docker, and more specifically Kubernetes, the idea that there’s this cloud infrastructure over there. The way that I became aware of it is exactly the way you’re talking about it. There’s open source coming out there that you can use and you can hack on yourself. There’s talks up there where people blow your mind.

    Martin: I think that actually this point, and certainly for all the listeners, I think this is one of the most salient points of this decade, actually, which is going to be a weird thing to say. At least an infrastructure’s the following.

    In the past, how did we create mindshare and customer bases? Well, there’s this whole pmm function and this whole sales function. And if you look at the big enterprise companies, if you look at Cisco and IBM and Oracle, really their power is this function. They own the sales channel.

    They have great relationships with the analysts, they’ve got great relationships with the customers, they’ve got certifications and the classes. So anytime they have a new concept, they take that concept and they cram it down the machine.

    As a startup, it doesn’t matter how cool your technology is, the hardest thing is to actually penetrate that channel.

    But when it comes to this new era of open source, now you actually have this hack that’s never existed before, where the buyer is shifting from an operator to a developer. Developers like open source, and you can actually create these concepts by getting open source out there and getting them to use it.

    Paul: We’ve been talking about this sort of thing, just the idea that it’s bottom-up now versus top-down.

    Martin: Not only that, it’s a totally different buyer that’s a totally different aesthetic. I don’t think developers care as much about analysts. I don’t think developers care as much about the 42-long sales dude with white teeth and the boat. I don’t think developers care as much.

    Paul: They won’t even answer his call.

    Martin: Exactly. About the channel, I think they don’t care about complex procurement processes.

    Edith: Having sold software for a little bit myself, if you get into a bigger corporation, you still don’t start to run into that.

    Paul: You get there, because the developers were first asking for it to exist.

    Edith: The longest part of our sale cycle is you get bounced to procurement and legal.

    Martin: Absolutely. I think this is the most important nuance of this argument, is exactly what you said. I’ve always maintained that open source and the developer is replacing marketing and pre-sales, not sales.

    You get credibility to the account, you can get things to technical close and technical validation, you can get account pull. So it’s almost like filling the pipeline, getting the transaction done, I still believe. For a long time, you need to have a professional, direct sales force that will be in there pushing things for procurement.

    Edith: It’s just a lot of work, because you pass the technical evaluation. They’re like, “Okay, go talk to procurement.” And to procurement, you’re a widget.

    Martin: Right, but the good news is, now it’s possible. Where in the past, to get that much legitimacy, to get things to technical close, was very difficult. You didn’t have the certification, you didn’t have access to the buyer, you didn’t have the account rep to make the introduction. You didn’t have any of that.

    Edith: That’s true.

    Martin: And now you do, and now it’s just about actually going through the logistics. And you’re absolutely right, it’s hard, it’s baroque. I’ve called it tribal bloodletting. It’s all of that, but you can do it as a startup.

    Paul: The thing that I think is really interesting is the converse of that. It’s that people who work in open source now are becoming actually incredible marketers. They don’t refer to themselves as such, but they have amazing social media presence.They build up tons of channels, whether it’s mailing lists or just Twitter followers.

    Edith: It’s like everybody recreates marketing. Everybody’s like, “Oh, we don’t need marketing.” Then you’re like, “Well, what is this thing over here where you have an email list?” Oh, that’s not marketing.

    Paul: It’s not marketing because I’m a developer.

    Martin: I really think that those that know how to do this are some of the most valuable people in the entire industry because this is the new entree for products into customer bases. I mean, there’s another kind of potential pitfall here, which is, let’s say you’re using open source for marketing.

    Now, there’s probably $7 billion have been invested in open-source companies and only about a $1 billion worth of returns. I’m not sure if those numbers are true, but I hear them quoted all of the time. But the reality is, a lot of money’s gone into it and not a lot of money has come back.

    The reason is because now, if you’re using open source to do marketing, you’re almost cannibalizing your sales motion. And so for a long time, the only open-source model we knew how to work was the Red Hat model, which is a services company. Only Red Hat’s ever pulled it off. It’s been very difficult.

    But another shift has happened in the last few years is people have figured out that actually you can use open source for marketing, but you don’t have to confuse that with your actual product offering.

    And with the rise of services, you can actually use open source for customer traction. You can monetize the service, and when it comes to services, no discussion of open source or closed source. So I think that we’re actually starting to unlock viable business models where you use open source for marketing.

    Edith: Who do you think does a good job of that?

    Martin: I think GitHub is a great example, right? Git isn’t even built by GitHub. It’s the standard in Linux. But if you look at GitHub, it’s a very fast-growing company that’s based on open source, and so everybody knows about it because of Git. So Git is doing the marketing. And this is just one example, I think there are many.

    Paul: It’s funny that you cite that as based in open source, because almost none of their stack is actually open source.

    Martin: Exactly, it’s a service offering. That’s what I’m saying.

    Paul: When you say service, you’re referring to software as a service, not service as in consulting?

    Martin: Absolutely. I should’ve been more clear. The long-standing question for us, certainly since 1995, is how do you build an open-source company that has the margins of a closed-source software company? Because you’re open source, basically you become a service and support company.

    What has happened in the last decade is exactly what you’re saying: people will realize that you can get account credibility and account traction, account knowledge, through open source.But then you can offer a back-end service offering as a service offering, and monetize that, and it’s often a closed-source stack on the back end of what you’re doing.

    But you don’t talk about open source/closed source as much with service offerings. It’s normally if you’re consuming something on-prem that you think about those things.

    Paul: How do you feel about this? People who are building, let’s say CircleCI as a good model, but there’s a ton of them. Building something that’s a SaaS product, a hosted SaaS product, but that then sells to people who use it in their own VPC and pseudo on-prem sort of thing.

    Edith: Oh gosh.

    Martin: I come from this somewhat traditional world of selling enterprise software. In there, there’s this whole discussion, and I was part of this discussion, of is it on-prem or is it off-prem in general? And I think that you framed this question perfectly. I’ve realized that that’s a false dichotomy. It’s actually not the right question to ask. I think as a service, is it actually an operational model and a delivery model? It doesn’t really matter if it’s on-prem or if it’s off-prem.

    I think that the days of selling closed-source software to companies is done.

    But you can sell them services and you actually can sell them appliances. But if you look at appliances, appliances almost look like services. They’re boxes that have services, and often the upgrade is happening remotely, management is happening remotely.

    I do think that the right model for these companies, I think CircleCI’s a great example of this, you have open source that customers use, you have a SaaS offering, but you also have an on-prem delivery offering as long as you don’t change the operating model, as long as it still looks like a SaaS model somehow, whether it’s a managed service or something like that.

    I think as soon as it becomes shippable software, where you have to manage a heterogeneous set of environments, you have to manage all the day-two ops. It becomes unwieldy.

    Edith: I had a really interesting discussion the other day with the VP of Sales from CircleCI and somebody from Microsoft. CircleCI was basically like, “We love that it’s selling on-prem. It’s helping you.” Microsoft is like, “We’re trying to get off-prem desperately.” So it was not the argument you thought was going to happen.

    Paul: I think it’s probably because of how sensitive the IP is. If you’re talking about something like, well, documents are pretty sensitive, but the source code is very sensitive. And so people are not as happy to ship that off-prem as we once thought was going to be the case.

    Edith: Microsoft’s reasoning was, and this might be the way Microsoft had done it, is they have 10-year maintenance agreements.

    Martin: We have, “If you have a problem, upgrade to the latest version and then we’ll deal with you.” I think that’s kind of what everyone has. There used to be this idea that you’d create a fork and then you’d backport things to that fork or the old version, or something like that.

    I don’t know anyone who’s doing this sort of services, the SaaS with a managed to on-prem version that does anything but “the latest version is the version.”

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market. Throughout the program, founders work on developer traction, product market fit and customer development. If you like what you hear, be sure to subscribe to Heavybit’s Podcast Network on iTunes.

    — Malia Powers

  • To Be Continuous Podcast: Have We Forgotten How To Code?

    To Be Continuous Podcast: Have We Forgotten How To Code?

    Welcome to To Be Continuous, a show about continuous delivery and software development, hosted by Heavybit members Paul Biggar, founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly.

    In this episode, Paul and Edith discuss the fallout after the widely covered “left-pad” incident, and while they agree that these sorts of moments move the entire open source community forward, they wonder: Have we forgotten how to code?

    [soundcloud url=”https://api.soundcloud.com/tracks/264605904?secret_token=s-Kv9Ep” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Edith: Hey, Paul. So, somebody unpublished a module called left-pad and the internet basically exploded. And now it’s been a couple weeks since then, so there’s a lot of different topics to cover in terms of what this means for open source, whether it means that people have forgotten how to program.

    Paul: So LaunchDarkly. Or, I’m sorry, the left-pad. I’m obviously looking at my own t-shirt.

    Edith: You’re looking at your LaunchDarkly t-shirt?

    Paul: The left-pad incident was pretty interesting and one of the things that we did when coming up to this was just, like, looking at all the Hacker News discussions and comments on it.

    And there’s wildly varying positions on what this means for open source, what it means for whether people can program, what it means for, like, about the Node ecosystem. So to kick this off, why don’t you give us a recap of the entire incident?

    Edith: Yeah. So, in short, a programmer had a module called Kik, K-I-K. Which is also [the name of] a very popular messaging app. I mean, even I’ve heard of it and I’m not a teenager.

    Paul: The Kik lawyers contacted the dude. Said, “Please take down your module, change it.” And the programmer said, “No. I basically don’t believe in copyright laws.” So then, the Kik people contacted NPM. The NPM people took down the module called Kik. And the guy, in I’d say, kind of a fit of pique, said, “Okay, you took down my one module, I’m gonna take down all my modules.” Basically, I’m taking my marbles and going home.

    So, he took down all of his modules. One of which turned out to be an incredibly popular module called left-pad, which did basically what the name implied. It sorted left pads. And without this in NPM, the quote I read was, “Builds all over the world started to fail very quickly.”

    Paul: So as I understand it, left-pad was used by another module, which was used by another module, which was used by something core, or like, this incredibly popular module—

    Edith: Yeah, it was basically turtles on top of turtles on top of turtles, and at the bottom turtle was this guy who—

    Paul: Took his marbles and went home.

    Edith: Yeah, or yank the bottom Jenga, basically.

    Paul: Gotcha. And so all the builds started to fail. People’s deploy started to fail. And people got pissed.

    Edith: People got pissed because they didn’t even realize that they had this dependency. So what happened was NPM basically republished it. They said that, even though you’re the original publisher—

    Paul: So there was a lot of outcry on the internet and then they republished it.

    Edith: Just basically said, “You don’t have the right to un-publish this.”

    Paul: Gotcha. So, um, but they had an Unpublish button, as I understand it. So he did something that was built into NPM to remove a module.

    Edith: Yeah, and since then they have tightened their rules a lot. They basically said that this should be something that’s used extremely rarely, if at all, and not something you can just do capriciously.

    Paul: So one thing that’s interesting about the NPM ecosystem, and the Node ecosystem in general, is they’ve had a bunch of things like this. Things that are security-related, or that are drama-related.

    And people have repeatedly come out and say, “Oh, no, JS is a tar pit, or ghetto,” or whatever it is that people use to describe poor ecosystems, that normal programmers don’t know how to code, and so on, so on.

    And what it looks to me like is, you know, each of these incidences brings the Node ecosystem quite a bit forward. To the point that, I mean Node is relatively new, it hasn’t had 15 years before prime time in the way that Ruby or Python did, or 20 years before it got big, and it’s a real, kind of bottom-up thing.

    I mean, if someone built it and there were some steroids and then someone else took over, and then NPM took over, and then. It’s this sort of weird language from the ecosystem perspective. It didn’t have the same advantages that, like, Java had, or that C or C++ had, in order to get to maturity. And so each of these incidences propels it closer to maturity.

    Edith: Yeah, I compare it somewhat to when electricity first started coming out. And then you were just working through all the issues that could happen. Like, for example, when a power line went down or a transformer blew.

    Like, you work through, Okay, what are our points of failure? Let’s not make that a point of failure again.

    Paul: Hmm. So, NPM changed their policies on this basis, right?

    Edith: Yep, so now you can’t have an individual contributor be able to unpublish, even if they originally published. Now you have to contact them, and it’s much more stringent.

    Paul: Gotcha. The response, despite people being … Well, people were annoyed at their response.

    Edith: It seems perfectly rational to me.

    Paul: Oh, I think it seems perfectly rational. This is the thing about the in trend. There’s always some set of people who are annoyed at everything you do.

    Edith: Well you know, the internet is full of billions of people. I mean, there’s gotta be some sort of standard deviation, or otherwise it would be a very boring internet.

    Paul: Right, right, and this is why there’s always haters on Hacker News. No matter what you do. If you do it one way, there’ll be one group of haters, if you do it another way, there’ll be a different group of haters.

    And so, with left-pad, there were people who were annoyed that NPM retroactively decided that they had ownership over their namespace, which previously, they had sort of gifted to people in the community, the first people to take the namespace.

    Edith: You mean the original thing about Kik?

    Paul: So there’s that, but there’s also the left-pad. They took away the sort of autonomy of the left-pad offer.

    Edith: I read a good article that basically says part of open source is, if it truly is open source, do you maintain the rights to it? Or is it a community project?

    Paul: Hmm. The traditional way of deciding that sort of ownership was the fork. That you could go, and you could fork, and when you forked, you weren’t necessarily entitled to the same name, you weren’t entitled to the same trademarks. When Debian forked Firefox, they called it Iceweasel.

    When Jenkins was forked, or Hudson was forked, it became Jenkins. GCC got forked. They all got new names. But what was the fork here, it wasn’t even a fork. It was a republish of the same thing, the same software, in a namespace that the original author didn’t control.

    So typically you’d own the URL, you know, if you have a software library, you’d own the URL. And they did own, in some sense, the URL within NPM, or the name within NPM’s space, and NPM, the sort of owners of that namespace, overrode it.

    Edith: I think they did what was necessary for good order.

    Paul: Sure. I completely agree, I think that was a good decision.

    Edith: I mean, so to take this to the logical conclusion, what if you had a module named left-pad, and you decided that, like, as a prank you were going rewrite it to be right-pad? Like, sure, that’s … Is that within your rights? Kind of, but it’s a real jerk thing to do that you probably want to prevent.

    Paul: This is kind of one of the dangers of the whole ecosystem around centralized package managers, of sorta trusted package managers, where there’s no actual trust involved. So, what I mean by that is what is published does not have any gatekeepers.

    Edith: Yeah, exactly.

    Paul: I mean, it’s great, it allows an ecosystem to grow much faster, but it has safety concerns because people generally don’t pin their packages to an exact version.

    Edith: You’re basically pinning it to a floating value.

    Paul: You’re pinning it to a floating value. And that floating value could change. There’s no guarantee of what a minor number means or what a major number means or what a patch number means or what a patch even means. There’s some sort of, like, general community guidelines in semantic versioning, but there’s no enforcement of that.

    Edith: Yeah, and I mean this basically goes back to the whole Cathedral versus Bazaar thing.

    Paul: Right, right. It’s very Bazaar. And intentionally so. It’s the thing that allows a community to grow, and is there a point at which open source communities need to, like, batten down the hatches to prevent that sort of behavior? Or prevent that sort of risk?

    Edith: Yeah, and it’s really interesting. I think this goes back to something we’ve talked about with Nadia Eghbal and Sean Byrnes a couple episodes ago about when you buy from a corporation, you have some sort of guarantee of standards that they publish somewhere. You know, this is our guarantees versus with open source, you don’t normally have those same sort of guarantees.

    Paul: Even when you do, it tends to be—

    Edith: There’s no enforcement mechanism.

    Paul: Right. So there’s no enforcement mechanism in the Bazaar.

    Edith: Well, there’s no, um … Like, so with a company, if a company has said, “Okay, we’re gonna release twice a year and it’s gonna have this level of quality,” and they break that, you can walk with your feet or your money. If an open source project stops or does something that you don’t like, there’s very little enforcement end.

    Paul: Theoretically, you could walk. You know, you could fork the thing, but when you end up in an ecosystem that has massive network effects, and in the way that NPM does, you can’t walk with it. Because the pends is all transitive. The left-pad was, you know, a turtle way at the bottom of the stack.

    Edith: It was almost like a turtle hatchling.

    Paul: So, to fork left-pad, you would have to fork the entire NPM ecosystem.

    Edith: Yeah, and it’s not even. The fork is not the issue, though. The issue was that people were depending on it being in a certain place in a certain way. It wasn’t that it wasn’t forked, it was just like this entire structure at the bottom was gone.

    Paul: I guess what I’m saying is that if you get pissed off with a certain set of developers or a certain set of practices within the NPM community, because they’re so interlinked, you don’t have the traditional open source option of forking.

    Edith: Absolutely. Absolutely.

    Paul: You could rename it. You could pick a different library, but because of the way—I’m not sure if they still do this, ’cause I read something about it, but in NPM, it used to be that the version of the software that your library took was the version that library got. Do you know what I mean by that, okay. So let’s say there’s some software, let’s call it left-pad.

    Edith: Let’s call it right-pad.

    Paul: Right-pad. So, right-pad version one is being used by library A. And right-pad version two is being used by library B. In the Java ecosystem, and many, many other ecosystems, only one version of right-pad would be used.

    And I believe in NPM, both versions of right-pad would be used. So library A will use version one, and library B will use version two. Now that’s definitely the way that it used to be, I’m not sure if they’ve actually changed that.

    Edith: Yeah, I don’t know, but that sounds very messy, very quickly.

    Paul: So both options are terrible. And there isn’t a right way to do it.

    So, really, what it depends on. So, the NPM way of doing it, every library get its own version of the dependencies, is fantastic. Because it lets that library do what it actually intended to do under the covers.

    If, however, you start having an object that gets sent around, or that two modules are, let’s say it receives a, I think, an HTTP packet or something like that would be a good example. Let’s say you take an HTTP packet from one module and just hand it to another module, and they have different understandings of the shape of that object, or what functions you can run on that object, then that’s messy.

    But, on the other hand, if you override the version of right-pad that some other library’s using, you end up with all sorts of problems. This is a major problem in the, sort of the Maven ecosystem. And at CircleCI, we used Clojure, and Clojure is built on top of the Maven ecosystem, and we would have bugs where the library that we used.

    One library that we used had a dependency, and that was in the package file for another thing. And so, that’s the version of the packets that we used. So we would have to pick. And as a result of that, we have to pick the library that works. And, often, there isn’t a library that works for both, or there isn’t a, sort of subdependency that worked for both. So those are close to forked no matter which way you do it.

    Edith: Yeah, I mean, we ran into the same issues with our STKs. I mean, ’cause we have dependencies on other libraries that sometimes conflict.

    Paul: Right, disaster.

    Edith: On the other hand, I mean, the bigger point, I think, was. So there was an interesting backlash, I think, that everybody kinda projected what they wanted on the NPM thing.

    People said, Oh, people don’t know how to program anymore. You know, why do you even use what they called left-pad? You know, why don’t you just grind your own wheat and make your own bread from scratch and eat it at lunch also?

    Paul: I have no sympathy at all for this view of the world.

    Edith: Uh, that you should make everything from scratch? Or—

    Paul: That, “Oh, we’ve forgotten how to program because we need left-pad.”

    Edith: Yeah. The fact that we have modules available that make it easier and easier and easier to program is a good thing.

    Paul: I think it’s kinda different to that. I think it’s more of a curmudgeon view of the world. It’s a get-off-my-lawn—

    Edith: Wait, that you’re the curmudgeon?

    Paul: No, the people who complain, “Have we forgotten to program anymore?” I think it’s a no true Scotsman kind of argument. Real programmers know how to write their own left-pad, real programmers write in statically typed languages, real programmers write in C, real programmers, and I’m going back in time here, real programmers write in Assembly, real programmers hand-code their. Tweak the needles on the.

    Edith: They do their own garbage collection.

    Paul: Oh, and it’s garbage collection, what is this? Garbage collection is for amateur coders, right? I mean, you could just as easily have written, have we forgotten how to manually allocate memory?

    Edith: Or, you might as well say, like, nobody knows how to make their own transistors anymore. I mean, like, the entire computing revolution has been that you no longer have to assemble a computer by hand.

    Paul: Right. And each layer of the ecosystem allows a vastly larger number of people to take part in it. So I think the thing that we’re seeing now. This has been happening for five or 10 years, but what’s getting really, really big now is people who only know how to front-type program.

    People who only know Javascript and HTML/CSS, and they go to boot camps and they come out, they maybe have some, like, vague real understanding.

    And they call themselves coders, and we, the real programmers, who know how to write a quick sort, and who know how to write left-pad, we don’t welcome them to our community. And frankly, I mean, I have felt this in the past, I have since changed my mind, but it’s a very, very common perspective.

    This podcast is brought to you by Heavybit, a program that helps startups turn products loved by developers into successful businesses. Throughout the program, founders work on developer traction, product market fit and customer development. If you like what you hear, be sure to subscribe to Heavybit’s Podcast Network on iTunes.

  • To Be Continuous Podcast: Staging Servers Must Die

    To Be Continuous Podcast: Staging Servers Must Die

    Welcome to To Be Continuous,  a show about continuous delivery and software development hosted by Paul Biggar, founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this episode, Edith and Paul discuss a blog post by Edith, in which she asserts that you should kill your staging servers so that continuous delivery can live.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/248544923?secret_token=s-fUn5T” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

     

    Paul: So today we’re gonna talk about a post that Edith wrote. The post is about the death of staging servers. Can you give us a little bit of an introduction to what the post is, because I think we’re gonna talk about it quite a bit.

    Edith: Sure Paul, happy to. By the way, did you read it yourself?

    Paul: I confess, I may have skimmed it, a little more than read it.

    Edith: Yeah, so I’ll give you the summary. And then we’ll talk about it. And then you should go read it, and then we’ll talk about it again.

    So the article’s an outgrowth from what I was seeing with LaunchDarkly, my company. So already one of the comments I saw on it is, and I like this comment because they talked about me in the third person. They said, “Harbaugh is biased.”

    Paul: Right, I would say so.

    Edith: But by the way, she does have a good point. So I’ll state upfront, yes, I’m very biased. My company makes a platform to manage feature flags. So my bias, though, is I’ve seen what our customers are doing.

    And it’s the reason I wrote the article, is that this was a national extension to the way I saw our customers using feature flagging.

    Paul: So were your customers using your product wrong, or was there a thing that was happening, that you’re talking about?

    Edith: So feature flagging at it’s most basic is very simple. It’s just an if-then statement. You don’t need LaunchDarkly or any sort of system to manage that, you can just put a conditional in your code. What happens after that is you get more sophisticated. And you wanna have some sort of dashboard where you could see the different feature flags that are visible and uplevel them. Not even just to the business side, but even amongst the various developers.

    You want to have a central place where you can manage. That was kind of our first version, is just you can manage feature flags.

    The next thing that our customers were asking for in our road map was to support for environments. They wanted to be able to have feature flags on dev, QA, staging production, have visibility all in one place.

    Paul: I’m a little confused when you say this, because surely an environment is just a flag?

    Edith: Yes, go on.

    Paul: It seems like you could say, have another flag that is a string, that is like staging or production or Dev or whatever. That you flag features on.

    Edith: And that’s basically what LaunchDarkly is doing, so for us, we just have another flag.

    So each environment gets an API key.

    Paul: So again, forgive ignorance on this, why not use the existing flag?

    What was missing from the existing feature flag infrastructure that meant environments had to be a separate top level, or first class feature?

    Edith: So now we’re getting a little meta. The reason why people liked using LaunchDarkly is because they can get a consolidated view of all their feature flags.

    So that they could not just have these floating around fig file for each machine, but have a roll up of what flags are turned on and off for different environments. And then on top of that the ability to manage the flags in different environments.

    Paul: Got you, so they wanted a view that applied to just their environments?

    Edith: Well, to see per environment what was turned on and off. So basically they were following a software life cycle, where you had dev people working on the developer boxes, pushing a QA, pushing a staging, and then pushing to productions.

    Paul: When features moved from one to the other, was there a human promoting these, or was there an API call promoting these?

    Edith: At this point it’s mainly human. I could see down the line that it would just be an API. And even beyond that … The next thing people ask for, once they got more people on board LaunchDarkly, we’re using more feature flags, is they wanted the ability to have lockdown of different flags in different environments.

    Paul: Lockdown like people can’t change the flags?

    Edith: So they wanted QA to have rights, and QA to change some flags, but not in production.

    Paul: I see, okay yeah.

    Edith: Then after that the natural step then is okay, if you have a feature flag in system where you can control any step, who sees permissions?

    Who gets to see what? Why do you really have a separate QA staging and production box? Why not just collapse all this, and manage visibility with a feature flag itself?

    Paul: Sure. That makes perfect sense.

    All the feature flags provide the primitives on which you can build the things that they want.

    Edith: The point I made in the article is that

    people use the substraction of a QA and a staging environment to basically try to encapsulate a change. So that was the original intent, and at the time it was very good, because the alternative was just to push everything to production to have everything break.

    But if you’re actually doing—

    Paul: If you’re doing feature flags properly, then the concept of really having a staging server doesn’t make that much sense.

    Because you’re not pushing a feature to the staging server, you’re pushing a feature to production and then slow rolling it to people and really having it in the staging server doesn’t make that much sense. Is that kind of the point?

    Edith: Yeah, so that was the title of the article, Kill the Staging Server.

    Paul: This might be a good point to actually stop and read the article.

    Edith: Yeah, I’d appreciate that. Thanks, Paul.

    Paul: We’re gonna take a, what’s gonna appear to be a 10-second break, but it’s actually going to be a five-minute break, while I read the article. I recommend you press pause and read the article yourselves right now.

    Edith:  So welcome back, Paul has been furiously whiteboarding. I’d love to hear his thoughts now.

    Paul: Yeah, so it’s a good article, I really liked it. The first thing that came to mind was the concept of names.

    So the idea of having multiple environments is a weird one in particular. So imagine that you have … Service rented architecture, and it’s got like six machines or something like that, or it’s got 20 machines or whatever. The idea that you can’t spin up a new instance of it is kind of a little bit odd.

    Edith: Yeah.

    Paul: So what people have done historically, and this goes back to when things ran on machines, or particular ports, or whatever, was that everything had a name. It had DNS name or had a staging name.

    And if you look in rails, configuration files, there’s a production name, there’s a staging name, there’s a dev name, there’s different environment variables for all of these. But it doesn’t actually make sense to have a name. Because a name implies there is one of them.

    Edith: Well this goes back, are you going down the whole pets vs. cattle thing?

    Paul: I’m going down the pets vs. cattle thing, yeah. If you’re naming your pets, then you can’t just suddenly get six of them, you can’t suddenly kill them. But what you really want is cattle.

    So a staging server is not, I mean a staging server is a pet. And a very very important pet that everybody loves, and everyone plays with, and it gets a little bit confused as a result of it. And that analogy went—

    Edith: I’m not sure if you’re agreeing with me, or disagreeing with me, so. We’ll continue, it’d be great if you disagreed, but I would also enjoy it if you agreed.

    Paul: If that analogy holds, then what you’re suggesting is that we killed the cherished family pet.

    Edith: Well, you know, sometimes … aw, I can’t even go there.

    Paul: Well the analogy breaks down, but let’s say it’s for the best if that pet spends some time on the farm.

    Edith: Went for a long walk.

    Paul: Right. So obviously production is an actual thing that needs a name, it is a unique environment. But nothing else is really unique environment.

    Edith: Yeah, and to fast forward, I think, I got a lot of feedback on the article, which I loved. That’s kind of why I wrote it, for feedback. I think there are cases where you do want to not go directly from a developer to production.

    I think there are many cases where you want to have other places to test them.

    Paul: I do agree with this, yes.

    Edith: I think a lot of the cases though, of a forced march of we go from this step to this step to this step to production is actually very harmful, when if you just pushed off much quicker and perhaps skipped some of these steps, you’d get the feedback you want directly.

    Paul: So what we did at CircleCI, and initially we had a staging environment. And we would occasionally use the staging environment if we want to test something that wasn’t that easy to write unitests for, or something along those lines. And usually for us that was stuff around LXC, or stuff around starting Amazon boxes.

    Things that you didn’t really do that often, and that either had an expense, or had a weird architectural thing, where you couldn’t just do it in software, and practice it in software. So I think there is that need. Occasionally you’ll have things where it’s not tested well enough, so you need to put it up somewhere where a human validates to the best of their knowledge that it actually works.

    And you can put that in your, I was gonna say you could run it on, but the whole point of the staging server in that situation is that it’s something which isn’t really that easy to put into VMs, or whatever. The other kind of use case that you see is where you don’t want to be running stuff on live production databases, and you can’t get a copy too quickly. So you see people like Haroku trying to build products that avoid the need for that.

    So with Haroku … You can take a copy of a database, you can have a view or a read only view on a database, that’s a copy on the right in some way or whatever. So that you don’t need a separate staging environment, or separate copy or separate staging database or whatever.

    But if everything is well-tested, if everything works in software, then there’s really no need for staging environments. So that says to me that staging environments is a sort of a yellow flag somewhere. It shouldn’t really exist, but sometimes you might need it.

    Edith: I actually want to write a follow up article now, of the staging server is dead, long live the staging server. Because I do think that there are used cases where you don’t want to push to prod. I do however think people—

    Paul: What are these cases where you don’t want to push to prod?

    Edith: So what I heard from Sean Burns, our advisor, is if you’re testing a really deep infrastructure change, for example switching some batch processing.

    Paul: Mm, no, I don’t believe that for a second.

    Edith: Alright.

    Paul: So if you’re testing a really deep infrastructural change, there’s two ways to do that. That deep infrastructural change. One of them is to say, “We’re gonna have these machines over here, we’re gonna have these machines over here.” And that’s the quarterly release cycle thing. We’re taking a big big risk, all hands on deck, whatever.

    The way that you want to be releasing that sort of thing is that you wanna have it in the code base. You wanna have an if statement, a feature flag, that controls how much of the data goes one way or the other, and you duplicate the data, or you put 1 percent of it through.

    Edith: I think the key there is to duplicate or have some fail safe. I mean the biggest risk you run with doing this is data loss, which is awful. If you’re cavalier about how you do this—

    Paul: Well, I don’t think data loss is the worst thing, the worst thing is your whole damn service goes down.

    Edith: Well, the worst thing is your whole service goes down, and you lose a case worth of somebody’s data.

    I think people are actually more forgiving of a five-minute flip than you losing like a lot of their analytics. So his point was he had been at Flurry, was that we cannot afford to lose people’s data.

    Paul: So it would be ludicrous then, in my mind, to create a whole brand new infrastructural thing where you’re gonna do some kind of overnight or immediate change, or wait for downtime change or something. If the data is all live, you can’t afford to have, we’re gonna switch over and see if it works. Regardless of how well it’s tested, like it’s ludicrous.

    Edith: Yeah, so that was, now I’m coming back around to my article. I mean that was the point I was making.

    People think that they are reducing risks by doing all this tested staging.

    Paul: Yeah, in fact they’re increasing risk.

    Edith: Yeah, you would think you’re—

    Paul: Well, the whole point of continuous delivery is that by having a harsh cross over between one thing and another thing, you increase risk even though you think you’re decreasing risk.

    Edith: Yeah, so Ket Beck wrote a really good article, about reversibility. So he’s at Facebook now, but he said everything at Facebook is reversible.

    Paul: Interesting.

    Edith: That this actually makes you much less risky, ’cause you’re like okay we make these risky changes—

    Paul: But they’re reversible, so it’s, right.

    Edith: Yeah, vs. the more cutover. As I called in the article waterfall deployments.

    Paul: I mean I think that’s a really good way of thinking about it. It ties it to a name that everybody knows is bad.

    Edith: Well it was a deliberate—

    Paul: Yeah, good choice. So the agile deployments then, are the ones where it happens seamlessly, and you can go back and forth and change the requirements and whatever else.

    Edith: So that was Sean’s example No. 1, was what if it’s a really risky back and change? He had some good examples from his own career.

    Paul: Did he have any examples that I would agree with?

    Edith: Well a priori without hearing them, I don’t know if you’re going to agree or disagree. Another thing people brought up, was it just seems very risky to people. It seems to increase risk because I think they’re used to thinking of staging server as a safe harbor.

    Paul: I think there’s some semantics around the use of staging server. You very often want to have a complete copy of your environment that you can test against. So is that a staging server, or is that where you type docker up, you know new environment.

    And then you run your testing on this whole brand new infrastructure that has never been touched by anything before?

    Edith: Yeah, and the critique I made in the article, is I think people do that a lot. And they spend a lot of energy testing there, but these are artificial test pieces.

    Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit.

    You can find the full episode of this podcast here. For more on Continuous Delivery and to hear other Heavybit podcasts, copy this link into your favorite podcast player. We’ll have show-specific RSS and iTunes subscription options soon. You can also follow To Be Continuous on Twitter at @continuouscast.

  • To Be Continuous Podcast: The Clash in Open Source

    To Be Continuous Podcast: The Clash in Open Source

    Welcome to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this episode, Edith and Paul talk about the clash in OSS between developers looking to get paid and software companies looking for things for free.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/244279037?secret_token=s-oeBCP” params=”auto_play=false&hide_related=true&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=’166′ iframe=”true” /]

    Edith: So Paul, what do you like about open source?

    Paul: So I guess that depends on whether we’re talking about writing open source or using open sources as part of a company. Writing open source, I like just having all my code out there.

    And I’ll do that as often as possible. Anytime I start a little bit of code, a little library, I’m looking to open-source it and have a readme and have it out there for other people to use. I had a very long discussion, which I eventually won, about the …

    Edith: I like the way you think of all discussions as win or loss.

    Paul: Well, yeah, yeah. It was more on the argumentative side of discussion. But with the other co-authors of the PHP compiler I wrote, which was originally GPL. And I argued very strongly that I actually don’t care if people give back, I just want people using my software. So to me, open source is a way of having people using my software.

    Edith: That’s really interesting. So, I liked what you said at the beginning about, like, you liked open source ’cause it meant getting stuff out sooner. ‘Cause I’m a big fan of that.

    Like, I always say you should just publish stuff. Like, my co-founder John said something really good at our team meeting last week. He’s like, if it’s 90 percent done and you don’t publish it, it’s actually 0 percent done.

    Paul: It’s 0 percent done, yeah.

    Edith: Yeah, and that’s a hard lesson, ’cause you always try for perfection lately.

    Paul: Right, right, right. Well, I think there’s also some really hard downsides about that. So when you publish something before it’s ready, before you really —

    Edith: Well, ready.

    Paul: Well, there’s ready ready, but let’s say ready is, let’s say you publish it before you clarify what the software is to do, or the values or the mission of the software.

    And then you get contributors who are saying, ‘Oh, this is great, but I really want it to do this other thing,’ and you’re like, ‘Well, that’s not exactly what I was going for here.’

    Edith: Well, I’d say most open source projects are so lucky as to get contributors at all.

    Paul: True, true, true. So I’ll give you an example. I wrote the V8 bindings for Closure. So a binding that allows you to write JavaScript and run it in Closure. And I did that for a purely selfish thing, for another, for a nascent pipeline that I was building called Stefan.

    And then people came along and they wanted to do other things with it. They’re like, ‘Okay, let’s add a binding for this other thing.’ I was like—

    Edith: Well, that’s kind of like a pivot.

    Paul: It is, it is, but I really was not interested in it.

    I’d rather someone took it over, took over the project and actually ran with it where they wanted to go, and I can stick on the old version, or I can see if the new version solves it for me, but I wasn’t interested in becoming a maintainer in this thing, I was interested in solving my itch.

    And I just happened to open-source it, because I felt it might be useful for other people.

    Edith: So Paul, I hate to say this, but I agree with you.

    Paul: No, not again.

    Edith: Not again.  No, I mean, I think what you said is exactly right, that you said, I did this for this reason, and I want somebody else to take it over now.

    I think where open source runs into issues where people are like, it’s only this purpose and you can’t do anything with it besides my original thing.

    Paul: So you think that’s a good thing, or you think that’s a problem? Or it causes problems in the community?

    Edith: I think open source is so misused for so many different purposes that it’s hard to even say. Like, so, in Europe, people say open source is good because it means free software.

    Paul: Libre or gratuit?

    Edith: I’m sorry?

    Paul: Free as in beer or free as in, what’s the other side of that?

    Edith: There’s no such thing as a free lunch, is that what you’re talking about?

    Paul: No no no, like people say open source software, it’s like free-as-in-beer software, as in software that has no cost vs. software that has no price tag vs. software that is, they use the French word libre for—

    Edith: Oh yeah, that’s what I was talking about. So I remember I went to Europe in, I think it was like 2004 or 2005, and they just had this big push for open source, ’cause they thought it meant free.

    Paul: Oh, this is like the German government and that sort of thing.

    Edith:  So literally like, so I went to this customer and they’re like, ‘So it doesn’t cost you any more to give us an extra copy.’ So why shouldn’t just everything be free?

    Paul: Oh, right, yeah, well.

    Edith: And so I think open source has been misused by many businesses who think of it as just like, hey, we get all these developers working for free.

    Paul: So I think that the main advantage to business of open source is not that it’s free, but it’s that it’s low-risk.

    Edith: Low-risk?

    Paul: Lower risk. So it removes—

    Edith: Ah, I would argue—

    Paul: Well, so, what I mean is that it removes a certain element of risk from it. So there’s, Joss Belsky wrote a blog post about how you shouldn’t use, I think it was a database that you didn’t have the source code to.

    Edith: Well, so I’ll argue the reverse, and I think open source is sometimes more risky. ‘Cause I talk to people now who are like, nobody’s maintained this open source project that I depend on.

    Paul: Right, right, right. But that vs. a closed-source version.

    Edith: Well, so the opposite risk is that you end up that you are responsible for this open source project, and you’re like, well, actually, I just wanted to use this thing. I didn’t want to do the maintaining.

    Paul: So absolutely, you don’t wanna end up the only person that, well, it’s actually fine to end up the only person which is using this thing.

    Edith: Well, no, it’s not—

    Paul: It’s not ideal.

    Edith: It’s that many people are using it, but nobody is paying money, and then you’re stuck.

    Paul: Because no one is maintaining it.

    Edith: No one is maintaining it, and you depend on it.

    Paul: Right, so, but imagine that no one maintained it and all that you could get was like a binary block. That’s the real problem. The company goes out of business. You have random database version seven, the binary, that only runs on 32-bit, that you’re maintaining, like, companies are literally maintaining COBOL—

    Fortran things from 30 years ago. It sits in a machine, you can’t touch it, you can’t fix it, you can’t apply security updates to it.

    Edith: Yeah, that’s fair. So Tim Chou, who is, he’s an awesome guy, he was early Oracle, he said that people don’t buy, when they buy software, they’re not buying it what it is today.

    Paul: Right.

    Edith: ‘Cause basically, anything that you’re buying today, you could just hire 20 engineers that could build it. What they’re buying—

    Paul: Yeah, you could just copy it.

    Edith: Yeah, what they’re buying is the promise of stability, that it will continue to be maintained. That there is a vision. And that’s actually, I went over to a big customer who won’t let us use their name yet, but they said they love LaunchDarkly because we thought every day about feature flags.

    Paul: Right, right, exactly. This is why people buy SAS, because, because people think all day about CI, about QA as a service, about queues, or whatever it is from various Heavybit companies that I just shilled for, they’re, you can probably figure out yourself which ones they are by looking at the homepage.

    Edith: Well, and Paul, by the way, thank you very much for wearing your LaunchDarkly t-shirt today. It looks very handsome.

    Paul: The, I definitely agree that the benefit of the thing is that you use it every day. But you can do that with open source as well. You can have open source software that, so, for example, our front end is open-sourced.

    Edith: Oh, I’m absolutely a fan of open source. Like, we use open source every day. Our client libraries are open source. I’m not saying that it’s, I don’t think it’s a religious thing vs. open source vs. not, I just think that many times people assume open source is something that it’s not.

    Paul: Yeah, fair enough, fair enough.

    Edith: I mean that if you have open source, that just means that developers spring out of the ground willing and happy to update your software every day.

    Paul: A mixed blessing, definitely. If it even happens.

    Edith: Like, I remember, so I remember when Sun, so I used to work on a, I used to be at Epicentric, and we had a portal server, and we were killing Sun. Absolutely killing Sun. And this was in 2003, so what they said is, ‘Oh, we’re gonna make our portal server open source.’

    Paul: Right.

    Edith: And so somehow this whole thing, we’re like, everything will be open source, and like, developers will spring out of the ground, and it’s like, well, no, developers, they don’t wake up every day like, ‘I’m gonna work on your thing.’

    Paul: Right, right. They will if it’s sexy or exciting. There’s lots of developers who are showing up at Mozilla just to contribute. And in fact, they’ve fought through incredibly large hurdles to contribute to Mozilla’s software.

    Edith: Well, that’s fascinating, ’cause I see so often the reverse, where people, so what do you think about Mozilla made people want to contribute?

    Paul: I think the fact that it was an open-source-first company, and that it was a nonprofit, and it was all about the free and the open web and that sort of thing. Also, people had had very good experiences with Firefox for, like, a decade at this point by the time I was there. Well, maybe seven years, but, you know.

    Edith: It’s fascinating, ’cause I went to dinner last night with the CEO of MySQL, and I don’t know if I should say this, but he said—

    Paul: This is Marten Mickos?

    Edith: Yep. So he gave a talk—

    Paul: He has a new startup?

    Edith: Called HackerOne?

    Paul: Oh, he joined HackerOne.

    Edith: Yeah, he’s the new CEO. So the dinner was kind of like a debutante ball for him.

    Paul: Nice, nice.

    Edith: So what he said was that somebody had asked him flat out how much of MySQL was actually from contributors. He said 1 percent.

    Paul: Seems about right.

    Edith: He said the open source was more just a branding thing. Like it wasn’t actually like a productivity thing. It was more just a branding thing.

    Paul: So you have this in Mozilla as well, that any time that there’s a really avid contributor, that Mozilla would hire them.

    Edith: Yeah, that’s what he said.

    Paul: Right, and if Mozilla didn’t hire them, very often they stopped being a contributor. Like, it was a real sort of a double-edged sword.

    So someone who did it for six months, they were doing it for the love of the game. They weren’t doing it to be hired at, at least I think for the most part, they weren’t doing it to be hired at Mozilla. And then six months in, someone’s like, well, you know, you’re doing a really great job, wouldn’t you love to work on this full-time? They’re like, yeah, that’s so exciting. They’re flown out for an interview. And then it’s like, you know, actually, I don’t think so.

    Edith: Wait, they didn’t wanna work for Mozilla, or Mozilla didn’t wanna hire them?

    Paul: Mozilla didn’t wanna hire them. I mean, this isn’t a particular case. But you know, some subset of people, Mozilla didn’t wanna hire. And then what happens? Well, you kind of don’t wanna keep contributing to the company that just told you to f*ck off.

    Edith: Yeah, I mean, it’s so true. I mean, I think there’s a very human thing about volunteering.

    Paul: Right, right.

    Edith: I volunteer, I was the moderator for a long time for the Lean Startup Circle, and somebody new came on, and they’re like, yeah, I’d love to raise some money so that we can get paid, and I’m like, they cannot afford to pay me.

    Like, I don’t do this to get paid, ’cause—

    Paul: Yeah, it’s like, it’s an altruism that people are trying to do.

    Edith: Yeah, I was contributing like 10 to 15 hours a week, and at my going rate, it’s not, it’s like, I’m not doing this to get paid, I’m doing it ’cause I like giving back.

    Paul: So there’s this thing that we were talking about earlier, about the current discussion around people getting paid in open source.

    Edith: Yeah, and I think a lot of what has fueled open source is the whole 20 percent time. Like, that Google will give their engineers 20 percent for open source, and I know that other big companies, like Twitter and so on, do that too. So these people are, in essence, getting paid.

    Paul: Right, right, there’s a, in most companies, in most startups at least, there’s an unspoken rule that you’re expected to contribute to the libraries that are in your application, and that sort of thing. If you’re spending time upstreaming paths that you’ve deployed, that’s work.

    Edith: Yeah, so I think we’re agreeing, which, I thought we agreed that we were not gonna agree. It’s funny, ’cause I’m Twitter friends with a guy I’ve never met, who’s, he started Jango, and right now what he tweets a lot about is—

    Paul: This is Jacob Kaplan-Moss?

    Edith: Yeah.

    Paul: Okay.

    Edith: That people are working their nights and weekends, and contributing to open source, and then just, people are kind of, for lack of a better word, jerks. People are like—

    Paul: People are jerks.

    Edith: Yeah, like—

    Paul: Even when you’re paid, people are jerks.

    Edith: But it’s more aggravating.

    Paul: Right, right, because you’re doing it for free, or whatever.

    Edith: Yeah, and so people will, like, they’ll be like, ‘I have this pull request and it’s been open for two hours. Why haven’t you reviewed it?’

    Paul: Right, right. I mean, we get the exact same thing. We get customers who email us, or more often it’s the people who get, who are using the free version, who send us a support request and then, 30 minutes later they’re on Twitter going, I can’t believe Circle hasn’t solved my problem yet. It’s been 30 minutes.

    Edith: What do you think drives that?

    Paul: There’s a pattern across the population in general, that people are jerks.

    Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of Circle CI, and Edith Harvaugh of LaunchDarkly.

    You can find the full episode of this podcast here. For more on Continuous Delivery and to hear other Heavybit podcasts, copy this link into your favorite podcast player. We’ll have show-specific RSS and iTunes subscription options soon. You can also follow To Be Continuous on Twitter at @continuouscast.