devopsy.kinsta.cloud

Tag: circleCI

  • Datree launches a partnership with CircleCI

    Datree, the GitOps management platform, has partnered with CircleCI, the leading automated Continuous Integration and Delivery system. As a CircleCI Technology Partner, datree has released CircleCI Orbs to help companies achieve technical excellence and improve production stability. Datree’s first CircleCI orb helps companies keep control of their package versioning, ensuring that only stable, approved software versions are pushed to production.

    CircleCI automates the deployment of software, taking the code in a repository, building it, testing it and then pushing it live to a staging or production environment. CircleCI introduced Orbs to make it easy to configure the platform to work with other tools.

    Datree is a centralized policy engine that works at the Git control level, automatically checking every pull request before it’s merged into production. Datree helps developers to follow best practices, preventing mistakes or misconfigurations from ever being included in a repo.

    Datree’s Orb for CircleCI brings datree’s advanced features to the entire CI/CD pipeline. For example, developers can use the datree Orb to align and control versions of packages across different repositories, projects and teams. Specific examples include:

    • UI package versioning alignment, stopping discrepancies between different versions of the same package UI components across different projects.
    • Internal package versioning alignment, ensuring the same up-to-date version of an internal package is in use across different projects.
    • Open source, letting developers create a blacklist of risky versions of specific open source packages.

    “CircleCI makes build, testing and deployment easy and fast,” said datree CEO Arthur Shmunk. “We’re excited to partner with CircleCI to an Orb for datree’s Git-level gatekeeper, giving developers control over their repositories and code at all stages in the Software development and deployment process”

     

  • Modernizing federal DevOps: CircleCI becomes first continuous integration tool with FedRAMP authorization

    CircleCI is now authorized by FedRAMP​(the federal government’s program for assessing and authorizing technology vendors), making CircleCI the first CI/CD tool to meet the rigorous security and privacy standards required by government agencies.

    Previously, government developers were forced to rely on cumbersome, legacy CI options that, while free, required too much overhead, were costly to maintain, and diminished productivity. Now, the government’s developer community can gain the same competitive advantage long available to the private sector, assured of the security and privacy controls that federal agencies require.

    “Federal managers need to start thinking about the total cost of ownership for their development tools. If your team is spending time troubleshooting a ‘free’ build tool, then it’s not free. We chose CircleCI after trying several different build tools, ranging from totally free open source tools to managed services. CircleCI just works, and that means our team only has to focus on building better government services. The total cost of ownership is much lower for a service like CircleCI.”

    – Ryan Hilliard, IT Specialist/Systems Developer, Small Business Administration

    Federal agencies like the Small Business Administration, General Services Administration, and the Department of Veterans Affairs are already leveraging CircleCI to automatically build, test, and deploy software every day. With this new authorization, the entire federal developer community now has access to the CI/CD tool trusted to run more than 12 million builds a month for some of the world’s most innovative companies, like Docker, GoPro, and Zenefits.

    What is FedRAMP?

    The Federal Risk and Authorization Management Program (FedRAMP) is an official U.S. government program that simplifies the purchase of cloud technologies to government agencies. The White House Office of Management and Budget developed​FedRAMP in early 2010 as a way to standardize the security vetting process for the shift to cloud-first initiatives. Prior to that, each agency had its own standards and guidelines which created difficult and long timelines for teams to get access to technology. FedRAMP has streamlined the security controls as a way to measure a company’s security and privacy posture against a common set of base guidelines. Whereas certifications like ISO27001 and SOC 2 are developed and assessed by private auditing firms, government officials developed FedRAMP using NIST security controls​ as the common baseline.

    Government agencies can manage risk by using the FedRAMP marketplace to identify tools that have already passed strict security and privacy requirements.

    “Traditional approaches to securing federal IT systems focus heavily on assessing and responding to security issues just before ‘go live’. This model is costly and burdensome because identified security issues can result in rework and do-overs. Using automated continuous integration and continuous deployment tools such as CircleCI, we’re able to move cybersecurity requirements ‘left’ in the process to achieve a true ‘build security in’ model.”

    – Beau Houser, Chief Information Security Officer at U.S. Small Business Administration

    What was involved?

    FedRAMP’s assessment of CircleCI in the low-impact SaaS (LI-SaaS) category included a complete inventory audit and assessment from network diagrams, to tables of third-party integrations, a master inventory of all infrastructure, and written responses to strict controls. The controls encompass a wide range of security protocols and examined CircleCI’s authorization of users, vulnerability scanning processes, and incident response. In essence, everything a federal Chief Information Security Officer should assess a new cloud vendor on.

    After successfully completing the initial vetting in June, FedRAMP officials hired a third-party auditor (a former federal employee with a top secret security clearance) to visit CircleCI’s San Francisco headquarters. The auditor spent four days in July with CircleCI employees and thoroughly reviewed how each control is handled. Their findings were reviewed by a federal security team for analysis, whereby CircleCI was granted authorization.

    CircleCI is now available on the FedRAMP Marketplace

    https://marketplace.fedramp.gov, and federal software teams can download a copy of the third-party assessment.

    To set up a demo or speak with our government team, please contact us at government@circleci.com.

  • DevOps Chat: CI/CD with CircleCI’s Rob Zuber

    DevOps Chat: CI/CD with CircleCI’s Rob Zuber

    Microsoft’s recent acquisition of GitHub has spurred a lot of conversations regarding which company is the next big acquisition target, and here on DevOps Chat is no exception. In this episode of DevOps Chat, I speak with Rob Zuber, CTO of CircleCI about the CI/CD space, the future of the market and, yes, M&A activity in the DevOps space.

    As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

    Alan Shimel: Hey, everyone, it’s Alan Shimel, devopsy.kinsta.cloud, and you’re listening to another DevOps Chat. Today’s DevOps Chat features Rob Zuber, CTO, CircleCI. Rob, welcome to DevOps Chat.

    Rob Zuber: Thanks. Good morning.

    Shimel: Good morning.

    Zuber: It’s a pleasure to be here.

    Shimel: Or afternoon, in my case, or evening. Who knows when people are listening to this? But, Rob, thanks for being our guest. I think this is the first time we’ve had anyone from CircleCI on a DevOps Chat here. So, you know, you are a mold-breaker and thanks very much.

    Zuber: I’m excited.

    Shimel: All right. So, Rob, our audience tends to be pretty DevOps-y, and so I’m assuming they know of CircleCI or at least have heard of CircleCI, but maybe they haven’t. So, in case they haven’t, Rob, give us a little background, just a real quick elevator, on CircleCI.

    Zuber: Yeah. So we’re a big part of the delivery pipeline for many software organizations. We offer continuous integration and continuous deployment, both as a cloud offering so that you don’t have to manage any of it yourself, but, if you’re interested in doing that, we also have a server-hosted version. And, you know, our objective is to help people focus on their business and get their software into market faster, taking out of the way the overhead of managing a lot of the tooling around doing that.

    Shimel: Yep. And, Rob, just in way of history and background and setting the table here, CircleCI really is – it was kind of a cloud-native CI/CD solution originally and it’s only relatively recently that you started having sort of an on-prem solution as well, offered as an option. So it’s kind of opposite of the usual SaaS migration, if you will, right, where people turn on-prem stuff into SaaS solutions. You were kind of a SssS solution and now offer an on-prem option.

    Zuber: Yeah, that’s exactly right. And that’s probably partly a function of when the company was started, so CircleCI launched in 2012. At that point, people were pretty comfortable with cloud offerings. GitHub had been around for a little while, so people were used to having the source code in someone else’s servers, and so we were able to leverage that a little bit. And we were able to grow quite effectively off of that base, but we did reach a point where there were very large customers with very specific needs, where it made a little more sense for them to run the software themselves and be able to have a little bit more opportunity to customize the environment in which it was being run, etc.

    One thing that’s interesting about that – and you talk about the difference in which way you go across that transition – because we built a multi-tenant platform from the beginning and really focused on our ability to operate that on behalf of other people, we find that, even in an on-prem solution, especially for very large organizations, we have a lot of the capability around that management at scale and centralized management that doesn’t – that still gives freedom to the individual developers to do their work. So it is actually quite interesting, how the genesis of the product affects how it operates in those multiple environments long-term.

    Shimel: Absolutely. Rob, another area – and I didn’t really talk to you off mic about this, but another area I wanted to explore was, historically, was CircleCI originally just CI or was it always CI/CD?

    Zuber: I think you could kind of look at that two ways. I mean, there was CD pretty early in the product, at least for some particular cases, but, if you look at the evolution of the software industry, many more people were doing CI in 2012 than CD. I mean, it was sort of this slow transition of people getting to, you know, “First, we’re gonna figure out Agile process and just how we do our actual software development. And then we’re gonna add CI on top of that. And then we’re gonna add CD.”

    So we always had some CD capabilities, but the number of our customers or the percentage or breakdown of our customers using CD has grown pretty significantly over that time, and I think that’s partly an effect of the kinds of tooling we’re offering, partly a shift in what CD even means, and then partly just growth in the market and comfort level of people doing that, right? Of – especially – and CD, of course, is always challenging ’cause some people are talking about continuous delivery and some about continuous deployment, but, in particular, on the continuous deployment side, other things had to come into play – you know, feature flagging, canaries, just better control of what it meant to actually put something in production. And so the comfort level and increase in that has led to more people using our platform for CD as well.

    Shimel: Absolutely. You know, as an observer of the space, sort of the market, I’d seen it come – when I first launched devopsy.kinsta.cloud, four years ago, Rob, there was this clear sort of distinction between “We’re not CD. CD is something else. We’re CI,” or reversed – “Those folks are just CI. They’re not CD.” And it kinda just seemed like an artificial wall had been built, right? Not to get all political.

    But – and so I, for one, am happy to see sort of that wall being torn down and the natural flow of the software development life cycle, of the modern software factory, if you will, taking place, where CI kinda naturally flows into CD. And I think the next thing we’re seeing is this natural connection to what Gartner and some of the others call “ARA.” Right? Which I don’t even know what the difference between that and CD is, but I don’t know. What do you think?

    Zuber: Well, I think that, overall, this evolution, as I was saying, it always takes a long time for people to get – or, let’s say, for everyone to get comfortable with each of these stages of the evolution. And we’re always looking to the next frontier, right? So, at one end of the spectrum, you have those early adopters, who, in 2012, were doing CD or before that. I’m trying to think of the original sort of posts around CD that I think came out of, I wanna say, Etsy, but I don’t know if that’s actually true or me projecting kind of them as a pillar of this type of thinking. But, you know – and we’re still at a point where some people are just getting comfortable with and adopting it, so the spectrum is really broad, from early adopters to sort of overall mass adoption.

    And the early adopters have moved on to, probably, things that I haven’t even thought about, and so we’re always seeing that and it’s nice to have people try those things and, in some cases, fail, and, in some cases, discover really new and interesting ways of approaching stuff. And it takes – it definitely takes a little bit of a mindset of “I’m just willing to try this. I’m willing to take some risk and do new things,” and the people that do those create opportunities for the rest of us to learn from, the things that go well and the things that go horribly wrong.

    And so it’s one of the reasons that I just love being in the space in general, is just the constant kind of desire to do things better and find new opportunities, of course, with a good balance towards actually focusing on getting stuff done, but there’s just such a creativity and desire around improving, in any way we can, this process and how we get stuff out and how we build it reliably and work efficiently.

    Shimel: Absolutely. So, Rob, let’s turn a little bit to – we’ve done a nice history. Let’s look a little bit to the future now, though.

    Zuber: Mm-hmm.

    Shimel: Talk to me – what did – how – you’re the CTO. You kinda have a lot of the vision for CircleCI. Where do you see CircleCI and the market in general going?

    Zuber: Yeah. Well, that’s a great question. I think a lot of people’s perspectives of this market was changed a little bit in the last week or so, with news around Microsoft and GitHub and just the value that people are starting to recognize in developers, the happiness of developers, the efficiency, the value that they place on the tools that they use. So, for the market overall, I think we’re just a really exciting time, where we talked about this – or I talked about this creativity and passion to do things better, but we’re at a sort of inflection point where there’s a really, really high-level recognition of the value of that.

    I would say there’s been a significant shift in the mentality of many companies. And we talk a lot about how every company is becoming a software company, in that, even if you’re an airline, your biggest issue, when you ground your planes for a day, is that you have some software glitch somewhere, right? You know, multinational trading markets are shutting down on software issues. So the impact and value of software is becoming really, really visible and, as a result, there’s a shift in mentality about how we build software, from “This is a cost center that I just have to manage down,” to, “This is the core of my business and I need to be really investing and doing this in the best possible way.”

    And so that’s obviously exciting for us. And one of the things that’s really interesting for us, as we look forward, is that we – we’ve had the fortune of some success and growing and, to the best of our knowledge, run the largest or one of the largest build platforms on the planet. As a result, we know a lot about how people are building software, and so we’re investing a lot of our time and energy in figuring out how to use that to then help those people, meaning, if you’re building on CircleCI, you don’t just get an understanding of your particular code change that you just made, but you can get an understanding of how your team is doing, how your whole process is doing, how effective you are, relative to other teams out there.

    And I think that’s a place where we have a lot of opportunity to really add additional value for our customers, in not just, again, direct feedback on this one thing that you did today, but really helping understand your process, maybe where things aren’t working well, maybe where you have some opportunities to improve. We’re all in this, trying to be better every day. And if we can help customers at that at a higher level, that’s really exciting for us.

    Shimel: Yep. You know, it’s interesting you brought up the GitHub thing and to justify all this. And so I have some views on it. What a surprise. But, you know, Rob, couple things. No. 1, when you look at GitHub and the price Microsoft paid, it obviously is a ridiculous multiple to revenue. I saw a chart earlier today. I forgot if they said it was 21 or 24 times revenue, which is wow. Right? That’s why people do software companies.

    But another way of looking at it is, look, Microsoft has always coveted the developer community. And, over the last – I mean, for as long as I’m in technology, 25, 30 years, they’ve done not a crappy job of working with the developer community. Yeah, there are haters and Linux folks and the open source, when Ballmer was there, you know, but they’ve always gone after the developer community hard. And so they had an opportunity to buy a company that has, some estimates say, 28 million people or potential developers, right, in that community that GitHub has. So, when you look at 28 million and what they paid – don’t get me wrong; it’s still a lot of money, but it’s a little bit – you could see where they saw the value, okay?

    Zuber: Mm-hmm.

    Shimel: No. 2, having been in the tech, software infrastructure business a long time, I learned a lesson from my friend Brad Feld, who I sold my first company to and he financed or was one of the key VCs behind many of the companies I’ve been involved in, and that is, look, first mover or first – not the first mover, first acquisition in the space always gets the lion’s share and the best multiple. The second one does okay. The third one does a little worse. If you’re not in the top three, don’t bother. Right? That was something Brad always preached to us.

    So, when you look at Git – or GitHub, excuse me, being the first kind of acquisition in that space, getting this kind of multiple, they deserve it. Right? They were first. They were the first ones bought; they’re gonna get the lion’s share. I think the message to other folks in the GitHub space, specifically, is “Hurry up and get bought ’cause you don’t wanna be number four or five.” Right? You don’t wanna be left in the musical chairs game when the music stops.

    Part and parcel of that, though, Rob, is that companies like Microsoft, like IBM – we used to say “like HP, Cisco,” you know, big, big public entities – it’s very hard for them to be innovative. It’s very hard for them to do their own R&D. It’s very hard for them to build these communities themselves. And so what they would spend in R&D or community actually goes into their M&A budget and they buy those things.

    Zuber: Mm-hmm.

    Shimel: So, when we look at a GitHub, we look at a CircleCI, we look at any – a Chef, a Puppet, any of the – even the Jenkins or CloudBees, any of the players in this DevOps space that is so hot right now – GitLab, another one – they all are gonna be very attractive to companies of a Microsoft kind of girth. Right? Because the big guys can’t – they can’t do what you’re doing, Rob. They don’t –

    Zuber: Yeah.

    Shimel: You know.

    Zuber: Yeah. I mean, I think this is classic innovator’s dilemma, right?

    Shimel: Yeah.

    Zuber: Did I get that right? So it’s difficult. I mean, if you look at the first five year – I mean, we all talk about these overnight successes, right, and they always – they’ve been around for 10 years. And the first five years many of those companies is extremely painful, it’s a grind, and the incremental increase, the amount of revenue you’re driving or whatever, would be so insignificant on any one of these companies’ radar that they just – they won’t put the time and energy into it, right?

    Whereas you’ve got a company like ours or anybody in the space, honestly – I’ve done a bunch of different startups in different spaces – you’re living and breathing it, right? And you’re so tuned into what’s happening that you can see it. And that kind of incremental growth, to you, is everything and it’s super exciting and you get this passion behind it that allows you to build a very focused and, honestly, a great product. And I think GitHub is a great product. And then, suddenly, you have 28 million users – I’m quoting your number so I’ll assume you’re right –

    Shimel: _____ I’ve seen.

    Zuber: And a massive community. You know, and then, going back, as you were talking about the time and tech, I mean, I got into tech with the quarterly mailing of MSDN CDs and, honestly –

    Shimel: Exactly. Yeah.

    Zuber: – my favorite IDE – I’m a Mac and Linux user, almost exclusively, but I used Visual J++ in the late ’90s, when I started doing Java development. It’s still the best IDE I ever used. So Microsoft has been good, very, very good, with developers, but the kind of developer community has started to drift away from where they were going. And so this is – I wouldn’t call it a course correction because they still have all of their community, from a .NET and sort of Windows world, but this is an opportunity to be less about Windows, you know? And we’ve seen Microsoft shifting away from –

    Shimel: Oh, yeah. Satya Nadella’s Microsoft, Rob –

    Zuber: – just pure kinda _____ –

    Shimel: Not to step on you, but Nadella’s Microsoft is a very, very different place than when Steve Ballmer was there. Right?

    Zuber: Yes.

    Shimel: And they are moving to Azure and though Azure does Windows well, it does containers well, it does Linux, it does – you know, there’s very little it doesn’t do.

    Zuber: Right.

    Shimel: You know? And I will tell ya – I’ve told people this – I think, one day, Microsoft buys Docker. That’s my personal opinion. I think it’s another kind of – not to get into it or badmouth or anything like that, I just think someone’s gonna buy Docker ’cause the revenue has to justify the valuation, and so – and Microsoft’s a great candidate to do so. But, anyway, you heard it here first. But, Rob, that’s all fine and dandy. GitHub plays in sort of a different segment of this DevOps market than you guys do, certainly.

    Zuber: Mm-hmm.

    Shimel: And we really haven’t seen the consolidation yet, that I think we will in the CI/CD and, if you wanna call it, ARA space, but it’s coming. I’m sure.

    Zuber: Yeah, we’ve seen movement in the market, but most of it has been off the bottom end, if you will, meaning sort of the smaller people we would consider to be our competitors but didn’t quite reach critical mass and then got pulled in by other companies, for the people or because the product was going to be useful to them internally, that sort of thing. Not a roll up and now we’re taking this and getting it scales or _____ –

    Shimel: So we used to say, when I worked with Brad, those are companies that really were a feature, not a product.

    Zuber: Mm-hmm.

    Shimel: But, with that being said, Rob, we went off on a little bit of tangent here. We’ve used up way more time than we were supposed to. But maybe we could have you back on and – ’cause I wanted to really get into a little nuts and bolts around continuous integration, delivery, what it means for developers, what it means for ops folks, but why don’t we do this? We’ll call it a day on this 25-minute DevOps Chat and we’ll have you back on maybe next month. And let’s get into what it means for developers and ops folks in even testing, in this new world of CI/CD.

    Zuber: That sounds great. I’m always happy to talk about that as well. There’s so much happening in this industry, in this space right now, that it’s hard to keep it short. [Chuckles]

    Shimel: You know what? Hey, man, it keeps my kids going to college, Rob. [Laughs] Anyway, hey, Rob, Zuber, CTO, CircleCI, this episode’s guest on DevOps Chat. Thanks for joining us. This is Alan Shimel. You’ve just listened to another DevOps Chat. Have a great day, everyone.

    — Alan Shimel

  • CircleCI Injects Workflow Flexibility into CI/CD Platform

    CircleCI Injects Workflow Flexibility into CI/CD Platform

    Nobody likes to be told how to get a specific task or job done. In fact, it’s arguably that very sentiment that holds backs adoption of DevOps processes inside many IT organizations. Aiming to provide IT organizations with a more flexible approach to DevOps, CircleCI has updated its continuous integration and continuous deployment (CI/CD) software to give IT organizations more granular control over workflows within a DevOps environment.

    CircleCI CEO Jim Rose says the issue with most CI/CD frameworks is that they are too monolithic. Rather than allowing IT organizations to apply different types of workflows against various types of scenarios, Rose says most CI/CD platforms are too prescriptive in how they define workflows.

    To provide some additional flexibility, version 2.0 of the company’s namesake CI/CD platform provides IT organizations with the ability to employ different types of workflows whenever appropriate, including:

    Sequential job execution: Provided in the first release of CirceCI, this class of workflow is optimal for approaching jobs in a lockstep build-deploy-test model.

    Parallel job execution: Allows IT organizations to run multiple jobs simultaneously in a way the resolves dependencies before deployment.

    Fan-in/fan-out in a repo: Ideal for a more complex build orchestrations involving different test suites against commits to get faster feedback.

    Branch-level filtering for job execution: Allows deployment of code off specific branches.

    Job execution from failed and job execution from start: Allows IT operations teams to choose where and how build reruns are implemented to save repeating unnecessary jobs. IT allows IT operations teams to start jobs from the last failed job, instead of rerunning every job or stage.

    Rose says the rise of microservices has begun to stress CI/CD platforms that were not designed to scale to support environments where Docker containers in any given microservice often are replaced. Any change to an element of microservice is likely to have a significant impact upstream and downstream. Rather than having to continually rerun every job, Rose says CircleCI now enables IT organizations that have embraced microservices to respond more adroitly to changes in the overall environment.

    In an ideal world, DevOps processes lead to faster rollout of new applications and updates. In reality, however, there are more dependencies than ever. Rose contends that CI/CD platforms need to evolve to the point where they can allow IT operations teams to cope with accelerated rates of change instead of becoming a bottleneck. The irony of that, of course, is that CI/CD platforms were adopted to eliminate bottlenecks. But as microservices architectures continue to evolve, in many cases the CI/CD platform has become a new bottleneck.

    It’s too early to say what impact microservices will have on DevOps. There’s no doubt that adoption of microservices will finally force the DevOps issue inside many organizations. But at the same many organizations that have already gone down the CI/CD path may soon find that some of their initial assumptions about what the word “continuous” meant were considerably misguided.

    — Mike Vizard

  • To Be Continuous Podcast: Have We Forgotten How To Code?

    To Be Continuous Podcast: Have We Forgotten How To Code?

    Welcome to To Be Continuous, a show about continuous delivery and software development, hosted by Heavybit members Paul Biggar, founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly.

    In this episode, Paul and Edith discuss the fallout after the widely covered “left-pad” incident, and while they agree that these sorts of moments move the entire open source community forward, they wonder: Have we forgotten how to code?

    [soundcloud url=”https://api.soundcloud.com/tracks/264605904?secret_token=s-Kv9Ep” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Edith: Hey, Paul. So, somebody unpublished a module called left-pad and the internet basically exploded. And now it’s been a couple weeks since then, so there’s a lot of different topics to cover in terms of what this means for open source, whether it means that people have forgotten how to program.

    Paul: So LaunchDarkly. Or, I’m sorry, the left-pad. I’m obviously looking at my own t-shirt.

    Edith: You’re looking at your LaunchDarkly t-shirt?

    Paul: The left-pad incident was pretty interesting and one of the things that we did when coming up to this was just, like, looking at all the Hacker News discussions and comments on it.

    And there’s wildly varying positions on what this means for open source, what it means for whether people can program, what it means for, like, about the Node ecosystem. So to kick this off, why don’t you give us a recap of the entire incident?

    Edith: Yeah. So, in short, a programmer had a module called Kik, K-I-K. Which is also [the name of] a very popular messaging app. I mean, even I’ve heard of it and I’m not a teenager.

    Paul: The Kik lawyers contacted the dude. Said, “Please take down your module, change it.” And the programmer said, “No. I basically don’t believe in copyright laws.” So then, the Kik people contacted NPM. The NPM people took down the module called Kik. And the guy, in I’d say, kind of a fit of pique, said, “Okay, you took down my one module, I’m gonna take down all my modules.” Basically, I’m taking my marbles and going home.

    So, he took down all of his modules. One of which turned out to be an incredibly popular module called left-pad, which did basically what the name implied. It sorted left pads. And without this in NPM, the quote I read was, “Builds all over the world started to fail very quickly.”

    Paul: So as I understand it, left-pad was used by another module, which was used by another module, which was used by something core, or like, this incredibly popular module—

    Edith: Yeah, it was basically turtles on top of turtles on top of turtles, and at the bottom turtle was this guy who—

    Paul: Took his marbles and went home.

    Edith: Yeah, or yank the bottom Jenga, basically.

    Paul: Gotcha. And so all the builds started to fail. People’s deploy started to fail. And people got pissed.

    Edith: People got pissed because they didn’t even realize that they had this dependency. So what happened was NPM basically republished it. They said that, even though you’re the original publisher—

    Paul: So there was a lot of outcry on the internet and then they republished it.

    Edith: Just basically said, “You don’t have the right to un-publish this.”

    Paul: Gotcha. So, um, but they had an Unpublish button, as I understand it. So he did something that was built into NPM to remove a module.

    Edith: Yeah, and since then they have tightened their rules a lot. They basically said that this should be something that’s used extremely rarely, if at all, and not something you can just do capriciously.

    Paul: So one thing that’s interesting about the NPM ecosystem, and the Node ecosystem in general, is they’ve had a bunch of things like this. Things that are security-related, or that are drama-related.

    And people have repeatedly come out and say, “Oh, no, JS is a tar pit, or ghetto,” or whatever it is that people use to describe poor ecosystems, that normal programmers don’t know how to code, and so on, so on.

    And what it looks to me like is, you know, each of these incidences brings the Node ecosystem quite a bit forward. To the point that, I mean Node is relatively new, it hasn’t had 15 years before prime time in the way that Ruby or Python did, or 20 years before it got big, and it’s a real, kind of bottom-up thing.

    I mean, if someone built it and there were some steroids and then someone else took over, and then NPM took over, and then. It’s this sort of weird language from the ecosystem perspective. It didn’t have the same advantages that, like, Java had, or that C or C++ had, in order to get to maturity. And so each of these incidences propels it closer to maturity.

    Edith: Yeah, I compare it somewhat to when electricity first started coming out. And then you were just working through all the issues that could happen. Like, for example, when a power line went down or a transformer blew.

    Like, you work through, Okay, what are our points of failure? Let’s not make that a point of failure again.

    Paul: Hmm. So, NPM changed their policies on this basis, right?

    Edith: Yep, so now you can’t have an individual contributor be able to unpublish, even if they originally published. Now you have to contact them, and it’s much more stringent.

    Paul: Gotcha. The response, despite people being … Well, people were annoyed at their response.

    Edith: It seems perfectly rational to me.

    Paul: Oh, I think it seems perfectly rational. This is the thing about the in trend. There’s always some set of people who are annoyed at everything you do.

    Edith: Well you know, the internet is full of billions of people. I mean, there’s gotta be some sort of standard deviation, or otherwise it would be a very boring internet.

    Paul: Right, right, and this is why there’s always haters on Hacker News. No matter what you do. If you do it one way, there’ll be one group of haters, if you do it another way, there’ll be a different group of haters.

    And so, with left-pad, there were people who were annoyed that NPM retroactively decided that they had ownership over their namespace, which previously, they had sort of gifted to people in the community, the first people to take the namespace.

    Edith: You mean the original thing about Kik?

    Paul: So there’s that, but there’s also the left-pad. They took away the sort of autonomy of the left-pad offer.

    Edith: I read a good article that basically says part of open source is, if it truly is open source, do you maintain the rights to it? Or is it a community project?

    Paul: Hmm. The traditional way of deciding that sort of ownership was the fork. That you could go, and you could fork, and when you forked, you weren’t necessarily entitled to the same name, you weren’t entitled to the same trademarks. When Debian forked Firefox, they called it Iceweasel.

    When Jenkins was forked, or Hudson was forked, it became Jenkins. GCC got forked. They all got new names. But what was the fork here, it wasn’t even a fork. It was a republish of the same thing, the same software, in a namespace that the original author didn’t control.

    So typically you’d own the URL, you know, if you have a software library, you’d own the URL. And they did own, in some sense, the URL within NPM, or the name within NPM’s space, and NPM, the sort of owners of that namespace, overrode it.

    Edith: I think they did what was necessary for good order.

    Paul: Sure. I completely agree, I think that was a good decision.

    Edith: I mean, so to take this to the logical conclusion, what if you had a module named left-pad, and you decided that, like, as a prank you were going rewrite it to be right-pad? Like, sure, that’s … Is that within your rights? Kind of, but it’s a real jerk thing to do that you probably want to prevent.

    Paul: This is kind of one of the dangers of the whole ecosystem around centralized package managers, of sorta trusted package managers, where there’s no actual trust involved. So, what I mean by that is what is published does not have any gatekeepers.

    Edith: Yeah, exactly.

    Paul: I mean, it’s great, it allows an ecosystem to grow much faster, but it has safety concerns because people generally don’t pin their packages to an exact version.

    Edith: You’re basically pinning it to a floating value.

    Paul: You’re pinning it to a floating value. And that floating value could change. There’s no guarantee of what a minor number means or what a major number means or what a patch number means or what a patch even means. There’s some sort of, like, general community guidelines in semantic versioning, but there’s no enforcement of that.

    Edith: Yeah, and I mean this basically goes back to the whole Cathedral versus Bazaar thing.

    Paul: Right, right. It’s very Bazaar. And intentionally so. It’s the thing that allows a community to grow, and is there a point at which open source communities need to, like, batten down the hatches to prevent that sort of behavior? Or prevent that sort of risk?

    Edith: Yeah, and it’s really interesting. I think this goes back to something we’ve talked about with Nadia Eghbal and Sean Byrnes a couple episodes ago about when you buy from a corporation, you have some sort of guarantee of standards that they publish somewhere. You know, this is our guarantees versus with open source, you don’t normally have those same sort of guarantees.

    Paul: Even when you do, it tends to be—

    Edith: There’s no enforcement mechanism.

    Paul: Right. So there’s no enforcement mechanism in the Bazaar.

    Edith: Well, there’s no, um … Like, so with a company, if a company has said, “Okay, we’re gonna release twice a year and it’s gonna have this level of quality,” and they break that, you can walk with your feet or your money. If an open source project stops or does something that you don’t like, there’s very little enforcement end.

    Paul: Theoretically, you could walk. You know, you could fork the thing, but when you end up in an ecosystem that has massive network effects, and in the way that NPM does, you can’t walk with it. Because the pends is all transitive. The left-pad was, you know, a turtle way at the bottom of the stack.

    Edith: It was almost like a turtle hatchling.

    Paul: So, to fork left-pad, you would have to fork the entire NPM ecosystem.

    Edith: Yeah, and it’s not even. The fork is not the issue, though. The issue was that people were depending on it being in a certain place in a certain way. It wasn’t that it wasn’t forked, it was just like this entire structure at the bottom was gone.

    Paul: I guess what I’m saying is that if you get pissed off with a certain set of developers or a certain set of practices within the NPM community, because they’re so interlinked, you don’t have the traditional open source option of forking.

    Edith: Absolutely. Absolutely.

    Paul: You could rename it. You could pick a different library, but because of the way—I’m not sure if they still do this, ’cause I read something about it, but in NPM, it used to be that the version of the software that your library took was the version that library got. Do you know what I mean by that, okay. So let’s say there’s some software, let’s call it left-pad.

    Edith: Let’s call it right-pad.

    Paul: Right-pad. So, right-pad version one is being used by library A. And right-pad version two is being used by library B. In the Java ecosystem, and many, many other ecosystems, only one version of right-pad would be used.

    And I believe in NPM, both versions of right-pad would be used. So library A will use version one, and library B will use version two. Now that’s definitely the way that it used to be, I’m not sure if they’ve actually changed that.

    Edith: Yeah, I don’t know, but that sounds very messy, very quickly.

    Paul: So both options are terrible. And there isn’t a right way to do it.

    So, really, what it depends on. So, the NPM way of doing it, every library get its own version of the dependencies, is fantastic. Because it lets that library do what it actually intended to do under the covers.

    If, however, you start having an object that gets sent around, or that two modules are, let’s say it receives a, I think, an HTTP packet or something like that would be a good example. Let’s say you take an HTTP packet from one module and just hand it to another module, and they have different understandings of the shape of that object, or what functions you can run on that object, then that’s messy.

    But, on the other hand, if you override the version of right-pad that some other library’s using, you end up with all sorts of problems. This is a major problem in the, sort of the Maven ecosystem. And at CircleCI, we used Clojure, and Clojure is built on top of the Maven ecosystem, and we would have bugs where the library that we used.

    One library that we used had a dependency, and that was in the package file for another thing. And so, that’s the version of the packets that we used. So we would have to pick. And as a result of that, we have to pick the library that works. And, often, there isn’t a library that works for both, or there isn’t a, sort of subdependency that worked for both. So those are close to forked no matter which way you do it.

    Edith: Yeah, I mean, we ran into the same issues with our STKs. I mean, ’cause we have dependencies on other libraries that sometimes conflict.

    Paul: Right, disaster.

    Edith: On the other hand, I mean, the bigger point, I think, was. So there was an interesting backlash, I think, that everybody kinda projected what they wanted on the NPM thing.

    People said, Oh, people don’t know how to program anymore. You know, why do you even use what they called left-pad? You know, why don’t you just grind your own wheat and make your own bread from scratch and eat it at lunch also?

    Paul: I have no sympathy at all for this view of the world.

    Edith: Uh, that you should make everything from scratch? Or—

    Paul: That, “Oh, we’ve forgotten how to program because we need left-pad.”

    Edith: Yeah. The fact that we have modules available that make it easier and easier and easier to program is a good thing.

    Paul: I think it’s kinda different to that. I think it’s more of a curmudgeon view of the world. It’s a get-off-my-lawn—

    Edith: Wait, that you’re the curmudgeon?

    Paul: No, the people who complain, “Have we forgotten to program anymore?” I think it’s a no true Scotsman kind of argument. Real programmers know how to write their own left-pad, real programmers write in statically typed languages, real programmers write in C, real programmers, and I’m going back in time here, real programmers write in Assembly, real programmers hand-code their. Tweak the needles on the.

    Edith: They do their own garbage collection.

    Paul: Oh, and it’s garbage collection, what is this? Garbage collection is for amateur coders, right? I mean, you could just as easily have written, have we forgotten how to manually allocate memory?

    Edith: Or, you might as well say, like, nobody knows how to make their own transistors anymore. I mean, like, the entire computing revolution has been that you no longer have to assemble a computer by hand.

    Paul: Right. And each layer of the ecosystem allows a vastly larger number of people to take part in it. So I think the thing that we’re seeing now. This has been happening for five or 10 years, but what’s getting really, really big now is people who only know how to front-type program.

    People who only know Javascript and HTML/CSS, and they go to boot camps and they come out, they maybe have some, like, vague real understanding.

    And they call themselves coders, and we, the real programmers, who know how to write a quick sort, and who know how to write left-pad, we don’t welcome them to our community. And frankly, I mean, I have felt this in the past, I have since changed my mind, but it’s a very, very common perspective.

    This podcast is brought to you by Heavybit, a program that helps startups turn products loved by developers into successful businesses. Throughout the program, founders work on developer traction, product market fit and customer development. If you like what you hear, be sure to subscribe to Heavybit’s Podcast Network on iTunes.

  • Q&A: CircleCI’s Kevin Bell on Continuous Delivery

    Q&A: CircleCI’s Kevin Bell on Continuous Delivery

    There’s been a lot of talk and momentum lately when it comes to continuous delivery. And it seems every enterprise has moved part, or all, of its development efforts to continuous delivery—or is at least talking about doing so. But there are many challenges such as security, dealing with legacy systems and internal politics associated with the move. With that in mind, we wanted to get a sense of where continuous delivery stands in the market today. And for that we turned to Kevin Bell of continuous integration and delivery platform maker CircleCI.

    Kevin Bell has been a both developer and product manager of cloud-based software products, and is currently developer evangelist at CircleCI, where he gets to play with almost every deployment-related technology out there, hack a little Clojure and absorb the wisdom of very smart colleagues and customers. We thought it’d be a good idea to get his perspective on continuous delivery in the marketplace.

    Kevin Bell Headshot

    devopsy.kinsta.cloud: I wanted to get a sense from you where your customers are in the range of continuous integration and continuous delivery? What are you observing in the market?

    Bell: I would say continuous integration is a hard prerequisite to continuous delivery. You need to have continuous integration first, and once you have that down, you can expand from that into continuous delivery. Maybe there are people that automate deployments with a minimum of testing in the middle, but you really want to have continuous integration set up.

    Generally, I would say of our customers, on the order of half of the active customers, are deep into continuous delivery. There are certainly people that just use us to test and then do something else to deploy, and then there are those who aggressively, many times a day, deploy to production directly from CircleCI in a totally automated fashion. So we definitely have both extremes and then people that are experimenting in the middle on trying to figure out how to make that jump.

    devopsy.kinsta.cloud: Have you seen many of these changes in the past year? And are many of the changes to continuous integration and deployment happening outside of Internet and SaaS (software-as-a-service) companies?

    Bell: Lots of companies want to do it. So one change I’ve seen in the past year is that, not necessarily a change in the market, but a change in CircleCI’s world is that we have rolled out an enterprise version, which basically means a version of CircleCI that you can deploy behind your firewall. We are seeing a lot of interest in this among bigger companies. And we’re talking to a lot of different types of companies that are not SaaS companies, but are just any kind of company from apparel to finance institutions—just all manner of companies that take their internal software development practice very seriously, and very much want to do continuous delivery and have a modern software delivery methodology.

    It’s kind of just becoming table stakes that you’re going to follow these kinds of practices today, even if you’re not a SaaS startup. It’s not just about software deployment, either. It’s also necessary to attract talent.

    devopsy.kinsta.cloud: I would imagine, because we’re starting to see pilots and success stories, enterprises are seeing that it’s not just speed when it comes to feature release, but once the continuous delivery pipeline is up and running, it’s a more effective way to create and maintain software.

    Bell: That’s definitely a big part of the promise of continuous delivery, and I think it delivers. Pardon the pun. It does let you get software features out faster. If you do end up releasing a bug, it lets you fix it faster. But I think some of the most emotional stories of success that I hear from people talking about the dark old days when they deployed at once every two weeks, or once every month, is just how risky that felt. And when they clicked the big deployment button in those days, how much was at stake and how many things could go wrong because it was such a big blob of software being deployed all at once. A big part of the idea of continuous delivery is that you don’t do those big bang software deployments. You’re always just putting out a little bit at a time, and keep everything in an equilibrium throughout the process.

    devopsy.kinsta.cloud: These enterprises that you were talking to, what are some of the challenges or hurdles they have moving from legacy development to continuous deployment? What are the barriers they need to overcome to get to where they want to be?

    Bell: There’s certainly various barriers, and it depends on the organization. Some organizations have a momentum built around a certain process. Maybe there’s political issues in an organization, maybe they have a new CIO that wants to change how they do things, but they have a lot of legacy software and processes that still operate in the old-mold paradigm that they need to update. So there are a lot of systematic challenges organizations run into. But there are definitely technical challenges, too. And I think just broadly to compare, for example, continuous integration and continuous delivery, I would say that continuous integration has been around longer now and has kind of matured a bit.

    And a lot of the challenges have been uncovered within continuous integration when it relates to complicated and offbeat projects, whether that’s using things like mobile emulators like you can use on CircleCI to do iOS testing, or mobile device clouds, or browser testing, or all these things to kind of fill in the gaps, and test suites for big complicated software has sort of been explored and addressed to a large extent. But in continuous delivery there remain complicated, nontrivial systems that they’re deploying, not just little apps that go to Heroku or something. There’s this whole world of different policies and structures of software and systems that need to be continuously delivered. I think the industry and toolset makers are still exploring how to provide that.

    devopsy.kinsta.cloud: Earlier you mentioned that if an organization wants to attract talent, they need to have these modern development processes in place. So I would imagine developers don’t want to get bogged down, in their view, in old methodologies and want to get aligned with companies that have these more modern practices as part of their career development?

    Bell: That’s come up often at big companies. I would say that category of Fortune 500 companies, and not just software companies, they need to assure potential hires that they have modern software practices, and that even though they’re not a software company per se, that they value developers, and that they’re going to be a good place to work.

  • CircleCI Secures $18 Million in Series B Led by Scale Venture Partners

    Industry Shift Toward DevOps Propels CircleCI as It Scales to Meet Rising Demand for Unified Web and Mobile CI/CD Solution

    SAN FRANCISCO, CA –(Marketwired – May 16, 2016) – CircleCI, the leader in continuous integration and continuous delivery, today announced it raised $18 million in Series B funding led by Scale Venture Partners. Existing investors DFJ, Baseline Ventures, and Harrison Metal also participated in the round, bringing total financing for the company to $28 million. Timed with the investment, Andy Vitus, Partner at Scale Venture Partners, has joined the CircleCI Board of Directors.

    Founded in 2011, CircleCI helps software teams rapidly release code they trust by automating the build, test, and deploy process. Built for developers, CircleCI offers a modern software delivery platform that lets teams ramp up quickly, scale easily, and build confidently every day. Thousands of technology-driven companies including Facebook, Sony, Kickstarter, and Spotify rely on CircleCI to accelerate delivery of their code, helping software developers focus on creating business value fast.

    The company plans to use the funds to drive product development, expand operations to support a growing global customer base, and to scale sales and infrastructure to support new and existing customers of both the SaaS and Enterprise solutions.

    CircleCI joins an impressive list of Scale Venture Partners portfolio companies, companies including Chef, Box and DocuSign. “We are in the midst of a transformation to the software delivery pipeline, mainly driven by the demands for speed as enterprises move toward releasing code daily and hourly. As complexity rises, enterprises large and small have new demands for speed, assurances of quality, and automation, making the software delivery phases a critical business driver,” said Andy Vitus, CircleCI Board Member and Partner at Scale Venture Partners. “CircleCI is at the core of helping modern organizations build software across languages in a single interface. We look forward to working with them as they continue their rapid growth.”

    CircleCI will use the investment to continue the tremendous momentum it experienced over the past year including:

    Key Product Launches

    • CircleCI Enterprise on-prem deployment further expands options for modern software teams who want powerful CI/CD behind their firewall. CircleCI Enterprise enables customers to maintain complete control over access to their secure source code, test databases, and other resources.
    • CircleCI for OS X enables mobile app developers to test and deploy iOS apps quickly and easily. After a successful public beta and limited release, the new hosted OS X testing platform is now available for companies of all sizes and across industries to purchase and use.

    Employee Accolades

    • CircleCI was recognized as a winner of the 2016 Bay Area Best Places To Work, an awards program presented by the San Francisco Business Times and the Silicon Valley Business Journal. CircleCI was among the top 30 small Bay Area companies whose employees rate them as the highest on such values as fun, collaborative culture, solid compensation and benefits offerings, and other amenities as well as management practices.

    “The market is shifting with a keen eye toward DevOps and developer-centric solutions for good reason,” said Jim Rose, Chief Executive Officer, CircleCI. “We provide the velocity developers need to turn the release cycle from days to minutes without sacrificing quality. We are experiencing incredible momentum and we are eager to continue to bring the best possible solution to market for our customers and partners.”

    About CircleCI

    CircleCI helps software teams rapidly build code they trust. Thousands of technology-driven companies rely on CircleCI’s modern continuous integration and delivery platform to automate the build, test, and deploy processes. CircleCI empowers developers to focus exclusively on creating business value fast.

    CircleCI is a Bay Area Best Places to Work 2016 award winner. Founded in 2011 and headquartered in San Francisco, CircleCI is venture backed by Scale Venture Partners, DFJ, Baseline Ventures, and Harrison Metal Capital.

    Follow CircleCI on Twitter. Visit circleci.com to get started.

    About Scale Venture Partners

    Scale Venture Partners invests in technology companies with a focus on SaaS and Cloud Infrastructure. We partner with entrepreneurs, from their first dollar in revenue and invest to accelerate growth and establish market leadership. ScaleVP has$1B in assets under management and is investing its fifth fund. Representative investments include: Box, Chef, Cloudhealth, DataStax, Demandbase, DocuSign, ExactTarget, HubSpot, PubNub, Omniture and WalkMe. ScaleVP is located in Silicon Valley and invests nationwide. For more information, visit www.scalevp.com or follow us on Twitter at @scalevp or LinkedIn at www.linkedin.com/company/scale-venture-partners.

    CONTACT INFORMATION

  • To Be Continuous: The Fear of Shipping

    To Be Continuous: The Fear of Shipping

    In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk about the fear of shipping and if code is an asset.

    Edith Harbaugh, CEO LaunchDarkly

    Edith is CEO and cofounder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

     

    Paul Biggar, CEO CircleCIPaul is the CEO and founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

     

    [soundcloud url=”https://api.soundcloud.com/tracks/229127881?secret_token=s-fslKp” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

     

    Paul: Okay, so one of the things we wanted to talk about this week was the idea of shipping velocity and how it’s affected by team structure, sort of miscellaneous factors that aren’t part of continuous delivery.

    Paul: So at the end of the last episode, Edith, you said, I think this was a quote from Yammer VP, “The organization you design is the software you built.”

    Edith: Yeah, it was actually from the CTO Adam Pisoni and it really struck home from me because I see this at so many companies, they have a lot of engineers and they wonder why they have a lot of code but not a lot of product. And it ties back to what you just said of specializing the product management role.

    Paul: Right. I think that this a name. I think it’s Conway’s Law and the way that I saw that expressed is that if you have, if you’re building a compiler and you have four different teams that are building a compiler, you’ll end up with a four-pass compiler.

    Edith: There’s no one vision.

    Paul: Right. When looking at lots of different teams and team structures, the interesting one that I found was the Heroku one.

    And they have a language team and they have an add-ons team and they have sort of sharp delineations in their software, or in their stack, that allows them to really focus on one particular area because they’re such sharp demarcations between the different areas of the product.

    Edith: I think that’s good if you’re a fairly mature product. I think in the early days of Heroku that would not have worked at all.

    Paul: I wouldn’t say it’s quite from the early days, but relative to now it was quite early, I think they had that sin Cedar, which was around 2010.

    Edith: I just more meant when you’re an early stage start-up, sometimes you change your entire product.

    Paul: Well okay, yes, yes. I mean, absolutely. I think once you get into, once you get past the first stage of the product, and if you’re able to draw very good interfaces between how your customers understand what your product is.

    Edith: I don’t know, I mean I’ve seen this go bad in so many organizations, where you have entrenched engineering organizations that care more about staying on their current project, then actually about where the market is going.

    You know, like we’ve always worked on this, so we need to stay here because we don’t know anything else versus being able to evolve to where the market is going.

    Paul: Right. This reminds me a little bit of something that I’m working on at the moment. We brought in some UX experts to look at our app and to help us sort of transform it into something that was a little more usable.

    And they did a fantastic job and I spent this afternoon reading some of the reports. But what was difficult was understanding where the product needed to be. So for us in particular, there’s not enough focus on the deployment.

    There’s a lot of focus on the build and there isn’t really sort of a broader look at what do engineers actually do when they’re trying to do continuous delivery. And so we ended up with what was in the product was redone in a really fantastic way, but there wasn’t much affordance made for here’s the thing that actually needs to be in the product.

    And when you talk to customers, it’s hard for customers to tell you oh here’s the thing that you actually need to be or they look within the box you’ve drawn for them.

    Edith: Yeah. I say this ’cause I had a similar evolution to you, I actually started off in engineering. And when I was in engineering it was very obvious what we should built next, extremely obvious. And so I always thought that our product manager was an idiot for not seeing as clearly as me. When I became a product manager, I realized how myopic I had been as an engineer.

    Paul: Can you give an example of what was the next thing to build that in retrospect was wrong?

    Edith: I would see all the little bug fixes that we should be doing instead of the next big features. Or not even the next big features, but the next big product.

    Paul: I think big or small, or big picture versus small picture, is a good way to distinguish these two. I think that it’s very easy when you’re talking about product management to get the idea that a product manager knows everything and the engineer is just an implementer.

    And I think this is where a lot of the resistance to product managers comes from with an engineering organization’s the idea that they’re going to be relegated to mirror kind of peasants in the…

    Edith: Code monkeys.

    Paul: Code monkeys, there we go.

    Edith: You know, nobody wants to be a code monkey. That doesn’t sound very fun.

    Paul: Right, right. I would disagree with that, but I think that’s–

    Edith: Wait a second, we never disagree.

    Paul: It is very frustrating trying to understand everything. And on the other hand, it’s very satisfying to ship things and to get your stuff in front to customers. So very often, the ability to just be a code monkey for a certain period of time is this sort of soothing feeling of just shipping software that fixes a lot of small problems.

    I remember reading one of the famous GitHub guys, I don’t remember which one it was, but let’s assume it was Kyle Neath, that spends a lot of time on big projects. And in between the big projects, he needs to find what is the next big project to work on.

    And it’s often very frustrating, or very you go down certain rabbit holes and whatever, and you end up kind of not shipping things, or you end up getting frustrated or whatever. And what he likes to do then, is just reach to the back log and just take a bunch of small fixes. And he spent like two weeks of just like implementing very small things.

    You don’t need to think about it, and it’s cathartic and it lets you ship and so a couple of weeks as a code monkey I think is a very useful thing to sort of refresh the head and that sort of thing.

    Edith: I agree, but I think nobody wants to do that full-time and I’ll also challenge something else you said, which is everybody wants to ship.

    I think there are a lot of people who find shipping terrifying. and they’d rather keep holding stuff on until it’s perfect.. Like, I’ve certainly been with in situations like this where it’s like–

    Paul: Right, where we can’t ship because it’s not perfect yet or it’s not complete.

    Edith: Yeah, and as an engineer you have this real battle of well what if people want this, or what if they want this or this might be not quite right.

    Paul: Yeah. The personal strategy that I use to manage that is to try to write the blog post that you’re going to launch this with and to, cause very often you’ll be like, “oh I can’t launch this cause it hasn’t got this feature, it hasn’t got this feature.”

    And in the blog post, assuming that you’re going to tell people how to use it, or you’re writing the doc maybe, if not the blog post. You get that sort of feedback as you’re trying to explain to your user how to do this you’re going to say, all you need to do is this, and you’ll realize that this is seven steps long instead of one step long.

    Edith: Yeah, the Amazon model. So at Amazon, they actually start with writing the press release first.

    Paul: Okay, right.

    Edith: And everything and that’s a really good guide back.

    Because too often, people do the other end of they’ve built this gargantuna thing and they’re trying to write a press release or blog post and they’re like, woah, we’ve built all this stuff but there’s nothing actually to talk about.

    Paul: Right. And part of that, and something which I think engineers have a difficult time thinking about is how to get this widely in use. So you can build the feature, but its no use having built it if no one uses it. So you need to build the bread crumbs, you need to figure out the ways that are subtly hinted that this is the feature that you want when it’s the feature that you want and to draw people’s attention to it.

    And sometimes that’s putting it in docs and sometimes that’s doing a big announcement, but more often it’s trying to get the product in a place where the UX naturally implies the right path or the right direction for users to discover parts of the product.

    Edith: Yeah, I mean the whole idea of responsive design, and I think even more, and this goes back to why I started LaunchDarkly, is you might have built it, but nobody might want to use it.

    Like, you could have put all this effort into building it and done all these bread crumbs that nobody follows. So that’s the idea of you actually start doing the bread crumbs first and see if people start following that path.

    Paul: So with LaunchDarkly, I’m guessing that the way that you see whether someone is using it is whether it’s enabled for them. Is that right?

    Edith: Well no, so what we do is we allow people to turn on features for certain users.

    Just turning on a feature for a certain user doesn’t necessarily mean that they start using it.

    Paul: Right, exactly. So do you tie this to mixed panel usage? Or some sort of analytic stuff?

    Edith: We could tie it to different back ends, like we tie it to New Relic, we tie it to actually optimize so you can see if people are even, and we have our own internal analytics.

    Paul: Gotcha, okay. So this is the thing for me, that I started a project recently, and the first thing I did is built the dashboards for adoption. And we’re still at the stage in the project where there’s no adoption, or there’s tiny amount of adoption amongst early users.

    A trickle. A trickle that you can’t even see on the graphs.

    Edith: So it’s more like a fine mist.

    Paul: A fine mist. But what you need to get is you need to get to the place where everyone is using this. ‘Cause if you just build it, they’re not going to come.

    They need to be told about it, they need to understand how to use it, and getting those first customers to using it and where it’s deployed amongst them is it gives you incredible feedback about how one actually ships that software to the larger customer base.

    Edith: Totally agree. I mean, this is classic Lean principles of just making sure some people can use it well before rolling it out further.

    Paul: We discovered a part of the product that exactly three customers were using.

    Edith: How did that make you feel?

    Paul: Well, we didn’t actually know it was a feature. This was the idea that you could do deployments in parallel. So at CircleCI we paralyze your build and so the idea is that basically we take your test suite and split it across 20 or 50 or whatever machines.

    But it turns out that that applied to deployment as well. And there were exactly three customers using that. And one of them had a valid use case for it. Out of thousands of customers, exactly one valid use case was there.

    Edith: So what did you do with the feature?

    Paul: We killed it.

    Edith: Did you tell him?

    Paul: I hope so. Yeah, I know. I think we reached out to that guy. There was another way for them to do it.

    Edith:

     The painful part of project management is also when you have a feature that you like to kill but that a subset of power users loves.

    So like at TripIt, we’re a mobile travel itinerary but we let people do a printout. And one time we’re like, oh nobody prints anymore, let’s just kill it. Turns out that people print and they really really like printouts.

    Paul: Right, right. I understand that, yeah.

    Edith: Like particularly if you’re traveling to a foreign country.

    Paul: Yeah, you’re not going to have internet or your phone’s going to be dead.

    Edith: Or you want to show something to a passport guard.

    Paul: Yeah, or a local. Without handing over your phone, like here’s what I’m doing.

    Edith: Yeah, so they were furious with us.

    Paul: Right. So, had you killed it at that stage?

    Edith: Oh we killed it. Like we were just like oh, we didn’t have good analytics on people printing, so we just said oh nobody’s printing, let’s kill it.

    Paul: Oh, wow wow.

    Edith: So our analytics later was that people complained. Quite loudly.

    Paul: And so had you properly killed it at that point, or had you merely disabled it to see if it went away?

    Edith: Let’s see. We disabled it. I think we could get it back but people were really really upset.

    Paul:

    I like the thing of shipping something turned off, rather than actually deleting the code. Even though it’s incredibly cathartic to delete the code and to hide and it and remove it. But the turning something off with a feature flag is just a lot better way to sunset something.

    Edith: Why do you think it’s cathartic?

    Paul: Oh deleting code is amazing. It’s like my favorite thing to do.

    Edith: It’s funny. My cofounder John, he was from Ex Atlassian, and he said the winner of their hack competition was always the person who deleted the most code.

    Paul: Right, right. That makes perfect sense.

    Edith: Cause that’s what they wanted a reward is tidiness.

    Paul: Yep, yep. It’s very much related to the idea of product management and validating things and making sure that you don’t build too much of the product. Code is not an asset.

    Code is an asset in the financial sense of it in that you think you want it but you actually don’t. You actually want the best performance with the least amount of code slash asset available.

    Edith: Yeah it’s like, so a friend asked me once, should I pay my developer’s more if they write more lines of code? I was like, no. It was like no, that’s a really easy metric to gain.

    Paul: Right. So we were talking about deleting features by feature flagging them. I think that this is an awesome way to delete a feature because it’s very, very easy to get back. It’s much easier to get back than a rollback.

    Rollbacks are this thing that nobody wants to do because they’re very, very painful. Especially if a ton of code has come in between. So if you ship something, or if you delete something by literally putting a feature flag in around it, and then you ship the code and then it’s still on, and you turned it off for a certain amount of people, see if anyone complains, see that it still works and that you know, show it to ten people, and then you delete it for everyone just by flicking the flag.

    And then if you get someone saying, you know, we really, really, really need print it, You can turn it back on for them while you have a think about how you really want to solve this problem.

    Edith: Yeah, I think feature flags is a really misleading term.

    So feature flags implies that it’s always on or off, when really it’s more of a feature control.

    It’s a way to encapsulate a portion of functionality such that you have total control over it, from the sunrise of it, you know, from launching it to certain people, seeing their reaction, getting analytics, and then all the way to the end, as you said, every feature eventually you want to kill.

    For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.

  • To Be Continuous: Shipping Velocity and Team Structure

    To Be Continuous: Shipping Velocity and Team Structure

    In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk about shipping velocity and it’s affect on team structure.

    Edith Harbaugh, CEO LaunchDarkly

    Edith is CEO and cofounder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

     

    Paul Biggar, CEO CircleCIPaul is the CEO and founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

     

    [soundcloud url=”https://api.soundcloud.com/tracks/226489524?secret_token=s-Xkrrb” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Paul: Okay, so one of the things we wanted to talk about this week was the idea of shipping velocity and how it’s affected by team structure, sort of miscellaneous factors that aren’t part of continuous delivery.

    Edith: Yeah, I mean, can you explain a little bit more about what you mean by velocity first?

    Paul: Sure, so velocity is a sort of nebulous term that they use a lot in product management and that sort of thing to indicate how quickly, or the sort of through-push with which engineers and product teams are able to put through code.

    One thing that isn’t particularly clear, and there’s this amazing blog post by a guy called Phil Calcado who worked at SoundCloud, and he talked about the idea that even though that they had a very high velocity, they weren’t able to ship features very, very quickly, so often it would take 60 days to ship a feature even though no one got blocked.

    The idea of shipping quickly, is a combination of velocity and latency, and it depends on your kind of business goals, whether you need to ship correctly or you just need to keep the through-put up.

    Edith: Well, do you think part of the issue with SoundCloud might have been that they were trying to do too large features? I mean, that’s somebody’s first reaction when they hear it takes 60 days.

    Paul: So in the blog post what he said is first someone would spec out something, and then they’d put onto the next person’s queue, and that would be maybe a designer who’d drop wire frames, and then the wire frames would end up on an engineer’s queue to implement it, and in between someone doing work and someone else doing work, there would be, sometimes, you know, 10 or 11 days of just waiting for it to get to the front of that person’s backlog.

    So it wasn’t so much, you know, are you biting off more than you can chew, it’s just the process wasn’t particularly set up to be able to ship things very quickly.

    When he got in there, the first thing he did is he said, “Oh, you’re not doing continuous delivery,” or “You aren’t doing continuous deployment,” so he turned on continuous deployment and got that set up, and it was a 66-day process before he finished, and when he got continuous deployment set up it was a 65-day process.

    Edith: So all that work to save a single day.

    Paul: Right. Well, it was more about, the discussion was about how it wasn’t really about, continuous delivery. It wasn’t really about the time that people were spending on certain things, the vast majority of it was the time that was spent waiting on it.

    Edith: Yeah, that takes me back a lot to the whole idea of just in time manufacturing and lean.

    The whole idea of lean was to reduce waste, and one area of waste is time.

    Like if you have people on a factory line who are waiting for the next part, or you know, stuff queuing up, then you’re being very wasteful.

    Paul: I think this is how people, I think that that’s the thing that the backlog is intended to fix, that you’re never waiting for things because there’s always something in the backlog that needs to get done.

    But what isn’t considered as part of that is something is sitting in the backlog, for five days, 10 days, two weeks or whatever waiting for someone to address it, then the speed with which you can get a feature out increases dramatically, or decreases dramatically, I guess.

    Edith: Well, that’s the whole idea behind the Toyota factory line is that they, the just in time delivery meant that there was always a part right then, not before and not after.

    Paul: Right, okay. But right when they’re ready. But you couldn’t say, “I need a car now.” I think it’s not really optimized around the idea of how quickly something goes from one end to the other.

    It’s more optimized around how quick, or how much, how little waste that you can have in the system itself, which means, in this case, how much developer, engineer, or designer engineer and project manager time is being spent doing nothing.

    And I think you can certainly reduce that time to very, very low without necessarily optimizing the speed that which it takes to assemble a car, which is something that you need, really, to get to market quickly, and that sort of thing.

    Edith: Yeah, and it’s a trade off. You know, I went to a talk that the former CTO of Yammer gave, Adam Pisoni, and he talked about how on projects they always had an extra engineer. Like if they thought a project would take four engineers, they would actually put five on it.

    Paul: Right, right.

    Edith: For precisely that reason, that they never wanted anybody to be sitting around blocked. So they would rather take the cost of an extra engineer, just so there was always somebody to do the next task.

    Paul: Right. And so there’s an interesting thing about how teams are structured in that. In the Yammer example, they have teams, and teams do a project, or do a particular feature, and I presume that they’re cross-functional, there’s a couple of designers, couple of engineers, whatever.

    A different way to structure it is to have teams that are aligned on a particular, by what they do, and you have the engineers who are sitting at the front of the front-end queue, or who work on the front-end rather than work on a particular project which has components of front-end and back-end.

    And you end up with a very different structure, or it’s obviously a very different structure, but you end up with a different ability to ship, depending on how you’re, depending on what you’re trying to do, and what’s the most important thing for your organization, also how your organization is staffed.

    So the backlog works very well if you’re short-staffed in any particular area, and so you spend a lot of time prioritizing so that the people, maybe you’re short on design time, so that the design time is spent really, really well, but it doesn’t necessarily allow you to turn around a feature from start to finish and get it out to your customers incredibly quickly.

    Edith: Yeah, and that’s a decision that different organizations make. Some organizations say, “We’d rather have 100% utilization, even if it means our features take longer.”

    Paul: Right. My feeling is that most organizations don’t actually make that decision consciously.

    Edith: It’s implicit.

    Paul: Right. Oh. So, what do you mean by that?

    Edith: Oh, well so I worked at a lot of big enterprises, and my saying was always,

    The lack of a decision sometimes is a decision.

    Like, we’re not going hire an extra designer to unblock a lot of people, so therefore our feature’s are always just gonna be gated by that.

    Paul: So when you say it’s implicit, you mean that, I guess you kinda mean the same thing as I do, that people aren’t really thinking about the advantages or the disadvantages of the structure of particular teams.

    Edith: Yeah, that people are just kinda, they say, “Oh, everybody looks busy,  and why does a feature take 65 days?”

    Paul: So one of the structures that I really liked, there was a talk at Heavybit, and I think it’s on the Heavybit website by Peter van Hardenberg from Heroku.

    Edith: Oh, he’s so great.

    Paul: So in the talk he talked about that at Heroku they have product teams that are around specific areas. And the product teams have two engineers and a PM, and maybe a designer and that’s kind of roughly it.

    Teams have like two to five people, and they’re focused on a particular area, so they do the prioritization within their areas, they do the support and the ops within their specific features, and then they’re able to trade-off, or they’re very, very close to the problem.

    They have then project managers who are working to protect them from other areas, or bleeding into other areas, but also working to solve their particular problems, rather than necessarily one backlog for the entire, maybe the entire platform or the entire web, or something along those lines.

    Edith: So basically, little small start-ups within a bigger company.

    Paul: Yes, and I think actually that’s how they structured it at the start. I remember them saying that there was a, they would actually have presentations to the board, and that each of the teams was structured, the project manager was like a CEO, and they actually like gave presentations to the board, and I think that didn’t necessarily last very long, but I think that was the initial idea behind it.

    Edith: Yeah, we did something very similar at TripIt. We had a, you know, we would have a new project we wanted to do. We were trying to do TripIt offers, and we had a project team just for that, and that’s what they did.

    And the idea was that they could focus. I’ve also heard of people who do stuff completely different. Like I talked to a guy who ran an agile shop and every project had to be a week long, and nobody had any ownership whatsoever.

    So he set it at, and not even like ownership in the way we would think of it. He said our front-end people were back-end, people were mobile, also.

    Thank you for listening, the entire Heavybit podcast can be found here.

  • To Be Continuous: Evolution of Continuous Delivery

    To Be Continuous: Evolution of Continuous Delivery

     

    In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar discuss the evolution of continuous delivery. This podcast is brought to you by Heavybit, a 9-month program for developer startups.

    Edith Harbaugh, CEO LaunchDarkly

    Edith is CEO and cofounder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

     

    Paul Biggar, CEO CircleCIPaul is the CEO and founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

     

    [soundcloud url=”https://api.soundcloud.com/tracks/224359194?secret_token=s-rlyTD” params=”color=ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Edith: So Paul, where did continuous delivery come from?

    Paul: This is one of those things that evolved over quite a long time but I think that that major catalyst of continuous delivery was the fact that we moved to SasS and that SasS now exists.

    Edith: So what was it like before there was continuous delivery? What was it like when you were starting out?

    Paul: The first company that I really worked for was a games company and I think that a lot of them are kind of in the same place now. You would release software on a disk that shipped to customers via stores and you had to have a gold master at some point. We were a startup so it was quite advanced.

    We had a daily build server and we actually were selling middleware so we shipped software to those companies but those companies would ship at the end of the day, they would ship a disk that went out once and there was no updates at the time.

    It was 2003, 2004, something like that. There wasn’t any way to automatically update on customers’ Playstation 2, unlike even games now have a little bit of continuous delivery but back then, that was, you got your chance to ship it and that was it.

    Edith: Yeah, I just had a brutal flashback to one of my first summer jobs, which was installing Windows 3.1. This is back when you actually had a set of 35 disks so my job as a summer intern was to go to every computer in the office and just literally sit there. Disk in, disk out, disk in, disk out, and if something went wrong on the install, I had to start over at the beginning.

    It was a two hour long process just to get everybody up to 3.1 so in those days, there was no change of a fix pack. I think everybody was going to be on this version until probably they got another sucker summer intern in there.

    Paul: So if you imagine people and their, people installing, even modern or relatively recent version of Windows where you have to think about the service pack one or service pack two or what level are you patched to, that was basically the old way of doing software.

    The first time that I remember realizing that there was a different way to do it was in a Joel on Software blog. One of the major things that they could do is that there was a bug, they could fix it and in the cloud, it would be fixed.

    Edith:

    Yeah, I think it’s good though to realize that the reason that there was so much emphasis back then on making sure that a build was correct was because there was such heartache and pain if it was wrong.

    Paul: Yeah, it made sense to hold up the release for new features or for something like that if it was promised to a customer because then, there was no alternative. It would be three, or six or twelve months later when they got the next opportunity to go ahead.

    I think frankly, building software in the cloud and having the possibility of doing a release at any time you like has actually just completely revolutionized how people make software.

    Edith: I completely agree. So to go back to the old days, all the emphasis was on extremely high quality and extremely stable.

    Paul: Right, right.

    Edith: Because you basically had once chance. You were lobbing this thing out into the world.

    Paul: And even things where they had quarterly release cycles or something like that, it was still, it felt a lot like one chance. I worked at Mozilla and I was there on the switch from Firefox 3.0 to Firefox 4.0. Firefox 4.0 took 18 months to commit and there was things that were ready to be released the first week of that release cycle and that had to be held up for all these unrelated releases.

    I think there was problems with the JavaScript engine and there was memory bloat and that sort of thing that kept pushing it back. And Firefox, during the process, or during the time that I was there, switched its process and it switched to what they call a release train model where basically, they ship a new version every six weeks and it doesn’t matter what’s shipped or what’s not shipped.

    They’ll hold it up for total life, or build breaking, it just doesn’t work at all but basically everything else would make it go ahead. The cadence and the velocity that they’re able to get since that happened has been massive.

    Edith: Yeah, so that was the first evolution.  It used to be let’s try to cram everything in and then you’d get this vicious cycle where it’s like well let’s just put this one more thing in and then you’d have to add on all the QA time.

    Edith: So that was the other reason why releases would take so long is you had so much QA.

    Paul: Interesting.

    Edith: I talked yesterday to a guy who had worked at Microsoft and he said, you know, quality had to be rock solid for Microsoft going back to my disk example.

    Paul: Right, and these are people who have unit testing, automated tests as part of their builds.

    Edith: But even beyond that, you have to have a person actually try it and then you have to actually go out and try it on all these different systems.

    Paul: On all the different systems, right.

    Paul: So one of the things that made it possible for the browsers to do, to move to the release train model is that they had the alpha channel and the canary channel, or whatever it is they called it. In Firefox, it’s the Aurora channel.

    That’s the super, actually that’s not even the super beta stuff, they even have a nightly channel. They have a million people using the nightly and that tells them whether things are broken and how they are in fact and in production. I guess it’s kind of the first evolution of feature flags.

    Edith: Yeah, I mean let’s talk about that for a second.

    Paul: Right, we probably need to, aren’t we going to define things in this one?

    Edith: Yes, this is the definitional episode, Paul.

    Paul: Right, let’s start with what’s a feature flag because I think we’ve already talked about six things that we haven’t defined but what’s a feature flag?

    Edith: A feature flag is basically the ability to hide code but yet have it public. So it used to be that everything that shipped was visible. So a feature flag is just basically an if then that wraps a piece of code and says it might be shipped but nobody can see it until the software team decides that it’s visible.

    Paul: Right, and so some examples of that could be in a browser that you change a preference and the preference is use the new version of this or another example might be in, that a feature goes out in Facebook that’s shown to 1% of customers in Northern California who are in this demographic. Yeah, and it’s very liberating.

    Edith:

    The very liberating thing about a feature flag is that if something’s not working, you don’t have to do a redeploy.

    So to go back to the old dark days where you had all these disks out in the world, the feature flag is useless because you’re going to ship all these disks out to people and they’re going to install it. A feature flag is useful when people are consuming stuff as SaaS.

    Paul: The way that I think about a feature flag is that it separates the active shipping code from the act of actually turning on that code.

    Edith: Yeah, deployment from roll out.

    Paul: Right.

    Edith: Yeah, so it’s deployed, it’s available, but it’s not visible.

    Paul: Right, so now that we know what they are, what were we actually talking about?

    Edith: We’re working our way through the evolution of software so we got up to release trains. You should talk a little about how continuous integration fits into that.

    Paul: So continuous integration, to define it, it’s basically the act of having an automated build server that runs every day or at some sort of cadence and tests that your software works.

    So CircleCI, my company, makes continuous integration as a service and what happens there is whenever you push to GitHub, we’ll automatically check your code out, we’ll build it, we’ll run your tests on it, and we can also deploy it to your customers, which is called continuous deployment or continuous delivery, depending on exactly how you think about the world.

    Edith: You make this seem so simple but it’s really revolutionary. I remember the days without this.

    Paul: Sure, sure. I remember in my first job, we had a daily build server and so we would find out every day whether or not the code actually compiled, even though we could get it all compiling on our own machines or some modules of it or whatever, compiling on our own machines, there’d be all sorts of build artifacts and dirtiness in the code that meant that this code didn’t actually work when it was ready to be shipped to customers.

    Edith: Yeah, or even worse, I remember it used to be a thing when you broke the build, it was very shameful. You would have to sing a song or you’d have to do something embarrassing and that just doesn’t really happen anymore.

    Paul: Right, right. My co-founder used to forget to add new files to Git so it constantly wouldn’t work although it was perfectly working on his machine but there’d be a file missing somewhere.

    Edith: The classic excuse of, “Oh well it worked on my machine.”

    Paul: Exactly, yeah.

    Edith: Yeah, I didn’t know about builds my first summer job. At my first summer job, I was programming in Visual Basic.

    Paul: Okay.

    Edith: I was basically making a war game. It was for a defense contractor and I was trying to justify a space based laser versus an airplane based laser and if you had all these missiles coming over from Russia. Okay, this is a long time ago.

    Paul: Yeah, yeah.

    Edith: How many could you shoot down? The thing worked perfectly locally and then at the end of the summer, they wanted it off my machine. I was just like, “Oh.”

    Paul: Okay, you’d need to put it on a plane or in space or one of those things?

    Edith: No, they wanted my actual files off my machine.

    Paul: Right, yes, in order to put it in space or on a plane?

    Edith: No, it was just a simulation. I wasn’t actually controlling missiles when I was 19. Though that would have been cool.

    Paul: I had this absolutely terrible release process at this place that I worked. It was O2, which is a really large phone company in the UK, in Ireland, and I think they got bought by someone recently but whatever.  They were launching this feature called i-mode and i-mode is this massive thing in Japan that completely failed to take off anywhere else.

    Edith: Are you saying it was literally big in Japan?

    Paul: Literally, it was big in Japan. And so they tried to launch this in Ireland. Everybody in the whole world knew that this was going to be a failure, but that was how the process worked anyway. And the release process was they had all this code somewhere that was going to be deployed and attached in servers and when they did that, there’s a manual. I was part of the manual team that manually validated with handsets, clicking through handsets manually, and vouched that this stuff actually worked.

    There was no automation. The entire release process had to be repeated dozens and dozens of times as they fixed bugs and they reconfigured. It was, it was just a complete cluster.

    Paul:

    When I moved to doing software that had any sort of continuous release cycle, it was liberating because you actually know the path from what you’re doing on a day to day basis to it arriving at customers. Everyone, every developer in the place can know what that path is and everyone knows how code acutally gets shipped. You don’t have to keep it in a continuous state of you know, oh maybe one day I’ll ship this.

    Edith: Yeah, I mean I think to go back when you had these longer and longer and longer release trains or even pre-release train, people got very relaxed.

    Because it just was like oh, I’m just going to sit in the corner and code away and sometime in the distant future, it might ship.

    Paul: Right, and then as you’re coming towards the end of the quarter, everyone is hammering things out as quickly as possible and maybe there’s a few things that shipped early in the quarter that never got cleaned up or never got quite ready.

    Edith: What do you mean by shipped right now?

    Paul: So I remember sometimes where you’d merge something into master because you needed to get it tested but it wasn’t quite production ready. It’s out with the beta testers or something like that, but in order for it to actually ship to hundreds of millions of people, you need to remove whatever hacky thing you put in place.

    Edith:

    I think one of the things that has driven continuous delivery is just the rise of test-driven development.

    Paul: That’s interesting. Yes, I mean, I think it’s obvious. You end up in a situation where when you’re trying to move to continuous delivery, you’re saying I’m afraid and we talked about the fear last week.

    In order to get past that fear, you need to have automated testing and I guess in the old days, there was manual release cycles and you can’t do manual release cycles 10, 20 times a day. It’s just not possible.

    Edith: You can’t depend on having a fleet, like to go back to the Microsoft example, you can’t rely on having two QA engineers for every developer who will, you know, you just fling out code and they’ll find everything.

    Paul: Even with two QA developers, you can’t have two QA developers when you push code every five minutes.

    Edith: Yeah, they’ll never keep up.

    Paul: Exactly and then they’ll never identify the change that actually caused the bug and so on.

    Edith: Yeah, I think the things we’ve talked about so far, SaaS has unlocked continuous delivery, test-driven development has unlocked it. What else do you think is really a foundational thing?

    Paul: I guess the other one is kind of this change in customer expectations. Customers, as a fact of some people doing continuous delivery, everyone else has had to do it so once you start to experience it a little bit, and you start to wonder why do other sites go down for six hours on a Sunday night, this doesn’t make any sense. Why do I have to download new versions of this software versus clicking the update version or why do I even have to think about this?

    Edith: Yeah.

    Paul: I think for the installable software market or the desktop software market, Chrome was that sort of killer app that everyone sort of realized that this has completely changed the game.

    Edith: Yeah, it’s completely changed the game.

    I think even now asking people to download a plugin seems too onerous. They don’t want to do it. They just want stuff to work.

    Paul: It’s almost tied to a conversion funnel. If you expect people to install 31 disks to upgrade to the latest version of Windows, people are just not going to do it. You’re not going to achieve the conversion rate that you’re expecting. There is too many alternatives to your app that are out there.

    Thank you for listening, the entire Heavybit podcast can be found here.