devopsy.kinsta.cloud

Author: Malia Powers

  • ‘Don’t Make Me Code’: Affordances for Developers

    ‘Don’t Make Me Code’: Affordances for Developers

     

    Thanks for listening to “Don’t Make Me Code,” a Heavybit podcast that discusses developer experience design and the unique challenges of building developer-facing products. Don’t Make Me Code is hosted by Steve Boak, co-founder and CPO at Opsee, and David Dollar, co-founder and CEO at Convox.

    In this episode David and Steve are joined by Dustin Larimer from Keen IO. They discuss affordances—the visual cues or attributes of a thing that inform you how it works – for developers.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/251905881?secret_token=s-mL7jb” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Steve: Welcome to “Don’t Make Me Code,” episode four. We’re calling this episode “Affordances for Developers,” and with us today is Dustin Larimer of Keen IO.

    And he wrote a really excellent blog post about how they refactored their code based on some documentation and we’re gonna get to that in a second. I think, first though, since we’re talking about affordances for developers, we need to start with what are affordances? Since not everyone may know or realize what they are. And so I don’t know if you want to kick that off, Dustin?

    Dustin: Yeah, sure. It’s a really cool concept. It comes more from product design and industrial design. We’re talking about the visual cues or properties or attributes of something that tell you how it works.

    Like a really great example, like we’ve got buttons and knobs and dials and levers all around us in our environments and when you glance at it you can kind of tell. Like, “Well, this flips up and down,” or, “I can push this in and push it out.” And so, with physical products, it’s possible to embed a lot of knowledge about how something works simply in how it looks and that’s the idea of affordance, is that the thing kind of teaches itself to you.

    You don’t have instruction manuals for coffee pots, unless it’s some kind of crazy new thing with coffee pot. Or, like, tea kettles or things like that. You know, if there is an instruction manual, it’s probably been poorly designed. Because it’s considered successful and you can just, kind of, figure it out. There’s not a huge learning curve to make it work.

    And this gets really interesting with software because we’re moving from design of physical things to the design of, pretty much, invisible things.

    Steve: Yeah, and in a digital space, we have things that we call affordances. Things that look like buttons or look like links and we’ve moved from these skeuomorphic designs to flat designs and a lot of that stuff, I would argue, is getting harder.

    We have to rely on patterns that we know and trust and I honestly can’t tell, sometimes, if that’s getting easier or harder as we get better at this.

    Dustin: Yeah, totally. Like early on things like tabs were kind of based on the little tab of a filing cabinet. Like filing sheets. Because people were familiar with those in the analog world and so it kind of transferred nicely into the screen world.

    You know, things like drag and drop were brand new back when, you know, operating systems became consumer-friendly, so how do you teach people about drag and drop? So the little mouse cursor was a hand that literally grabbed and let go of things. So, it’s kind of a transition and the skeuomorphic era was largely to introduce mainstream public to how these interfaces work.

    ‘Cause they were familiar with how they, you know, they’ve seen them before so if something looked like a big ugly dial on the screen, you knew it could turn. But then, I guess Apple’s a good lead for moving away from that with their big redesign a few years ago. They justified it by saying, “We’ve moved into a time where enough people have learned about interfaces and screens.” We have a larger shared competency around these things so we don’t really need to make things that painfully obvious and rooted in the analog world anymore.

    People have a little more capacity for figuring this stuff out and so they could move into a new world where there’s not really a lot of basis for how these things work. They just invent new patterns and it succeeds or it fails. So, a hamburger menu was one of those. Everybody agreed it worked and now they’re everywhere. To a detriment, I think. But things like that, these new languages, new vocabulary kind of goes mainstream. New words become very popular or fall out of favor.

    Steve: Yeah, and they come and go kind of like fashion. The hamburger menu, again, being a good example of that. I feel like there’s been a lot of backlash against that because of the cost of using it that we only realized after playing with the implementation of it for a while.

    Dustin: Yeah, it kind of also became kind of a crutch. People didn’t have to think so much about their information architecture because they could just hide it all behind a hamburger menu.

    And that was the mobile, a mobile invention that made it into desktop and larger screens and a lot of the usability testing around that and, sort of, analytics around things like that find that it’s not a great idea. You know, it’s not a very effective design feature to just paste in arbitrarily anywhere. You kind of have to keep thinking about things.

    David: It is interesting to think about though, because even if you do end up with a bad design, like if everybody comes on the same design, then there is some sort of value to that, right? Like, even if the hamburger menu is terrible, now that I’ve encountered my 550th one I kind of know what you expect of me here.

    Dustin: You figured that one out. Yeah, totally. That kind of becomes part of our shared vocabulary around these interfaces.

    Steve: 
    And in the digital world, it’s kind of one abstraction away from the physical world, but then dev tools feel like a whole other level beyond that. Because now there isn’t even a clicky UI, necessarily. There’s nothing to latch on to, visually.

    And that’s part of what made your story so interesting. And maybe we can start talking about that now. Like, what was the problem you were trying to solve and how you tackled it.

    Dustin: At Keen IO I’m on the Experience Team, I’m a product designer, I’ve been doing engineering, also, for a long time. Kind of both interchangeably and when I came to Keen IO I took over all of our Java Script tooling, data visualization, dashboards and all that stuff. And so, that was the first time I was ever really faced with having to shepherd along a product that’s made of code and designed for people. There’s a lot of interesting moving pieces to that. But the idea was that we could keep adding new features and tacking new things on and just calling it whatever we wanted to and sort of letting feature bloat run away and do that or we could try to consolidate and shrink the footprint.

    A big challenge is that when you’re designing a product you like to believe that it’s this really important crazy thing that people are going to spend a lot of time on and love, but then you have to recognize that you’re just one more vendor in their stack. You’re just one more thing to integrate so they can get on to doing what they really care about and you can’t afford to require people to learn about you. I think that’s going to become very competitive of developer facing companies is that time to value.

    How long does it take for me to figure out what the hell you’ve created and how to use it and how to plug it in. And so, looking at that, we started designing with that in mind first. Like, how quickly can somebody find the code, install it, wire it up with minimal impact, minimal footprint and start getting immediate feedback that it’s working. And so we can’t just keep tacking features on and expanding the docs forever, we have to start shrinking at some point. So, with Java Script SVK where I’ve put all of my time, that’s a huge factor, it’s our most popular SVK.

    Most of the data that goes in and out of Keen IO goes through that. Most of it’s coming from browser traffic and that sort of thing. So, its got to be super effective at people being able to learn it, install it, use it. And a big part of that is having confidence to know that the thing you’re using, or the way you’re using it is the right way. That’s another big one. Where a lot of things fall down. Where you’ve set it up and you’re running it, but you’re not totally sure if you did it right. The things, the names of methods, and the way that things are called. It’s sort of ambiguous and it leaves you wondering if you’re gonna get surprised one of these days, like you just did it wrong.

    Steve: And we’ve talked before on the show, too, about the importance of boiler plate code and CLI’s and examples. Like, all the things that you rely on. And so for your Java Script API, this was just all in one place, it was one giant document?

    Dustin: Yeah, it was one monolithic library. One big document. It did tracking, recording events, querying, running analysis and then visualizing it all in one monolithic pile and we started breaking those out into their own standalone libraries and we wanted each one of those to be very purposeful and effective and super easy to glance over and get it.

    So, another thing we’ve run into with SDKs is the method name to log and event or just save an event into Keen IO is called a lot of different things. Like cross different SDKs it’s add event, or record event, or push event or whatever. The common thread being events, we talked about events, which is another point of confusion sometimes.

    And so we thought, “Well, let’s just shore that up.” Let’s only say record events and let’s say it everywhere and that way when you see something that says record events you know with absolutely no hesitancy whatsoever that this is the thing you’re looking for, you’ve found the right method, this is the way to do it, it works in a predictable way.

    So it’s really consolidating our vocabulary and shrinking the footprint so that it makes our messaging more effective because it works across all these different languages and it helps us focus the message and give people confidence that they’ve found what they’re looking for.

    So they don’t have to spend hours hunting through docs and looking for examples and asking questions on stack overflow to get that assurance that, yes, you’ve done it right.

    Steve: I thought one of the interesting things about your approach, too, was the team that you put together. It sounded like they’re developers, designers, product managers, people from all disciplines in the company that got together to work on the problem. Can you talk more about how you approached it?

    Dustin: Totally. Yeah, so part of that is to do with Keen IO itself. So, we’ve always been hedged on generalists. Mission-driven generalists, specifically. Where we like to have people come in who might have some, in most cases do have some, really great specialty and competency in one important area, but have a lot of lateral interests across other areas and have spent time doing marketing or content strategy or design, in addition to engineering.

    We find people who were doing one thing in life and then became platform engineers. And so, they bring a lot of really amazing breadth to the perspective when we solve these types of problems. And so, on my team, experience team, we are part of engineering, but we’re the part that makes the API fit for human consumption. So we built apps and interfaces and tools, developer tools, point-and-click user tools, all that parts of the website that people interact with when they log in. So we own the developer experience and the, just general point-and-click user experience. We want people to find what they’re looking for and get value fast. And so, to do that, we have some amazing engineers on the team. Some really brilliant engineers.

    A lot of our stuff specializes in Java Script, ’cause that’s just the environment that we spend most of our time in, we build stuff in. But also people who have also done product management and design and content strategy and things like that in the past. So, we’re a very small team with a lot of very broad experience. And I think that important, to sort of keep breaking yourself out of the bubble where you might only see things inside of one paradigm and I think that’s where you get caught.

    Steve: Yeah, and you were talking about the need for consistency, both in the Lexicon, and in the function of the tools. And so, it sounds like you ended up figuring out a logic to break what was one big monolith into pieces. And there was some pretty logic, it sounded like, that came out of that. So can you talk about the results?

    Dustin: So the way we went about it first is we used the idea of breaking it out and having standalone SDKs for these three functions. For tracking, querying and visualization. We looked at it as a chance to, kind of, break away from some of the patterns of earlier stuff and try to build, like the ideal and then try to work our way back and make it somewhat backward-compatible. But we took the chance to really get our thinking clear first and to that.

    And that’s where the thesis and that blog post came from is if we were building physical product, we wouldn’t start building a tea kettle by bending aluminum and hammering something out in the shop. We would probably just get a big piece of paper and start sketching and start thinking through all of the different what-ifs and maybes and to really push the forum of that until we could remove all the things that weren’t necessary and we could focus on what mattered.

    We could get our thinking really, really clear. And so, we did that with these SDKs, also, where we realized that software is more or less invisible in that sense, but its form is its documentation. So we started out just sketching docs, just doing install guides, writing pseudo code for how to install and run this thing. Looking at all the different method names and functionalities people wanted and making sure it was consistent, you know?

    It’s like if you, you don’t want the situation where if you use this method, it gives you a callback with these arguments, and if you use another one, it gives you a callback with different arguments, or maybe no callback with no explanation whatsoever.

    So we wanted that consistency to become part of the vocabulary for how the thing functions. And by doing that, we came to a very, very clear spec of not only how the thing works, but how I use it. How I install it and run it and under what examples and situations. So we wrote a lot of example code. And all this was before we wrote a single line of code.

    The beauty in that is once you have that perfect clear picture of what you’re building with good rationale for why it needs to exist, putting the code together, actually took no time at all.

    Steve: Yeah, and it sounds like a lot of the code was already there and you figured out a really nice way to sensibly reorganize the documents and then you reorganized the code to fit that.

    Dustin: Yeah, yep. So we started by designing our thinking and designing the conceptual form of it and then filled in the rest. And really wrote a lot of tests before we even wrote the code itself. So that really changed the way that I thought about making software. Where, now, if I ever catch myself getting stuck on something new, I realize it’s because I’m looking at a very granular level and kind of pinning myself into a corner, so stop, drop it, back up, and start writing out pseudo code and docs for it and then it makes sense.

    You can follow the show on twitter @dontmakemecode and email Stephen and David at dmmc@heavybit.com to suggest a topic or guest for the show. You can also subscribe to all of our shows by copying this link into your favorite podcast app. Stay tuned for show-specific feeds!

  • To Be Continuous Podcast: Staging Servers Must Die

    To Be Continuous Podcast: Staging Servers Must Die

    Welcome to To Be Continuous,  a show about continuous delivery and software development hosted by Paul Biggar, founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this episode, Edith and Paul discuss a blog post by Edith, in which she asserts that you should kill your staging servers so that continuous delivery can live.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/248544923?secret_token=s-fUn5T” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

     

    Paul: So today we’re gonna talk about a post that Edith wrote. The post is about the death of staging servers. Can you give us a little bit of an introduction to what the post is, because I think we’re gonna talk about it quite a bit.

    Edith: Sure Paul, happy to. By the way, did you read it yourself?

    Paul: I confess, I may have skimmed it, a little more than read it.

    Edith: Yeah, so I’ll give you the summary. And then we’ll talk about it. And then you should go read it, and then we’ll talk about it again.

    So the article’s an outgrowth from what I was seeing with LaunchDarkly, my company. So already one of the comments I saw on it is, and I like this comment because they talked about me in the third person. They said, “Harbaugh is biased.”

    Paul: Right, I would say so.

    Edith: But by the way, she does have a good point. So I’ll state upfront, yes, I’m very biased. My company makes a platform to manage feature flags. So my bias, though, is I’ve seen what our customers are doing.

    And it’s the reason I wrote the article, is that this was a national extension to the way I saw our customers using feature flagging.

    Paul: So were your customers using your product wrong, or was there a thing that was happening, that you’re talking about?

    Edith: So feature flagging at it’s most basic is very simple. It’s just an if-then statement. You don’t need LaunchDarkly or any sort of system to manage that, you can just put a conditional in your code. What happens after that is you get more sophisticated. And you wanna have some sort of dashboard where you could see the different feature flags that are visible and uplevel them. Not even just to the business side, but even amongst the various developers.

    You want to have a central place where you can manage. That was kind of our first version, is just you can manage feature flags.

    The next thing that our customers were asking for in our road map was to support for environments. They wanted to be able to have feature flags on dev, QA, staging production, have visibility all in one place.

    Paul: I’m a little confused when you say this, because surely an environment is just a flag?

    Edith: Yes, go on.

    Paul: It seems like you could say, have another flag that is a string, that is like staging or production or Dev or whatever. That you flag features on.

    Edith: And that’s basically what LaunchDarkly is doing, so for us, we just have another flag.

    So each environment gets an API key.

    Paul: So again, forgive ignorance on this, why not use the existing flag?

    What was missing from the existing feature flag infrastructure that meant environments had to be a separate top level, or first class feature?

    Edith: So now we’re getting a little meta. The reason why people liked using LaunchDarkly is because they can get a consolidated view of all their feature flags.

    So that they could not just have these floating around fig file for each machine, but have a roll up of what flags are turned on and off for different environments. And then on top of that the ability to manage the flags in different environments.

    Paul: Got you, so they wanted a view that applied to just their environments?

    Edith: Well, to see per environment what was turned on and off. So basically they were following a software life cycle, where you had dev people working on the developer boxes, pushing a QA, pushing a staging, and then pushing to productions.

    Paul: When features moved from one to the other, was there a human promoting these, or was there an API call promoting these?

    Edith: At this point it’s mainly human. I could see down the line that it would just be an API. And even beyond that … The next thing people ask for, once they got more people on board LaunchDarkly, we’re using more feature flags, is they wanted the ability to have lockdown of different flags in different environments.

    Paul: Lockdown like people can’t change the flags?

    Edith: So they wanted QA to have rights, and QA to change some flags, but not in production.

    Paul: I see, okay yeah.

    Edith: Then after that the natural step then is okay, if you have a feature flag in system where you can control any step, who sees permissions?

    Who gets to see what? Why do you really have a separate QA staging and production box? Why not just collapse all this, and manage visibility with a feature flag itself?

    Paul: Sure. That makes perfect sense.

    All the feature flags provide the primitives on which you can build the things that they want.

    Edith: The point I made in the article is that

    people use the substraction of a QA and a staging environment to basically try to encapsulate a change. So that was the original intent, and at the time it was very good, because the alternative was just to push everything to production to have everything break.

    But if you’re actually doing—

    Paul: If you’re doing feature flags properly, then the concept of really having a staging server doesn’t make that much sense.

    Because you’re not pushing a feature to the staging server, you’re pushing a feature to production and then slow rolling it to people and really having it in the staging server doesn’t make that much sense. Is that kind of the point?

    Edith: Yeah, so that was the title of the article, Kill the Staging Server.

    Paul: This might be a good point to actually stop and read the article.

    Edith: Yeah, I’d appreciate that. Thanks, Paul.

    Paul: We’re gonna take a, what’s gonna appear to be a 10-second break, but it’s actually going to be a five-minute break, while I read the article. I recommend you press pause and read the article yourselves right now.

    Edith:  So welcome back, Paul has been furiously whiteboarding. I’d love to hear his thoughts now.

    Paul: Yeah, so it’s a good article, I really liked it. The first thing that came to mind was the concept of names.

    So the idea of having multiple environments is a weird one in particular. So imagine that you have … Service rented architecture, and it’s got like six machines or something like that, or it’s got 20 machines or whatever. The idea that you can’t spin up a new instance of it is kind of a little bit odd.

    Edith: Yeah.

    Paul: So what people have done historically, and this goes back to when things ran on machines, or particular ports, or whatever, was that everything had a name. It had DNS name or had a staging name.

    And if you look in rails, configuration files, there’s a production name, there’s a staging name, there’s a dev name, there’s different environment variables for all of these. But it doesn’t actually make sense to have a name. Because a name implies there is one of them.

    Edith: Well this goes back, are you going down the whole pets vs. cattle thing?

    Paul: I’m going down the pets vs. cattle thing, yeah. If you’re naming your pets, then you can’t just suddenly get six of them, you can’t suddenly kill them. But what you really want is cattle.

    So a staging server is not, I mean a staging server is a pet. And a very very important pet that everybody loves, and everyone plays with, and it gets a little bit confused as a result of it. And that analogy went—

    Edith: I’m not sure if you’re agreeing with me, or disagreeing with me, so. We’ll continue, it’d be great if you disagreed, but I would also enjoy it if you agreed.

    Paul: If that analogy holds, then what you’re suggesting is that we killed the cherished family pet.

    Edith: Well, you know, sometimes … aw, I can’t even go there.

    Paul: Well the analogy breaks down, but let’s say it’s for the best if that pet spends some time on the farm.

    Edith: Went for a long walk.

    Paul: Right. So obviously production is an actual thing that needs a name, it is a unique environment. But nothing else is really unique environment.

    Edith: Yeah, and to fast forward, I think, I got a lot of feedback on the article, which I loved. That’s kind of why I wrote it, for feedback. I think there are cases where you do want to not go directly from a developer to production.

    I think there are many cases where you want to have other places to test them.

    Paul: I do agree with this, yes.

    Edith: I think a lot of the cases though, of a forced march of we go from this step to this step to this step to production is actually very harmful, when if you just pushed off much quicker and perhaps skipped some of these steps, you’d get the feedback you want directly.

    Paul: So what we did at CircleCI, and initially we had a staging environment. And we would occasionally use the staging environment if we want to test something that wasn’t that easy to write unitests for, or something along those lines. And usually for us that was stuff around LXC, or stuff around starting Amazon boxes.

    Things that you didn’t really do that often, and that either had an expense, or had a weird architectural thing, where you couldn’t just do it in software, and practice it in software. So I think there is that need. Occasionally you’ll have things where it’s not tested well enough, so you need to put it up somewhere where a human validates to the best of their knowledge that it actually works.

    And you can put that in your, I was gonna say you could run it on, but the whole point of the staging server in that situation is that it’s something which isn’t really that easy to put into VMs, or whatever. The other kind of use case that you see is where you don’t want to be running stuff on live production databases, and you can’t get a copy too quickly. So you see people like Haroku trying to build products that avoid the need for that.

    So with Haroku … You can take a copy of a database, you can have a view or a read only view on a database, that’s a copy on the right in some way or whatever. So that you don’t need a separate staging environment, or separate copy or separate staging database or whatever.

    But if everything is well-tested, if everything works in software, then there’s really no need for staging environments. So that says to me that staging environments is a sort of a yellow flag somewhere. It shouldn’t really exist, but sometimes you might need it.

    Edith: I actually want to write a follow up article now, of the staging server is dead, long live the staging server. Because I do think that there are used cases where you don’t want to push to prod. I do however think people—

    Paul: What are these cases where you don’t want to push to prod?

    Edith: So what I heard from Sean Burns, our advisor, is if you’re testing a really deep infrastructure change, for example switching some batch processing.

    Paul: Mm, no, I don’t believe that for a second.

    Edith: Alright.

    Paul: So if you’re testing a really deep infrastructural change, there’s two ways to do that. That deep infrastructural change. One of them is to say, “We’re gonna have these machines over here, we’re gonna have these machines over here.” And that’s the quarterly release cycle thing. We’re taking a big big risk, all hands on deck, whatever.

    The way that you want to be releasing that sort of thing is that you wanna have it in the code base. You wanna have an if statement, a feature flag, that controls how much of the data goes one way or the other, and you duplicate the data, or you put 1 percent of it through.

    Edith: I think the key there is to duplicate or have some fail safe. I mean the biggest risk you run with doing this is data loss, which is awful. If you’re cavalier about how you do this—

    Paul: Well, I don’t think data loss is the worst thing, the worst thing is your whole damn service goes down.

    Edith: Well, the worst thing is your whole service goes down, and you lose a case worth of somebody’s data.

    I think people are actually more forgiving of a five-minute flip than you losing like a lot of their analytics. So his point was he had been at Flurry, was that we cannot afford to lose people’s data.

    Paul: So it would be ludicrous then, in my mind, to create a whole brand new infrastructural thing where you’re gonna do some kind of overnight or immediate change, or wait for downtime change or something. If the data is all live, you can’t afford to have, we’re gonna switch over and see if it works. Regardless of how well it’s tested, like it’s ludicrous.

    Edith: Yeah, so that was, now I’m coming back around to my article. I mean that was the point I was making.

    People think that they are reducing risks by doing all this tested staging.

    Paul: Yeah, in fact they’re increasing risk.

    Edith: Yeah, you would think you’re—

    Paul: Well, the whole point of continuous delivery is that by having a harsh cross over between one thing and another thing, you increase risk even though you think you’re decreasing risk.

    Edith: Yeah, so Ket Beck wrote a really good article, about reversibility. So he’s at Facebook now, but he said everything at Facebook is reversible.

    Paul: Interesting.

    Edith: That this actually makes you much less risky, ’cause you’re like okay we make these risky changes—

    Paul: But they’re reversible, so it’s, right.

    Edith: Yeah, vs. the more cutover. As I called in the article waterfall deployments.

    Paul: I mean I think that’s a really good way of thinking about it. It ties it to a name that everybody knows is bad.

    Edith: Well it was a deliberate—

    Paul: Yeah, good choice. So the agile deployments then, are the ones where it happens seamlessly, and you can go back and forth and change the requirements and whatever else.

    Edith: So that was Sean’s example No. 1, was what if it’s a really risky back and change? He had some good examples from his own career.

    Paul: Did he have any examples that I would agree with?

    Edith: Well a priori without hearing them, I don’t know if you’re going to agree or disagree. Another thing people brought up, was it just seems very risky to people. It seems to increase risk because I think they’re used to thinking of staging server as a safe harbor.

    Paul: I think there’s some semantics around the use of staging server. You very often want to have a complete copy of your environment that you can test against. So is that a staging server, or is that where you type docker up, you know new environment.

    And then you run your testing on this whole brand new infrastructure that has never been touched by anything before?

    Edith: Yeah, and the critique I made in the article, is I think people do that a lot. And they spend a lot of energy testing there, but these are artificial test pieces.

    Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit.

    You can find the full episode of this podcast here. For more on Continuous Delivery and to hear other Heavybit podcasts, copy this link into your favorite podcast player. We’ll have show-specific RSS and iTunes subscription options soon. You can also follow To Be Continuous on Twitter at @continuouscast.

  • Don’t Make Me Code Podcast: Onboarding Developers

    Don’t Make Me Code Podcast: Onboarding Developers

    Thanks for listening to Don’t Make Me Code, a podcast that discusses developer experience design and the unique challenges of building developer-facing products. Don’t Make Me Code is hosted by Steve Boak, co-founder and CPO at Opsee, and David Dollar, co-founder and CEO at Convox.

    In this episode of Don’t Make Me Code, Steve and David talk about some of the challenges of onboarding developers, and ways to make that experience better.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/245152401?secret_token=s-4SUlM” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Steve: So, today we’re talking about onboarding, and what a pain it is for developer tool companies. It’s weird to me that, in the consumer world, we’ve got products that we can happily onboard into in seconds, and in the dev tools world, we go in the range of minutes, you know, five or 10 minutes feels pretty good.

    David: Yeah, we’re dealing with like, complicated things that people are integrating into these other complicated things that they’ve built, and it’s often not quite as simple as ‘just type in your email address and click a button.’ You know, you have to add some pieces to your code, or add some dependencies, or start instrumenting things—there’s all this stuff you might have to do to get started.

    Steve: And those requirements feel just unreasonable at times, I mean I can’t stand going through it, having to write code to onboard into a product.

    I remember getting stopped using New Relic for the first time, and having to actually write code before I could start using the product, and it’s discouraging, but required. And it’s just this infinite frustration and we, in our product, we’re trying to eliminate that as much as we possibly can, but, it’s a unique frustration of our world.

    David: Definitely, I mean, we’re trying to deploy applications to the Internet and they can be, you know, the variation to those is so huge, and so we spend a lot of time focusing on making it work out of the box without you having to make a whole bunch of changes right out of the gate. I think that’s really important.

    Steve: See, you and I are both at early-stage companies, we’re dealing with our Beta customers now, and that’s a very high-touch scenario, we’re interacting with them very closely and one at a time. But, we often have to deal with things like personal email addresses, where they’re signing up for the product as a company, but using their personal email, so, I spend about 10 or 15 minutes just discovering who the person I’m dealing with even is.

    David: Yeah, I mean, we definitely see that a lot, and it’s actually not just personal email addresses, you know, they’re actually probably trying us out in the context of personal accounts to begin with, or personal, you know, maybe they’re coming to bring a personal project like their blog, or something like that, to begin with, and they want to like, ‘Yeah, is this thing going to work, before I actually start to try and introduce this at work?’

    Steve: Yeah, and you’re trying to demonstrate value in an environment that, one, is not the intended environment, and two, might have totally different requirements around it. And, you know, for us at Amazon for example, we’re dealing with people’s personal accounts that have things like EC2 Classic support, which has been deprecated, and their production environments at work are a lot less likely to have that, but it’s something that we, as the makers of the product, have to consider just because of how people might onboard into it.

    David: Yeah, definitely.

    Steve: And even if we’re lucky enough to get somebody’s work email address, we still have to go through the whole test flow, where they put us in their staging environment, and give us a try. And it’s a tough constraint to be in a place where you’re going to have measurably less value, but still have to prove that you can be valuable to the customer.

    David: Yeah, they want to see, is this tool going to work for me.

    No matter how good your documentation, no matter how good your examples, most people are going to want to actually put their hands on it, and see how it works, and see where it falls down. And any experimentation is sort of like the name of the game for our target customers.

    So I think providing some sort of backdraft for that, and making it really easy to play with your product is pretty important.

    Steve: Yeah, and really considering how you’re going to provide value in these circumstances where you might be restricted is super important. Although, I have to say, this morning, we had a guy sign up for the product, put us in an environment that had literally nothing in it. It was an AWS environment with 0 instances and 0 groups, and we’re a product that health checks other things in the environment, and there was nothing to health check. That was a tough sell.

    David: Now he knows that you’re not going to take down his Amazon account, I guess.

    Steve: Yeah, and that too, the security check, right, like, half of the battle is convincing people that, one, you’re not going to break, two, you’re not going to break their stuff.

    David: Yeah, I think, and that’s certainly another driver for why people try with their personal accounts first, right? It’s like, ‘I’m not going to install this tool into my production account at work that I don’t know anything about.’

    Steve: Yeah, it’s been really funny actually to watch this, ’cause we do a lot of onboards in person over video chat, or even just watching their laptop screens, and we’re dealing with incredibly smart people. They watch, and they can see what our onboarding process is doing to their environment, and they go and audit everything, they’re checking every change to preferences, and they know exactly what’s happening. And they check us on everything, which is, you know, worrying, but also kind of cool to see just how good they are at what they’re doing.

    David: We basically, we get the same thing, people trying to install us into their actual Amazon account, it’s a pretty tough sell, we’ve had a lot of luck getting early users to just either create a new Amazon account and install everything there. Or we actually, we will create a whole bunch of, we pre-create a whole bunch of Amazon accounts, and just hand them out to people to let them start trying stuff. Anything we can do to reduce the amount of stuff that they have to do to get in and get experimenting, I think is really valuable.

    Steve: Yeah, that’s a really good point. Something we’re not doing yet at my current company but we’ve done in the past is create these real demo environments, where, you know, without creds even, people can try the product out online and get a sense for what it will be like without signing up, which has been a really useful tool for me in the past.

    David: Yeah, it can be pretty tough to do that with developer tools, just because, you know, you are talking about complex integrations in a lot of cases, but yeah, I think hugely valuable, when you get that right. If I’m a couple clicks away from actually playing around with the product, and maybe even a couple more clicks from you know, taking what I’ve been playing with and going live with it, any friction you can reduce in that path is going to help.

    Steve: Yeah, that actually raises an interesting point about the transition over the last few years from how enterprise software companies used to sell to how they seem to be selling now, which is the transparency, that now, things like pricing, and a real sense of what the product does. It all sounds pretty basic, but, these were things that you wouldn’t expect to see, five, 10 years ago. That selling a B-to-B product—you would have a button to contact the salesperson, and you might not even know what the product did on the homepage.

    David: Yeah, that’s exactly right, I mean, you still see a lot of that today, and it’s incredibly frustrating as a buyer of software to run into one of those. We try and keep in that in mind a lot when, from the other end of that it’s knowing how much it’s going to cost to knowing exactly the value that it’s going to provide for you, and how it’s going to make your life better, is really important.

    Steve: Yeah, it seems so obvious just to design your marketing at least so that people know what the product does when they come to your website. I can’t imagine a consumer company dealing with some of these issues, like the problem of multiple environments, and that we get people signing up with personal accounts, or with staging environments that are not the intended use of the product at all, but in order to win these customers over, we have to show value in them.

    David: Yeah, it’s tough, I mean, those environments can basically be completely different too, right? Like, my personal IDS account maybe has nothing in common with what my company’s looks like, you know, it’s like almost as if somebody wants to test your iPhone app by installing it on their Android phone at home first. It can be wildly different.

    Steve: Yeah, we were actually talking the other day about having kind of a ‘clean up’ button, where because we know so many people are onboarding this way, putting us in a staging environment or something first, that, they’ll have to kind of decommission some things and remove some things that they created in that first environment before moving to the other one. And so we’re trying to make that process easier for them too, as part of our onboarding. Which just seems ludicrous, but, it’s funny to talk about.

    David: Yeah, it’s almost worth trying to, is there some way that you can, through permission sets or something, like actually introduce them slowly into the production environment in a trusted manner, as opposed to here making them go through this experimentation phase first.

    Maybe if it was very guided, it might be possible to pull that off. So, you mentioned dealing with customers sort of on a one-on-one basis, in kind of a very high-touch manner, and that’s something that we’ve had a lot of success with, and we actually think is really important, you know. We have a public slack channel, we invite people to come in, we talk with people all the time, they ask questions about the product, that ask, you know, if they’re having problems, or if they, you know, even questions about Amazon, sort of things that are not even directly related to us, and kind of get a conversation going about that stuff.

    It’s been hugely valuable to have that kind of real-time, you know, I can just ping somebody and kind of get to know our customers individually, and know what they’re building, kind of the problems that they’re having, and that’s been really helpful.

    Steve: Yeah and, not only that we get to interact with them in real time, but we have a way of reaching out to them, and that, one of the issues we have, I think, as an early-stage company is that people are trying the product, they’re not necessarily engaged with the product, and they may not tell us about that.

    And so, having every possible way to reach out to them, but also then making part of our onboarding process actively reaching out to them during this process, and making sure they’re getting value, and seeing if they’ve found bugs, and just doing everything that we can on our side to make sure that they’re happy and engaged.

    David: Yeah definitely, I think, I mean reaching out, kind of keeping the engagement level high, and reaching out constantly, like, you almost always have to solicit feedback. I mean, you’ll get some really awesome customers that will come and tell you everything that’s wrong with your product, and those are great, but, for most people, you have to actually solicit that feedback. A lot of the best learning we’ve done in our onboarding process, and finding where the friction is, is just talking to somebody and seeing the questions they ask, you know, sometimes they’ll be like, completely off base, or weren’t even in the ballpark of things you were expecting them to ask, you know, you’re expecting them to ask, ‘How does something work,’ and they’re asking, ‘What does this thing even do?’

    Steve: Yeah, I think you raised two really interesting points there: one is like, when I was new to this, both new to startups and new to designing technology products, I used to take offense from that feedback. ‘How dare you criticize my work.’ But, as you were saying,

    The most vocal customers are the best ones. The ones who are going to tell you about every single bug, I cherish that now. That’s what we look for in our Beta customers.

    David: Two reasons: one, it exposes all of the things that you hadn’t seen, and two, those customers are actually your biggest opportunity to make them successful. If they’re engaging with you enough to tell you what’s wrong, they want it to work badly enough that you can just help them a little bit and they’re going to be great.

    Steve: Yeah, totally, if nothing else, if they’re complaining it means they care enough to complain, and like we were talking about earlier, partly because our onboarding processes tend to ask a lot of our customers, and because that can take significant time, it’s really on us to engage with our customers and figure out how we can help through that and how we can make it easier for them, and so, because of the burden on them, it’s a big thing for us to both learn how we can make that better, and also help them help customers through it as best we can.

    David: One of the interesting things that we’ve been seeing is that we get not only really technical people coming through our onboarding process. We get some very technical developers that kind of understand the basics of what’s going on, and they have one set of questions.

    We get other people that are maybe a development manager, or, in a lot of cases, I had one ops guy and he’s now AWOL and I can’t get in touch with him, and I just need help, and they have a completely different set of things that they’re looking for when they come to talk to us. It’s been really enlightening to be able to talk to all of those groups, and kind of see what it is that they’re looking for, because it’s very different.

    Steve:  Yeah, we, in speaking to the different kinds of people using the product, there seems to be a transition happening in that way dev teams are set up. And so, we’re dealing sometimes with developers, sometimes with operations people, sometimes people who identify as DevOps, and it’s almost like we have to learn something about the organization first, how their team is set up, and how they deploy infrastructure to even get a basis for the level of conversation that we’re having. And the level of knowledge—yeah, like you were saying, it could be anyone from a manager, I even had a couple of product people contact me. And so yeah, just establishing how the team is set up has been really important to figuring out how to talk with our customers.

    David: Yeah, understanding who makes decisions about that kind of stuff, understanding how new technology is deployed at their company, you know, it’s different for everybody. Like, some people are very gung ho, they just take new stuff and go crazy with it all the time, and some companies are very conservative, and it’s a lot different to get sort of the new shiny things rolled out for them.

    Steve: I think the contrast of, and we were talking about this a bit earlier, broad vs. deep, and open source vs. SaaS, so I think, Convox’s case is really interesting because you’ve got both an open source product and a SaaS product. Can you talk a little bit about the contrast and the priorities of designing those two products?

    David: Sure, yeah, I mean, so, it starts right at the beginning. Like you mentioned, we have the open source, we have SaaS, our SaaS product is not open source, so, there’s just fundamentally different ways that we work on those projects even, you know, with the open source we work in the open. We do a lot of discussion with the community before we make changes, you know, pull requests, and trying to get a lot of people involved in the changes and make sure we’re taking all kinds of viewpoints into account that is going to be useful for a broad range of people.

    And then, with the SaaS layer we’re basically saying, ‘Okay, we know have this opinionated, this is how you should be using this, and this is we’re going to codify the happy path, the way that you, the easy, couple-of-clicks way into the product that uses all of this crazy technology that we’ve built.’

    Steve: Yeah, and the contrast there is really interesting because open source tools almost by definition have to appeal to a wide audience, you want as many people as possible adopting them and contributing to them. That’s I think a big part of how they develop traction is you get an engaged community of developers who want to help, and SaaS almost seems the opposite, that, a really common startup mantra is this: ‘Make a few customers love you as opposed to a lot of people like you,’ and so, making a great, polished experience for a much smaller number of people seems really important to developing a SaaS business.

    At Opsee we’re dealing with this too, we’re only SaaS, we have no open source component, and so we really are trying to focus on a narrow set of use cases, and then make them as polished and perfect as possible, and we are competing with open source tools. So, sometimes we will see these responses like, ‘Oh, why would I pay for your tool when I can get the open source for free?’ But then there’s this back and forth about the development time required to customize it, and the different nature of those kinds of tools.

    David: And operating it, you know, it may be free, but somebody has to run it, somebody has to write a page up for it, and it’s easy to lose sight of that stuff.

    If you’re interested in being a guest on the show, or if you have a DX topic you’d like us to dive into you can reach us at dmmc@heavybit.com or @dontmakemecode.

  • To Be Continuous Podcast: The Clash in Open Source

    To Be Continuous Podcast: The Clash in Open Source

    Welcome to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this episode, Edith and Paul talk about the clash in OSS between developers looking to get paid and software companies looking for things for free.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/244279037?secret_token=s-oeBCP” params=”auto_play=false&hide_related=true&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=’166′ iframe=”true” /]

    Edith: So Paul, what do you like about open source?

    Paul: So I guess that depends on whether we’re talking about writing open source or using open sources as part of a company. Writing open source, I like just having all my code out there.

    And I’ll do that as often as possible. Anytime I start a little bit of code, a little library, I’m looking to open-source it and have a readme and have it out there for other people to use. I had a very long discussion, which I eventually won, about the …

    Edith: I like the way you think of all discussions as win or loss.

    Paul: Well, yeah, yeah. It was more on the argumentative side of discussion. But with the other co-authors of the PHP compiler I wrote, which was originally GPL. And I argued very strongly that I actually don’t care if people give back, I just want people using my software. So to me, open source is a way of having people using my software.

    Edith: That’s really interesting. So, I liked what you said at the beginning about, like, you liked open source ’cause it meant getting stuff out sooner. ‘Cause I’m a big fan of that.

    Like, I always say you should just publish stuff. Like, my co-founder John said something really good at our team meeting last week. He’s like, if it’s 90 percent done and you don’t publish it, it’s actually 0 percent done.

    Paul: It’s 0 percent done, yeah.

    Edith: Yeah, and that’s a hard lesson, ’cause you always try for perfection lately.

    Paul: Right, right, right. Well, I think there’s also some really hard downsides about that. So when you publish something before it’s ready, before you really —

    Edith: Well, ready.

    Paul: Well, there’s ready ready, but let’s say ready is, let’s say you publish it before you clarify what the software is to do, or the values or the mission of the software.

    And then you get contributors who are saying, ‘Oh, this is great, but I really want it to do this other thing,’ and you’re like, ‘Well, that’s not exactly what I was going for here.’

    Edith: Well, I’d say most open source projects are so lucky as to get contributors at all.

    Paul: True, true, true. So I’ll give you an example. I wrote the V8 bindings for Closure. So a binding that allows you to write JavaScript and run it in Closure. And I did that for a purely selfish thing, for another, for a nascent pipeline that I was building called Stefan.

    And then people came along and they wanted to do other things with it. They’re like, ‘Okay, let’s add a binding for this other thing.’ I was like—

    Edith: Well, that’s kind of like a pivot.

    Paul: It is, it is, but I really was not interested in it.

    I’d rather someone took it over, took over the project and actually ran with it where they wanted to go, and I can stick on the old version, or I can see if the new version solves it for me, but I wasn’t interested in becoming a maintainer in this thing, I was interested in solving my itch.

    And I just happened to open-source it, because I felt it might be useful for other people.

    Edith: So Paul, I hate to say this, but I agree with you.

    Paul: No, not again.

    Edith: Not again.  No, I mean, I think what you said is exactly right, that you said, I did this for this reason, and I want somebody else to take it over now.

    I think where open source runs into issues where people are like, it’s only this purpose and you can’t do anything with it besides my original thing.

    Paul: So you think that’s a good thing, or you think that’s a problem? Or it causes problems in the community?

    Edith: I think open source is so misused for so many different purposes that it’s hard to even say. Like, so, in Europe, people say open source is good because it means free software.

    Paul: Libre or gratuit?

    Edith: I’m sorry?

    Paul: Free as in beer or free as in, what’s the other side of that?

    Edith: There’s no such thing as a free lunch, is that what you’re talking about?

    Paul: No no no, like people say open source software, it’s like free-as-in-beer software, as in software that has no cost vs. software that has no price tag vs. software that is, they use the French word libre for—

    Edith: Oh yeah, that’s what I was talking about. So I remember I went to Europe in, I think it was like 2004 or 2005, and they just had this big push for open source, ’cause they thought it meant free.

    Paul: Oh, this is like the German government and that sort of thing.

    Edith:  So literally like, so I went to this customer and they’re like, ‘So it doesn’t cost you any more to give us an extra copy.’ So why shouldn’t just everything be free?

    Paul: Oh, right, yeah, well.

    Edith: And so I think open source has been misused by many businesses who think of it as just like, hey, we get all these developers working for free.

    Paul: So I think that the main advantage to business of open source is not that it’s free, but it’s that it’s low-risk.

    Edith: Low-risk?

    Paul: Lower risk. So it removes—

    Edith: Ah, I would argue—

    Paul: Well, so, what I mean is that it removes a certain element of risk from it. So there’s, Joss Belsky wrote a blog post about how you shouldn’t use, I think it was a database that you didn’t have the source code to.

    Edith: Well, so I’ll argue the reverse, and I think open source is sometimes more risky. ‘Cause I talk to people now who are like, nobody’s maintained this open source project that I depend on.

    Paul: Right, right, right. But that vs. a closed-source version.

    Edith: Well, so the opposite risk is that you end up that you are responsible for this open source project, and you’re like, well, actually, I just wanted to use this thing. I didn’t want to do the maintaining.

    Paul: So absolutely, you don’t wanna end up the only person that, well, it’s actually fine to end up the only person which is using this thing.

    Edith: Well, no, it’s not—

    Paul: It’s not ideal.

    Edith: It’s that many people are using it, but nobody is paying money, and then you’re stuck.

    Paul: Because no one is maintaining it.

    Edith: No one is maintaining it, and you depend on it.

    Paul: Right, so, but imagine that no one maintained it and all that you could get was like a binary block. That’s the real problem. The company goes out of business. You have random database version seven, the binary, that only runs on 32-bit, that you’re maintaining, like, companies are literally maintaining COBOL—

    Fortran things from 30 years ago. It sits in a machine, you can’t touch it, you can’t fix it, you can’t apply security updates to it.

    Edith: Yeah, that’s fair. So Tim Chou, who is, he’s an awesome guy, he was early Oracle, he said that people don’t buy, when they buy software, they’re not buying it what it is today.

    Paul: Right.

    Edith: ‘Cause basically, anything that you’re buying today, you could just hire 20 engineers that could build it. What they’re buying—

    Paul: Yeah, you could just copy it.

    Edith: Yeah, what they’re buying is the promise of stability, that it will continue to be maintained. That there is a vision. And that’s actually, I went over to a big customer who won’t let us use their name yet, but they said they love LaunchDarkly because we thought every day about feature flags.

    Paul: Right, right, exactly. This is why people buy SAS, because, because people think all day about CI, about QA as a service, about queues, or whatever it is from various Heavybit companies that I just shilled for, they’re, you can probably figure out yourself which ones they are by looking at the homepage.

    Edith: Well, and Paul, by the way, thank you very much for wearing your LaunchDarkly t-shirt today. It looks very handsome.

    Paul: The, I definitely agree that the benefit of the thing is that you use it every day. But you can do that with open source as well. You can have open source software that, so, for example, our front end is open-sourced.

    Edith: Oh, I’m absolutely a fan of open source. Like, we use open source every day. Our client libraries are open source. I’m not saying that it’s, I don’t think it’s a religious thing vs. open source vs. not, I just think that many times people assume open source is something that it’s not.

    Paul: Yeah, fair enough, fair enough.

    Edith: I mean that if you have open source, that just means that developers spring out of the ground willing and happy to update your software every day.

    Paul: A mixed blessing, definitely. If it even happens.

    Edith: Like, I remember, so I remember when Sun, so I used to work on a, I used to be at Epicentric, and we had a portal server, and we were killing Sun. Absolutely killing Sun. And this was in 2003, so what they said is, ‘Oh, we’re gonna make our portal server open source.’

    Paul: Right.

    Edith: And so somehow this whole thing, we’re like, everything will be open source, and like, developers will spring out of the ground, and it’s like, well, no, developers, they don’t wake up every day like, ‘I’m gonna work on your thing.’

    Paul: Right, right. They will if it’s sexy or exciting. There’s lots of developers who are showing up at Mozilla just to contribute. And in fact, they’ve fought through incredibly large hurdles to contribute to Mozilla’s software.

    Edith: Well, that’s fascinating, ’cause I see so often the reverse, where people, so what do you think about Mozilla made people want to contribute?

    Paul: I think the fact that it was an open-source-first company, and that it was a nonprofit, and it was all about the free and the open web and that sort of thing. Also, people had had very good experiences with Firefox for, like, a decade at this point by the time I was there. Well, maybe seven years, but, you know.

    Edith: It’s fascinating, ’cause I went to dinner last night with the CEO of MySQL, and I don’t know if I should say this, but he said—

    Paul: This is Marten Mickos?

    Edith: Yep. So he gave a talk—

    Paul: He has a new startup?

    Edith: Called HackerOne?

    Paul: Oh, he joined HackerOne.

    Edith: Yeah, he’s the new CEO. So the dinner was kind of like a debutante ball for him.

    Paul: Nice, nice.

    Edith: So what he said was that somebody had asked him flat out how much of MySQL was actually from contributors. He said 1 percent.

    Paul: Seems about right.

    Edith: He said the open source was more just a branding thing. Like it wasn’t actually like a productivity thing. It was more just a branding thing.

    Paul: So you have this in Mozilla as well, that any time that there’s a really avid contributor, that Mozilla would hire them.

    Edith: Yeah, that’s what he said.

    Paul: Right, and if Mozilla didn’t hire them, very often they stopped being a contributor. Like, it was a real sort of a double-edged sword.

    So someone who did it for six months, they were doing it for the love of the game. They weren’t doing it to be hired at, at least I think for the most part, they weren’t doing it to be hired at Mozilla. And then six months in, someone’s like, well, you know, you’re doing a really great job, wouldn’t you love to work on this full-time? They’re like, yeah, that’s so exciting. They’re flown out for an interview. And then it’s like, you know, actually, I don’t think so.

    Edith: Wait, they didn’t wanna work for Mozilla, or Mozilla didn’t wanna hire them?

    Paul: Mozilla didn’t wanna hire them. I mean, this isn’t a particular case. But you know, some subset of people, Mozilla didn’t wanna hire. And then what happens? Well, you kind of don’t wanna keep contributing to the company that just told you to f*ck off.

    Edith: Yeah, I mean, it’s so true. I mean, I think there’s a very human thing about volunteering.

    Paul: Right, right.

    Edith: I volunteer, I was the moderator for a long time for the Lean Startup Circle, and somebody new came on, and they’re like, yeah, I’d love to raise some money so that we can get paid, and I’m like, they cannot afford to pay me.

    Like, I don’t do this to get paid, ’cause—

    Paul: Yeah, it’s like, it’s an altruism that people are trying to do.

    Edith: Yeah, I was contributing like 10 to 15 hours a week, and at my going rate, it’s not, it’s like, I’m not doing this to get paid, I’m doing it ’cause I like giving back.

    Paul: So there’s this thing that we were talking about earlier, about the current discussion around people getting paid in open source.

    Edith: Yeah, and I think a lot of what has fueled open source is the whole 20 percent time. Like, that Google will give their engineers 20 percent for open source, and I know that other big companies, like Twitter and so on, do that too. So these people are, in essence, getting paid.

    Paul: Right, right, there’s a, in most companies, in most startups at least, there’s an unspoken rule that you’re expected to contribute to the libraries that are in your application, and that sort of thing. If you’re spending time upstreaming paths that you’ve deployed, that’s work.

    Edith: Yeah, so I think we’re agreeing, which, I thought we agreed that we were not gonna agree. It’s funny, ’cause I’m Twitter friends with a guy I’ve never met, who’s, he started Jango, and right now what he tweets a lot about is—

    Paul: This is Jacob Kaplan-Moss?

    Edith: Yeah.

    Paul: Okay.

    Edith: That people are working their nights and weekends, and contributing to open source, and then just, people are kind of, for lack of a better word, jerks. People are like—

    Paul: People are jerks.

    Edith: Yeah, like—

    Paul: Even when you’re paid, people are jerks.

    Edith: But it’s more aggravating.

    Paul: Right, right, because you’re doing it for free, or whatever.

    Edith: Yeah, and so people will, like, they’ll be like, ‘I have this pull request and it’s been open for two hours. Why haven’t you reviewed it?’

    Paul: Right, right. I mean, we get the exact same thing. We get customers who email us, or more often it’s the people who get, who are using the free version, who send us a support request and then, 30 minutes later they’re on Twitter going, I can’t believe Circle hasn’t solved my problem yet. It’s been 30 minutes.

    Edith: What do you think drives that?

    Paul: There’s a pattern across the population in general, that people are jerks.

    Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of Circle CI, and Edith Harvaugh of LaunchDarkly.

    You can find the full episode of this podcast here. For more on Continuous Delivery and to hear other Heavybit podcasts, copy this link into your favorite podcast player. We’ll have show-specific RSS and iTunes subscription options soon. You can also follow To Be Continuous on Twitter at @continuouscast.

  • Don’t Make Me Code Podcast: A New Heavybit Series on Developer Experience

    Don’t Make Me Code Podcast: A New Heavybit Series on Developer Experience

    Welcome to Don’t Make Me Code, a new podcast that discusses developer experience design and the unique challenges you face when building developer facing products. Don’t Make Me Code is hosted by Steve Boak, co-founder and CPO at Opsee, and David Dollar, co-founder and CEO at Convox.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    In this first episode, Steve and David discuss the origins of the term “DX” (developer experience), they share their thoughts on what makes a great developer experience, and they talk about some of the challenges they face when designing developer-facing products.

    [soundcloud url=”https://api.soundcloud.com/tracks/242787724?secret_token=s-09pht” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    David: Hi, I’m David Dollar. I’m CEO and co-founder of Convox, a Heavybit company. I’ve been working in developer experience for about the last three or four years now.

    I’ve founded the DX team at Heroku, worked on a whole bunch of open source tools including Foreman, and now working on an open source platform as a service.

    Steve: I’m Steve Boak, the co-founder and head of product at Opsee. We are a monitoring company, and I’ve been working at several monitoring companies over the past six years, helping to help developers improve the experience of their products as well as find bugs in their code and improve the quality of their products.

    And though I didn’t hear the term DX for the first time until a few months ago, it’s been a passion of mine for several years, and it’s nice to finally attach a name to it. And I’m excited about this first episode because we get to unpack the term a little bit and figure out what it means.

    I’ve thought one of the most interesting parts of your background, David, was that you were part of this founding team at Heroku, that sort of created DX, and I’m interested to hear more about that.

    David: Yeah, so I guess this was maybe late 2012, maybe early 2013, right as at Heroku we started to expand, out of being just a Ruby host into being a host for really any language. That meant we also started to see more, you know our standard user before that had been sort of like, and I was ten user building rails and was very, there was sort of one happy path for that.

    As we expanded languages we had Windows users and Linux users and sort of, everybody coming together to try and build really any thing. And that started to get out of hand really fast. So, really, we formed the DX team, brought together basically the people that were in charge of the various languages at Heroku. Somebody in charge of the Ruby, and somebody in charge of Python and Node. Brought us all together into one team and really decided to try and make a consistent experience, sort of the Heroku way, across all of these languages and …

    Steve: And was it called the Developer Experience team from the beginning?

    David: It was, yeah.

    So the first time I heard the term was from Adam Wiggins. He sort of like came to me and said, “Hey, I think we should build this DX team. What do you think?” And yeah, so I mean it sort of exactly described what it is we were trying to do, which was make this nice experience for people building things on Heroku.

    Steve: And when was that? What year was that?

    David: I think it was the end of 2012, but it’s kinda fuzzy at this point. But yeah, it was a while ago.

    Steve: And it does seem to continue to be an important thing for Heroku. I, just in researching the term of it, saw that there’s a Heroku.com/dx, this is a big part of how they market themselves.

    And clearly it’s been important to Heroku for a long time. Like I said, it’s interesting to now talk about this with a name because a lot of us are doing this and have been doing it for a while but, giving it a name gives it so much more weight and it kind of marks how important it is for all of us. Mmm-hmm.

    David: It’s interest … you think of DX and what does that actually even mean? And its, there’s a certain type of user that you’re talking about now, right? When you’re talking about designing an experience for this user that likes to kind of get under the hood and break things apart and do things their own way and sort of knows what’s going on and maybe you could even consider a power user.

    Yeah, you want to make sure that they can sort of do whatever it is they’re trying to do but that the options aren’t overwhelming and that it’s not complete, just utter chaos.

    Steve: Yeah, it’s almost like this partnership that we as the makers of tools that developers then use to build tools for their customers, if they don’t like the product that they’re using that may show in the product that they’re ultimately trying to build. And so it’s this partnership where we all try to write good code and create great products together.

    And I think that’s part of what makes this different from user experience design as a whole, and why it’s interesting as a topic all unto itself.

    David: I definitely agree with that.

    The most rewarding thing about working, I guess in developer experience that I found is when people do things that you never expected them to do.

    You put all the pieces in place, you thought like, “Oh, maybe somebody will use these pieces to build X,” and it turns out that somebody comes and builds something you never even thought of. I think that that’s just really rewarding to see that.

    Steve: And it’s interesting that, I think a lot of product builders in general would probably share that sentiment in terms of the unexpected ways that people find to use their products, and the hacks that they create, and I’ve heard, even with consumer products you know, people being surprised by how the users are actually using the product in ways they never expected, but with developers that really is to another level because they may try to break your APIs.

    They’re much better at exploring the depths of your code and then creating tools that you never expected with it, which is both surprising and wonderful for us.

    David: It’s still largely about aesthetics, right? Like, you’re a good user experience, a good developer experience.

    At the end of the day is this thing easy to use? Is it pleasant to use? Are you able to figure out what’s going on? Does it make me happy to use the thing at the end of the day? So yeah, it takes a different set of things when somebody’s trying to come in and actually build something on top of your product.

    Steve: I remember in the early days of the Google Maps API just how many experiments and apps were being written with that.

    I think, partly, because of how easy and accessible they made it for developers. And I heard a talk from someone on that team talking about that the product that we all knew and loved then was not the first iteration of it and they actually had a much harder to use initial version. Do you know more about the history of that?

    David: I don’t really know.

    Steve: Yeah, but that was one of my first exposures to using another company’s API and it was a really rewarding experience. And they had all this boilerplate code that you could very easily lay down to plot pins on a map and see that.

    Especially for me as a designer it was so rewarding to get that gratifying output. The actual map on the screen with the pins that I had laid out. I think that is a really great way to measure how effective and how elegant your developer experience design is, in how quickly a developer can make something themselves.

    David: One of the barometers I use for the things that I’m building is you look for the blog-posts people are writing around your tools.

    Are they writing a blog post of like this thing was so insane I couldn’t figure out how to use it? Like here’s my giant blog post teaching you actually how to use it because I had to figure all this out. Or are they talking about how easy it made their life? Like, look how awesome this is, you should try it, too.

    Steve: Yeah, and that makes me think of the JavaScript world today and just how many frameworks are being launched, and the competing arguments for this or that. And in some ways this has really gotten good for the development community as a whole because there are so many choices now, and so many good frameworks to choose from. Almost too many, actually.

    And I think we’re starting to see some of that in the JavaScript world. Just competing ideas of how things should be done. And that almost raises a counter-argument to the value of developer experience design that when we open up too many possibilities and don’t create a set of tools, I think you described it in your Heavybit talks as like, sharply …

    David: I like to compose sharp tools. I mean it’s sort of like the Unix philosophy, right? Like, you want to make tiny tools that do one thing really well. You generally want to have some good examples or defaults, like, this is how to use these tools in combinations to make interesting things. But having those tools be sort of separate and composable is where you start to get really interesting things happening that you didn’t expect.

    Steve:

    Yeah, that each piece does its thing really well but also that when combined they can form something greater and that the whole is greater than the sum of the parts.

    David: Right. Like, a good example of that, it’s not really developer experience, but I have TTT, it’s sort of like an infinite variation of all these little components that you can put together, but they also have some default pipelines that come out of the box, you say, like, “Okay I really just want to make my email go to Google drive or something like, just like these nice defaults.”

    It’s really important, I think, when you have all of these choices to really also have the guides through them.

    Steve: Yeah. We use the Amazon APIs a lot now, and they’re an ecosystem all unto themselves. For every product Amazon offers inside of AWS there’s a unique API and many of those APIs seem to have differing views of the world and so, they in some sense they’re prolific in the amount of tooling they create for developers and they clearly have won a huge share of the market with just how much they offer, but it still feels to me like they’re a counter example.

    They seem to have APIs that a lot of us use but not APIs and tools that a lot of developers enjoy using.

    David: Even despite that, like pretty much everybody is still using their Sphinx, right?

    Steve: Yeah.

    David: So there’s something to that.

    Steve: You know, Facebook’s API or Twitter’s API I can’t speak to how good or bad those are but man, a lot of people use them. And there’s something to that, too, that the product as a whole has so much value and impact on the world that you are sort of forced into a place where you need to use it.

    And it doesn’t absolve you from the responsibility to make your tools great for your developers and your partners in this, but yeah, it’s sort of mind blowing to see how much can be done when you are hitting on a big share of the market.

    David: The AWS database is another great example of that. There’s so many options, so many ways to do things that you can, like, it’s great. You can really do anything. Like the sky’s the limit. But there is, like, finding the happy path can often times be, it’s just a lot of trial and error at the beginning.

    You end up just having to build up some expertise around that stuff and eventually you can, okay now I can understand how to design things using these tiny permanents, but yeah there’s not a really great, like I just have some code and I want to put it on the Internet.

    Steve: Yeah, and part of Convox’s business is really helping developers do this, right? That there are in fact so many possibilities to create an infrastructure in AWS, that giving a little guidance is probably a good way of helping developers through that process.

    David: Yeah it’s basically just trying to create a happy path. I think there’s a lot of choices that you don’t necessarily have to make.

    Steve: Yeah, we too, at Opsee, use AWS heavily. It’s a monitoring product designed just for Amazon web hosting. And when we were creating this product we were assuming we would see a pretty high level of consistency in how our customers are configuring their environments, but that was shot down almost immediately.

    There have been no two environments that are the same and so now we’re dealing with a wide variety of customer environments and it’s an example of just how much people can do with these tools, but how differently they’re choosing to do them.

    That though does get to this other interesting question of how we measure the quality of developer experience and how we know that we’re doing well. And we talked already a bit about time to value and how quickly developers can make tools with our products, but then I think it’s also interesting to talk about how that becomes part of how we price these tools. Like, we try to look at how much development time we can save our customers and then try to price the product based on that. Do you try to do something similar?

    David: We’re trying to think about this as a peer developer tool, so rather than, you know a lot of people try to charge for resources, things like that, we’re doing more of like charging per developer that’s using the thing. So we actually, actually creating value for developers I think is sort of the core of what we’re trying to do, so we’re approaching it from that angle.

    Steve: Yeah, and monitoring being a little bit different, we’re mostly trying to measure how much code our developers don’t have to write when they’re using this tool, and what they would have to do in a competing world, but

    I think the one thing that all Heavybit companies, certainly in all developer tools companies share is that at it’s core we’re trying to measure our value that way: How much code can we save developers from writing, and how much time can we save them?

    And that feels in many ways different from normal user experience design and consumer user experience design. That we have to look at that, the value that we’re creating for our users in order to assess our own value. So as designers of developer experience products I think it’s interesting to talk about the inspirations and the other companies that are doing this well.

    The first time I had ever really thought about developer experience as something to be measured was looking at Stripe and how … You said earlier how blog posts and recognition within the community can mean a lot. And I remember hearing lots of developers talk about the quality of Stripe’s documentation, to the point that it became synonymous with a great developer product.

    Like do you have Stripe-level docs? And so they were always a barometer for, you know, are we doing well and who do we measure ourselves against?

    And I feel like they’ve done a great job not just in creating great documentation and great APIs but in great products. I mean even their web design. They’ve taken all aspects of their design to a level that I think was not common in our world when they were doing it.

    David: Yeah definitely. I mean, we use Stripe and I just remember when I was first clicking through their web I was just like, “Wow, this is really nice.” You just have that thought to yourself while you’re using it, so yeah it’s very pleasant to use.

    Steve: And I don’t know, you know in the early days of Heroku I don’t know if you saw much of this but certainly five, six years ago if we designed a UI that was polished and really nice looking there was almost a bit of hesitation among our developer customers that we didn’t have the street cred like, “No, you can’t be good, you can’t be real because no one in our world would do this, would make a product like that.”

    And I think that’s changed a lot over the last few years, but we were almost fighting this uphill battle to try to create an elegant and beautiful user experience because people didn’t expect that.

    David: I certainly think that sentiment has changed. I think having a good design aesthetic, a sense of that, is actually like seen as a good thing in most developer circles now, as opposed to being like, maybe you don’t know what you’re talking about if you have a sense of style or something.

    Yeah, it seems to be important. People really look for it when they’re looking for tools or products and that’s also something that we tend to see celebrated.

    Steve: That’s about all we have time for today.

    If you’re interested in being a guest on the show, or if you have a DX topic you’d like us to dive into you can reach us at dmmc@heavybit.com or on Twitter @dontmakemecode. To learn more about Heavybit visit heavybit.com. And while you’re there, check out the library. It’s packed with amazing talks from founders of developer tools companies and other industry leaders.

  • To Be Continuous Podcast: How Do You Deal with Your Data?

    To Be Continuous Podcast: How Do You Deal with Your Data?

    Welcome to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. Edith Harbaugh and Paul Biggar talk with Peter van Hardenberg about about continuous delivery in the context of organizational structure, databases and pricing.

    Peter is a co-founder of the Heroku Postgres product team. For more than three years he has been responsible for designing, developing and operating the largest cluster of Postgres databases in the world. He knows how to get databases running, keep them safe and ticking fast.

    This podcast is brought to you by Heavybit, a program that helps developer startups take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/241491956?secret_token=s-oZ2T7″ params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=’166′ iframe=”true” /]

    Paul: So Peter, it’s traditional to ask you, what do you like best about continuous delivery?

    Peter: Well, I’m increasingly old and I’ve done a lot of jobs and so I remember delivering Gameboy cartridges to the market and a Gameboy cartridge is a physical device that goes into a Gameboy, hence the name, and that process involves physical manufacturing and you get one shot and so there’s a tremendous amount of basically wasted and un-fun energy that goes into making sure you never make any mistakes.

    My favorite thing about continuous delivery is the ability to make mistakes and move on with them.

    Paul: Nice.

    Edith: This would be a great time for you to introduce yourself.

    Peter: Yeah, hi, my name is Peter van Hardenberg. I’m a Heroku old-timer, I’ve been with Heroku about six years now and I co-founded the Heroku Postgres team and have built a lot of products since then.

    Edith: So it’s funny you’re talking about the Gameboy because Paul and I were just talking about that.

    Peter: About Gameboys.

    Edith: Because I said one of the things that had fueled the rise of Zynga was continuous delivery.

    Paul: That’s the first thing we found about continuous delivery that was actually bad.

    Peter: So I guess they’re probably still continuous delivering somewhere. In the great startup in the sky. They’re still around, aren’t they?

    Paul: They’re still around, they’re still public and,

    Peter: Farmville will never go away, I’m sure.

    Paul: I think they’ve moved into other lesser things.

    Peter: Developer tools.

    Paul: Imagine.

    Peter: Gamified developers. Well actually, it’s how all developer tools are becoming now.

    Edith: Github is gamified. How many chickens do you have, can you make a nice pattern out of them.

    Paul: New watches.

    Peter: It’s a bit like a game except if you’re losing the game, you lose your job.

    Paul: Or you just can’t get new jobs.

    Peter: Well, that’s actually probably more accurate, yeah. Yeah it’d be a weird company if they let you go because you didn’t have a long enough commit streak.

    Paul: On open source projects that weren’t your job.

    Peter: I like that, actually, let’s make that a game.

    Paul: We were talking in the last episode, as well, about open source funding models, or the anger over lack of open source funding models.

    Peter: That’s a tough problem to solve.

    Paul: That would be an interesting one. You get fired if you don’t have enough contributions.

    Peter: Yeah, just put it on your quarterly review, current Github streak.

    Paul: Let’s put this suggestion on Hacker News and see if people agree with it. I can pretty much picture how this is going to go.

    Peter: I try not to read the comments.

    Edith: So what do you think is liberating about the ability to make mistakes?

    Peter: Well, I mean, more important than the ability to make mistakes is the ability to try things, right, I mean we, I’m so accustomed now to being able to see in great detail because of a SaaS model, sort of the reality of what my product is in the market and then vice-versa, I can turn that around and say, well, basically live and in real time, what’s happening if we put something out there, do people care?

    Does it help with adoption? Is it suddenly throwing a bunch of errors? And if there’s a problem, you can roll back very quickly. You know we run a database product and so availability is really, really important but only for a very small core of the service. People are generally pretty understanding as long as your, sort of, problems and outages are brief.

    Small and, sort of, blast radius and quickly remediated and I think that they prefer to get features vs. stability, sort of, at the periphery. At the core, they need stability over features, so, you know if you’re a stripe, right, you need to keep the payment processing online but if some ancillary piece of your API has a problem that’s probably not as big of a deal.

    Paul: The dash part is less important—

    Peter: Right.

    Paul: than the payment going through.

    Peter: Well, and responsiveness and progress are so important, and I think because we’re in such a changing market, the ability to drive forward is at least as important as the ability to keep things stable but you can’t do both in the same place, basically, right?

    I think continuous delivery is cool because it’s separated, sort of, a slow-moving core from a fast-moving periphery.

    Paul: So when you’re creating a software product, you really want every single component of it to be continuously deliverable, which means you want to be able to migrate from the thing that you are doing now to the new thing.

    And so it’s interesting, or there’s an interesting thing about databases because databases, I think, were fundamentally not built for this model.

    Peter: They’re heavy.

    Paul: Right. We had a problem recently where we deleted an index.

    Peter: Ah, yeah.

    Paul: And, whoops, that index was in use. And we thought it wasn’t in use but it was in use.

    Peter: You should’ve been using Heroku Postgres, it would’ve told you.

    Paul: Oh so, that’s interesting. So the thing that I wanted was to be able to, you know, partially delete that or let’s run all the queries without that and then after an hour, I can actually delete it. What does Heroku Postgres offer you?

    Peter: Well so we have a number of diagnostic tools where you can see exactly how often an index is being used. In addition, next time you’re trying to delete an index, wrap that in a transaction, so say, begin and then drop the index, then do whatever and then say, rollback, and that way, you can actually test as though you didn’t have the index because Postgres MVCC model means that all that stuff is totally transactional.

    Paul: So you’re, when you say do all the stuff in between, what do you mean?

    Peter: Whatever queries you want it to run. Also, and we’re getting into minutia and then some baseball here but you can actually disable indexes specifically, sort of at a session level as well.

    Paul: Oh, interesting. This is all Postgres or Heroku?

    Peter: Yeah, that part’s just Postgres I mean, we built the tooling around it so that you can do these things, kind of, the easy way but, I’ll tell ya, if you read that 3,000-page Postgres manual, you’ll find some real gems.

    Paul: Well so part of the problem might be using a thing that’s not a real database?

    Peter: Oh, you’re not using a real database?

    Paul: We’re not—

    Peter: Oh, that’s right, I forgot.

    Paul: I mean we are using a real database, we’re using many databases but that particular one was not in a real database.

    Peter: Well you know, even baby databases grow up to be real databases someday.

    Paul: Yeah, well, fingers crossed that that particular database that starts with an M and that we shall not discuss.

    Peter: Oh, we don’t like to speak ill of anything.

    Paul: No, no not at all, especially Mongo.

    Edith: No, you said the word!

    Peter: You said the M word! So coming back to, sort of continuous delivery of every part of the stack, yeah, it’s interesting, I mean we’ve had some interesting challenges with databases in particular and I actually gave a talk at Heavybit a while ago about, kind of, how we treat databases less as part of a service in sort of the classical sense but more like a factory.

    So basically what happens is when people request databases from us, we stamp one out and then, you know, we try and keep it pretty low entropy from that point on so we have a lot of tooling and support services around the database but the actual database itself, once it’s given to a customer, we mostly try and keep our hands off it.

    Now if there’s security issues or performance issues or whatever, we’ll go in and we’ll kind of warm it up and do some work on it. But even still we mostly prefer to actually replace the database. What we do is we create a replica, bring it up to speed and then we transition the load over to it. And that’s kind of a form of continuous delivery for data services, which has been surprisingly powerful as an abstraction.

    Edith: So you’re basically, you’re treating them as cattle, not pets.

    Peter:

    Oh absolutely, it’s all about the cattle and model and when you’re farming a very large crew of livestock, you can’t really afford to give each one their own personality.

    Paul: This reminds me a lot of, I think, what started out as an IMVU way of doing data migrations, I’m sorry, of doing schema migrations without locking a whole table. For our people at home, I’ll describe this.

    So when you want to add, let’s say you want to add a new value and it has a default to a particular table in your schema. There’s a possibility of that if it’s a particularly large table, for it to be locked while that happens. At least this used to be the way, I think modern databases are much better at this now.

    Peter: I believe for some default values, it’s now the case that you don’t need to do a table lock but that might be pushed until 9.6 or Postgres, at least but it’s a common and challenging operation.

    Paul: And so what IMVU started doing was instead of adding to a schema, they said that for every table, that schema is immutable and instead they created a new table and then they migrated the data from that table so every time you want to read from the user, you’re reading from user table 38 at the moment and then you start to read from user table 39. If the user isn’t in user table 39,

    Peter: You fall back to 38.

    Paul: fall back to 38, find it, migrate it, write into 39 and then do the thing. Then you have a background process.

    Peter: If you have any listeners who are listening from the world of the Internet and would like to try implementing this technique for themselves, Postgres has something called a schema search path so you can actually, basically make each schema its own version and then you have a search path so if the table wasn’t in the newest version, it would use the last version’s search path.

    Paul: Oh, wow.

    Peter: You could set a version on your code then basically, as you make new versions of the tables and the new schema, you can kind of move them forward. This actually sounds like it would probably work. Now it’s really easy to say that from sitting around a table here in beautiful San Francisco but you have any listeners out there who do give this a try, have them send me a note.

    Paul: I was not expecting this to be a session about continuous delivery databases and the intricacies of modern versions of Postgres.

    Peter: I have to say, it’s one of the hardest things is basically, how do you deal with your data? I mean, data is sort of fundamentally heavy, I mean I think about it as having mass, which is to say, if you’ve got a few terabytes of data on a disk somewhere, just getting it out of that server and into another server is gonna take hours.

    Paul: Right.

    Edith: Right, just querying the data, loading it all into memory, that’s gonna take time and so this is really a challenge for continuous delivery and people have tried all kinds of schema versioning and replication strategies and sharding strategies but, you know, the old question, okay you deploy a new version of your code and it needs the new column but the new column’s not there yet. Okay so you move the data to the new column and then you deploy your code but then the old code doesn’t have it.

    Paul: Right, right.

    Peter:

    And these are hard problems and no one’s really done a great job of answering them. So far, sort of the state of the art is, you know, you continuously deploy the new version of your code that supports both schemas then you continuously migrate your database and then you deploy yet another version.

    Now without continuous deploy this is even harder—

    Paul: This has to be tough on a Gameboy cartridge.

    Peter: Oh yeah, let me tell you, I think we had 4k of RAM or something like that so at least it was fast. But yeah I think data and continuous delivery are both challenging interactions for sure.

    Paul: We end up at this problem in the testing world when large customers, in particular, large enterprises use the old way of testing, which is you actually use a dump of your production database as part of your test suite and one of the major challenges we get when we’re looking at older companies who are coming to the new world, is, you know, your server’s over here, my server’s over there and I want to get a 5 Gig database dump onto your server to run my tests, how do I do that?

    Peter: Right and it’s trying to, like, move a bathtub through a straw. So how do you do that? You just wait, right?

    Paul: We wait until they stop doing that and then they can become our customers.

    Edith: So do you turn customers away then?

    Paul: We don’t turn them away but we don’t really support that use case, they’re going to have a miserable experience.

    Peter: I mean there’s data migration again, right, just coming back to the physics problems. There are things that are getting better so,

    Paul: The speed of light is improving?

    Peter: the speed of light has gone up 13 percent in the last year, did you know that?

    Paul: I don’t know if you’re joking, I presume you’re not.

    Peter: It absolutely has not, yeah. They call it a constant for a reason. I suppose it could.

    Paul: I thought there might be something about, like,

    Peter: Dark matter, no but it was very convincing the way I said it, wasn’t it?

    Edith: Like this is Hindenburg’s Principle?

    Peter: It’s uncertain. So anyway, the logical replication scene is getting better. Amazon’s just launched a new service, it’s three letters long.

    Paul: Oh really?

    Peter: Yeah, uh, got it, hang on. DMS, Data Migration Service. And so through black magic and probably by not working very well, it will move all your data from one database to another in sort of a streaming online way which is really cool if it works.

    Paul: Okay.

    Peter: But, you know, moving from Oracle to MySQL

    Paul: So you can talk to this service as if it’s MySQL and it will pull it from—

    Peter: The way it works is you basically set up a migration job and if you think about it, it basically queries one database and writes to another and it’s a little bit of a hack job in the sense that you have to run, like, a desktop client if you’re moving from a local machine. It’s all very complicated and kind of baroque but,

    Paul: So it’s an Amazon service? It isn’t you’ve used Amazon before, I gather.

    Peter: But yeah, no, it’s what everybody’s always wanted which is to move their data from one place to another without having to think about it and, of course, it’s never gonna work.

    Paul: Okay.

    Peter: But, you know, it’ll help, it can help, things could be less terrible. I think ultimately that’s what got me into databases was the understanding that no matter how impossible it would be to make things good, at least we could make them a little bit less bad.

    Paul: Right.

    Paul: The soul-crushing misery of writing software could at least be slightly less crushy.

    Edith: I feel like writing software is one of the greatest joys in life because you get to create something from nothing.

    Peter: So you don’t work in databases.

    Paul: I’m not sure you work in software.

    Edith: So what are the biggest changes you’ve seen in continuous delivery?

    Peter: Oh, ubiquity, certainly. It’s rare to meet someone these days that, maybe I’m just lucky enough, I live in San Francisco.

    Edith: You hang out at Heavybit.

    Peter: And I hang out at Heavybit with a bunch of other crazy people but I guess I would say growth then, right, I mean we’re seeing large enterprises practicing continuous delivery—we’re seeing small shops practicing continuous delivery, it’s not, sort of, a bleeding-edge concept that only, sort of, a small cadre of, like, sort of, avant-garde developers are doing. It’s really, I think it’s kind of the mainstream.

    Listen to full podcast here to hear Peter talk about continuous delivery in the context of organizational structure, databases and pricing.

     

  • To Be Continuous: Empathy in Software Development

    To Be Continuous: Empathy in Software Development

    Thanks for listening to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk with Sam Stokes about empathy in software development, how mini-services are the new microservices, and constantly updating software in a constantly updating world. Sam is currently a freelance software consultant and previously was the co-founder of Rapportive.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/237727552?secret_token=s-k6OKe” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=flase” width=”100%” height=’166′ iframe=”true” /]

    Paul: Alright, Sam, so what’s your favorite thing about continuous delivery?

    Sam: I like to think about it terms of shortening feedback loops. A lot of things that startups do can basically be looked at as doing something and then waiting to get some feedback on it, and then deciding whether to do more of that thing or to do a different thing.

    And continuous delivery is, people talk about sort of reducing the batch size and also about shipping stuff faster and clearly having some kind of automation. And continuous delivery is going to help you with both of those things.

    Paul: So is this specifically on the, sort of on the journey of a small startup towards product market fit? Is that kind of the sense you’re thinking of it as?

    Sam: I see it as applying to teams of different sizes. So for sure in a small startup you’ve got you need to iterate fast, you’ve got all these product ideas. But I think even if you’re a small team in a big company as I was recently working on, and still have a lot of the same pressures you have, product managers who want to try new ideas, you have customers who are sending in feedback all the time—some of which you want to listen to, some of which you want to tune out.

    It’s sort of, in some ways, a lot like being in a startup. At least, ideally it would be. And a lot of the problems you can run into are basically about not being able to close those loops quickly enough.

    Edith: Sam you’re someone who knows a lot about loops, so now might be a good time to introduce yourself.

    Sam: Sure, my name is Sam Stokes. Until recently I was working as a staff software engineer at LinkedIn. And right now I am, I guess a freelance software consultant, thereabout.

    Paul: I feel there’s a little bit of history missing there.

    Sam: Alright. Going back in time a little bit. So I was a co-founder of a startup called Rapportive. That was a plugin for Gmail that when somebody emailed you, would show you a social profile of that person in the right-hand side next to their email.

    So that you could get some insight into who you were talking to and put some more humanity into your communications with people—that’s sort of the way we saw it.

    Paul: That’s still the only plugin to Gmail that I use.

    Edith: I use it, too, and it’s for exactly that reason, ’cause I do a lot of customer development, and it helps so much to do, to empathize. To see that this—

    Paul: Right, right, yeah is a real person.

    Edith: You know, to see their face and you feel a connection.

    Paul: It’s interesting just with kind of customer communication. One of the tools that we use is Intercom. And once I started using Intercom it was the same sort of feeling I got the first time I used Rapportive, that like I feel that there’s a human at the other end of this and I feel I know who that human is and it just changes how you communicate with the people.

    Sam: It’s really motivating to hear and to see other examples as well. We really felt like email and sort of a lot of business communication in general can be sort of cold and formulaic.

    Yeah, just bringing that human element, even just seeing a face. I haven’t used Intercom but I’m guessing they have some similar ideas.

    Paul: Similar, when you send out messages or reply to someone it does the person’s face there.

    Sam: Right, we had a sort of plugin API in the, it was a sort of private beta version of our product and one of the things people used it for, so this would let you plug into the sidebar, people would plug it into their customer database and so they could see something like, they’d get an email from a customer it shows them how long they’ve been a customer, how much they’ve spent in their store—

    Paul: Right, right.

    Sam: What version of the app they’re running, that sort of thing.

    Edith: Was your original idea to do that sort of empathy or was that something that just you iterated into?

    Sam: The empathy was where we started. There were lots of ideas about taking all the information from all these different sources, you know public sources like, truly public sources like Google, quasi public sources like a Facebook profile if you’re friends with them on Facebook, and then private sources like stuff in your company database.

    All stuff that you would be able to search for if you could be bothered to go and do the searching but doesn’t necessarily bring itself to you while you’re talking to that person. And yeah, it was very much, what should you know when you’re talking to this person in order to make them feel like they’re interacting with a real person and an empathetic one.

    Paul: So I remember when you launched Rapportive it was like, or it felt to me like it was, an instant hit and maybe that was just the overnight success that comes after years of trying. I had a doco and what was the kind of software delivery process of that?

    Sam: Hmm, that’s a great question. I guess there’s two parts to that really. So how did it get there and what was our process that allowed us to survive it? So I guess how did we get there, it was a certain amount of luck in that process as there probably always is. We built our prototype and it was a pretty minimal prototype.

    You know, it had the functionality that was like this awful demo website up. The only reason there was a website up was we were trying to raise a seed round at the time, we were applying to OIC. Y Combinator requires that you have a demo site that’s public. They don’t like it to be password-protected. And so we had this thing up where you could install it and we had like 20 users. One of our friends decided that he was going to email a friend of his at one of the blogs, I think it was ReadWriteWeb and say, you know, hey you should check this out.

    And his friend checked it out and it turned out that one of the demographics that really liked the product that we built was journalists because they get a lot of emails from people they don’t know and they want to sort of know an extra nugget. I guess that was one of our takeaways was like, if your startup is really useful for journalists, that’s a great way to get some early press.

    Edith: That’s what worked for TripIt because journalists travel a lot. So TripIt just got all this free press basically ’cause journalists used it, loved it, and blogged about it. Unfortunately, journalists do not do continuous delivery so much.

    Sam: So how do we survive that? ‘Cause what happened in that 24-hour segment was that we went from 20 users up to 20,000.

    Paul: Wow.

    Sam: So I guess one part of it was our CEO had made the original platform choice. His name’s Rahul Vohra, he has a computer science degree, he’s an awesome technologist, but his focus is not generally on writing the software. He’d been writing the prototype, and he was like, ‘I need the quickest thing that I can get to get this market, so I’ll put it on Heroku.’

    Which turned out to be a life-saving decision because when the 1,000 X traffic spike arrived we just added some more dynos. So that was how we didn’t burn down. What did we do after that? I guess in terms of software process we were pretty young as a company, so it was pretty much hack away, read all the tweets. But there was a lot of, what you might call customer development more than software development at the time. So we pretty quickly decided that we were going to try and respond to every single tweet and every single email and so on to like try and live this idea of a good customer experience.

    And I think that helped to sort of amplify the viral effect of that press spike because people were sort of retweeting the things we were saying and getting a generally good feeling about us.

    And so a lot of those tweets were like, ‘Oh I wish I did this,’ or ‘I wish I did that,’ or ‘I wish you could see Twitter profiles in there.’ So it was a little bit of sort of customer support-driven development. Which obviously doesn’t scale but it can be a pretty good way to build some goodwill early on.

    And I think a lot of our users were sort of quite passionate about using us, ’cause they felt like they got that personalized attention.

    Paul: So did the, when you scaled the dynos did it just work?

    Sam: It worked perfectly until we hit the rate limit for the API that we were getting all our data from.

    Paul: The LinkedIn API?

    Sam: At the time we were using a service called RapLeaf.

    Paul: Oh yeah, yeah, I remember them.

    Sam: It wasn’t their main product, but they provided an API which … So basically the thing that Rapportive does, which is take an email address and produce social profiles about that email address. At the time that was entirely just UI wrapped around RapLeaf. We ended up building sort of our own version of that.

    And so we’d been using their API on a sort of demo basis. Because again, 20 users. Yeah, we immediately had to learn about things like request queues, so that our app wasn’t just falling over waiting for their API, and then frantically Biz Dev-ing to try and increase our rate limit, while at the same time figuring out the best ways to get around the rate limit without upsetting the Biz Dev deal. Yeah, we didn’t really have much of a sort of continuous delivery process at the time.

    But I guess not as many people were talking about it—this was 2010. I think the first blog post had come out from Iron View and so on, but I think it wasn’t as, you know, there weren’t full startups supporting it. What we did have was the Heroku deployment model, which was basically like within five seconds of writing the code you can deploy it to production.

    Which if you squint at it, kind of is continuous delivery because you’re saying, all of the bits of your deployment that you have are automated. We didn’t really have many tests at the time, so the tests we had were automated ’cause there weren’t any. Our deployment was absolutely, you do it get push, you don’t have to like move any servers around or anything like that.

    Paul: Seems like the testing model around, or the continuous delivery model around a Gmail plugin is probably going to be a fairly difficult process anyway.

    Sam: We never really had a good answer to, you know, what happens when Gmail changes their UI. The surprising part was it didn’t happen that often. We thought it would be breaking every week and we’d need all kinds of canary accounts and end-to-end testing to monitor that kind of thing. In the end, it happened rarely enough and we noticed quickly enough that we could usually just fix it on the spot. So even when we got better at testing and that sort of cycle of things.

    Paul: We had a problem or I guess solution with GitHub. So GitHub used to be super flaky around two, two and a half years ago, when they were still on Rackspace. And whenever something would go wrong with GitHub we would find out immediately. All of our, all the developments would start failing and so before they knew that there was a problem, we knew there was a problem.

    And we would often like, declare on Twitter, oh we think GitHub’s having a bit of an issue and two minutes later their status would tweet us and then we would retweet their tweet.

    Edith: So they were using you for their free testing.

    Paul: Well, our customers at least.

    Edith: No I mean you were acting as their tester and telling them that—

    Paul: Well, I mean it was very, you know, automated on our but we didn’t have necessarily automated fixes at the start. It kind of happened every, you know, two or three months and they would swear this is the last time and—

    Edith: It’s always the last time.

    Paul: And in the end it happened like six or eight times and by that time we had built a thing that allowed, you know, certain fixes, but it wouldn’t be quite, you know, fully automated process that you would expect if it was happening all the time ’cause it always seemed like it was gonna be the last time.

    Sam: And it’s probably always happening for a slightly different reason or in a slightly different way. It’s actually, it’s an interesting question to me in the subject of continuous delivery in general. People always talk about every time you commit you run your tests, every time you commit, you deploy.

    But that’s actually only one of the kinds of events that you’re dealing with in a software life cycle, right, when something’s actually running. I mean, sure you’re changing the code but also the world’s changing around you. And I feel like you always end up needing a little bit of both—you need something that’s gonna trigger when you change it, but you also need sort of an eye on everything you depend on. And, of course, even figuring out what all those things are can be interesting.

    Paul:

    So I wrote a blog post about this on the Heavybit Blog, I think a while back, and it was about third-party APIs you know, how do you test third-party APIs. And the conclusion that I came to was that, so if you run the tests as part of your test suite, then if something breaks, your test suite will be broken, but also some things will just like randomly break in production.

    And then of course you can’t deploy because your test suite is broken. So you’re in kind of the worst of both worlds. And so what I considered is, is that you need something which, you know, is just permanently monitoring. Like whatever API things you do constantly hits your personal accounts or your test accounts for those things, and then stub it out in your test suite.

    And that way you kind of get that best of both worlds, your test suite doesn’t fall over and you get an actual monitoring event when the third-party service goes down in production.

    Sam: I can tie this to an experience from LinkedIn, actually. So LinkedIn has the model where there’s a staging environment and a production environment. And the staging environment is—I don’t know if it’s technically called a staging environment because it’s not exactly production-like, but what they actually call it is early integration. Because it’s the environment in which teams integrate with other teams.

    And so the idea is you deploy your stuff into EI, early integration, as soon as you can so that if any other team depends on you or vice versa, you can find out there before the thing hits prod. But what you’ve effectively got in EI is this constantly changing environment full of dependencies and you have exactly this problem and different teams which using different solutions to, well, let’s write some integration tests which actually hit the dependencies and then we’ll run them in our test suite when we deploy to EI or when in the build step which—

    Paul: I can’t tell as you’re describing this whether this is a good thing or bad thing.

    Edith: Yeah.

    Sam: I think there wasn’t a clear conclusion, but my conclusion is that it doesn’t work for the same reason that you’re saying.

    You need to know that if your test suite fails, it fails because of something you did. And then you need something else that can fail because of something someone else did.

    But if you’re ever unsure which of things it is, then the signal is lost for both systems.

    Be sure to listen to the full episode. While you’re there check out Heavybit’s library, home to great educational talks from other developer company founders and industry leaders.

  • A Year in Review: Top Developer Tools Trends in 2015

    A Year in Review: Top Developer Tools Trends in 2015

    The ability to ship software quickly is now a competitive advantage for every business. As a result, continuous delivery, distributed microservices, containers and asynchronous computing have become the norm.

    With these changes, come new complexities, markets and dev tool players. It’s clear that the biggest levers for business success rely on developer productivity. Great developer tools have a dramatic impact on how quickly and how well software is built.

    Heavybit’s community of developer-focused entrepreneurs, advisors and investors reflect on the major trends of 2015:

    1. Continuous Delivery is Mainstream. “Continuous delivery is starting to become the norm, rather than just the bleeding edge,” says Edith Harbaugh, CEO and co-founder at LaunchDarkly. “Even huge corporations are adopting a release of once daily where they used to be once every three years. They are seeing a corresponding increase in developer productivity and user value.”
    2. Packaged Applications Replaced by SaaS. “In 2015 we’re finally getting the tools we need to orchestrate, monitor, and maintain service-oriented architectures with lots of small, independently controlled moving parts.” Says Ben Sigelman, co-creator of Google’s Dapper distributed tracing system, “The forthcoming architectural decomposition is going to turn software development on its head. Subsystems that used to be integrated as packaged applications will now be integrated as services, and the entire developer toolchain will need to adjust accordingly.”
    3. All Companies Become Software Companies. “Executive teams finally appreciate DevOps as they transition to becoming software organizations. Their competitive advantage is defined by their ability to write better software faster than their competitors,” says Lenny Pruss, Principal at Redpoint Ventures. “Consequently, DevOps and continuous delivery practices are transitioning from being nebulous concepts to actual initiatives that are being put into practice. Almost every organization I’ve talked to is actively re-evaluating how they deploy.”
    4. Modular development is here to stay. Says Bessemer Venture Partners’ Ethan Kurzweil, “Services like npm that facilitate the sharing and deployment of code modules will usher in a new era of collaboration among developers.”
    5. Revenue Traction in Dev Sales. “For anyone selling developer tools, there is a big question of monetization,” says Ian Saunders, CTO and co-founder at BillForward. “These sort of tools were sold as large license deals. The issue with this is that there was friction to upgrade, and a massive push to re-sell product rather than reinvesting in existing product. This area of pricing has seen a lot of traction, allowing developer tool companies to sell their tools in a more modern way.”
    6. Infrastructure as code is all the buzz. “By having your infrastructure as code, it’s far easier to automate builds and deploys and manage change,” says Neil Mansilla, VP of developer relations at Runscope. “The first AutomaCon, an Infrastructure as Code conference, showed that many developers were interested in adopting infrastructure as code but hadn’t put it into production yet, so the trend will likely grow as more developers and companies learn to adopt and implement it.”
    7. Container Category as an Ecosystem. Noah Zoschke, CTO and co-founder at Convox adds, “Everyone is working on building new tools and services for this container future (i.e. Weave container networking and visualization). AWS Lambda looks like a potentially huge breakthrough, eliminating everything about servers and providing ‘magic containers’ that spin up on demand to events.”
    8. Automation no longer requires a Heavyweight PaaS. “Containers are enforcing a uniformity of interface that will allow the types of automation that previously required a heavyweight PaaS, and therefore are opening up new niches and product categories,” says Cliff Moon, CEO and co-founder at Opsee.
    9. Data and Analytics Management Evolves. Says Justin Johnson, VP of developer advocacy at Keen IO. “The conversation is rapidly changing in the data analytics space — from how to aggregate data and make it available for the business to consume, to how to create a robust and fluid data pipeline.”
    10. Distributed Tools Require Greater Visibility. “Developers usually like to build their own tools, but first-in-class dev tool providers remind us that we have better things to do than build outside of our core competency,” says Hiro Yoshikawa, CEO and co-founder at Treasure Data. “As developers increase the number of tools they use, the need for integration and visibility increases. Businesses must integrate data sources (web, mobile, IoT, SaaS) and target systems (databases, CRM, visualization tools, notification systems). As a result, there’s a rise in new tools trying to solve this problem in the data and analytics space.”
    11. Dev to production parity remains challenging. Says Tal Weiss, CEO and co-founder at Takipi, “With continuous integration, you get to deliver features faster and see your code making an immediate impact. The problem is that a fast changing production environment amplifies vulnerabilities. No matter how much time and effort goes into testing and staging environments, your code is really only truly tested in production. This is a problem domain that attracts lots of interest as the answer lies with better tooling.”
    12. Continuous Integration Demands Continuous QA. “Companies who are implementing CI are hitting a bottleneck when it comes to QA. They’ve begun to realize that multiple daily releases require continuous QA practices and tools.” says Fred Stevens-Smith, CEO and co-founder at Rainforest QA.
  • To Be Continuous: Continuous Delivery and Mobile Development

    To Be Continuous: Continuous Delivery and Mobile Development

    In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk with Kevin Henrikson about about succeeding with Continuous Delivery and mobile development. This podcast is brought to you by Heavybit, a 9-month program for developer startups.

    Screen Shot 2015-11-30 at 11.31.57 AMKevin is partner director of engineering at Microsoft where he manages the Outlook teams.

     

     

    Edith Harbaugh, CEO LaunchDarklyEdith is CEO and co-founder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

     

    Paul Biggar, CEO CircleCIPaul is founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

     

    [soundcloud url=”https://api.soundcloud.com/tracks/235452076?secret_token=s-a8Rxo” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Paul: All right, Kevin. So, what’s your favorite thing about Continuous Delivery?

    Kevin: Good question. So, Acompli started kind of the approach, when we first started was just a couple people. And our deal was like, “Hey, you’re trying to get this thing to work, send emails.” So it was like, keep checking it in, get closer and closer to kind of the definition of it was working.

    And then as we started to experiment farther out, we said, you know, “What’s the right sprint cadence?” And we’d come from kind of traditional enterprise companies, where, you know you ship every six weeks or every four weeks and we’re like, oh, we can go faster.

    People are doing this agile thing. It’s like two week sprints, and we tried two weeks and realized that most of the work came in on Thursday afternoon or Friday kind of thing, ’cause the end of the second week and so what if we just do one week then most of that work will come in, you know, the first Friday. And sure enough, it worked, right? So 80% of the work still comes in on the last day or kind of completes on the last day or gets to the point of ready. And so, what we liked is that it gave us a much smaller kind of chunk of deliverable, right.

    So, like the train keeps moving but the size of what’s on the actual boxcars is smaller and so if you have a problem, it’s easier to roll back, easier to fix.

    Or if you have a problem, you can just patch it quickly ’cause you’re used to moving quickly.

    Edith: Wow. Okay.

    Kevin: So unlike in a traditional world where you ship something every six weeks, you’re like oh, we made a mistake, we need to go patch this. Like, the patch felt like an exception and you’re like, your muscle memory for patching was not there, right? So you’re like do we do it now? Is it worth it?

    And then it’s like well, if you’re five weeks in and you discover that thing, it’s like well, the six-week thing is out there where with, if you’re shipping every week, you’re like oh let’s patch it. We shipped yesterday? We just click the button again, and it builds and it goes. So you’ve built the muscle memory to move quickly, right. And so I think, what I like the most, kind of going back to the specific question, is just the fact that it lets you get good things done quickly, but also correct mistakes quickly.

    Edith: Wow. So this is a good time for you to introduce yourself.

    Kevin: Cool. So, Kevin Henrikson. I currently manage the Outlook, kind of non-Windows teams at Microsoft. So Outlook for iOS, Android, and Mac. Previously, I was VP of Engineering at Acompli, which was a mobile startup, focused on iOS and Android.

    Edith: Yeah so you had a lot of good thoughts right then. One of the things I wanted you to talk more about is why do you think 70-90% of the work is always on the last day?

    Kevin: Yeah, I think people are kind of deadline driven. I think generally they want to work until the last minute and I think software development ends up being this thing where you’re never done. And so the only way to define done is to actually draw a line and say this is done and then still some things miss that line. And so the tighter you give the window for things to get done, the more often they’ll hit that window because you’ve kind of broke things into smaller chunks.

    And also, because software’s kind of this long-lasting thing that never is kind of done done that you end up with, if the chunks are small, people are able to estimate and think is that going to take me a day? Very rarely does somebody think something’s going to take them a day that takes months.

    But very often, like if you think something will take months, it may take many, many months. Because it’s hard to guess, there’s so many variables, so many things are going to change in the environment, turn what you’re doing before you get there.

    So, I think the short answer is if you start with a smaller window and more discrete small tasks, those tasks are easier to estimate, they’re easier to consume, they’re easier if you get disturbed, like oh I’m going to go down and get a soda, or somebody asks me a random question ’cause some fire comes up, I can kind of make it up because I was only expecting to spend half my day working on that anyways.

    So, I think that kind of works out in our favor.

    Paul: So, there’s an implication with what you said there, that the tasks sort of grew because it was two weeks,, rather than–

    Kevin: Yes.

    Paul: So, I guess the phrasing is, when it was two weeks, was it that a one-week task had grown to two weeks? Or was it that things got cut when it got shrunk down to one week, and you ended up with technical debt, or something along those lines?

    Kevin: I think it was a couple combinations. I think if you feel like you have two weeks to work on a task, you’ll spend more time polishing or potentially doing things that aren’t on the critical path to getting it done.

    And I think that’s natural, right? Like, maybe it is you’re paying off too much technical debt, right? In the start-up world, you have to have that lever. And even just in general software development, like, you have to have a lever. We’re not launching the space shuttle here. There’s room for some wiggle room and imperfection.

    And I think breaking it into small pieces lets people focus on, “Hey, I’m only gonna do this” and “I have this much time to do it.” Where you say, “Hey, it’s got two weeks,” you may be deep into the second week and saying, “You know, I just don’t like my approach. I’m gonna change it.” And you basically end up throwing away a week’s of work, whereas, if you’re thinking that “Hey, the deadline’s coming in a week,” you’re gonna make that kind of epiphany, like, “Oh, I made a mistake,” or “I’ve taken the wrong path” much quicker.

    So, I think it allows you to break your decision-making and optimize on getting it done, because the window is kind of quickly approaching, being a one-week spread versus two.

    Edith: This might sound like Seven-Minute Abs, but why not two-day sprints?

    Kevin: Yeah, so I think in two days we– or one-day sprints, right, how do you continue to make it faster? So, for us, there was sort of one artificial gate, which was the fact that you really can’t ship quicker than about a week with Apple.

    And even then, it sometimes slips by, because, you know, four or five days, and then during a peak season, if there’s lots of apps, like a Christmas holiday, or around a platform update, you’ll see a spike in the review times. So, Apple kind of, and we run iOS app first before we port it to Android. So that ended up driving us towards a little bit to a week. Cause it was like, if you shipped on Monday, the app would get in the store some time end of that week. And that was a natural cadence to start the next one.

    So, it’s really hard without pushing the emergency release button, which Apple only lets you do once or twice a year, to ship more than once a week. And then, we kind of let the rest of the team fall behind that. If you’re only shipping the app once a week, let’s keep the new chunks of new work, or big bug fixes to that path on our service.

    The way our architecture is, and we can talk about that more, we spend as much, you know, 77% of the logic of the app is down deep in the service, because that we can update all the time. So, when we push a release and don’t wanna fix something, we fix from the service the same day.

    Edith: Interesting.

    Kevin: So, that allows us to have a much more agile approach, because the service is constantly being updated, but the app is kind of gated in one-week chunks.

    Paul:

    So, what you’re saying is that the biggest impediment to your shipping frequently is Apple.

    Kevin: The platform, yeah. The platform, basically, Apple’s– And specifically, it’s the fact that Apple has a review process, which is great. That Apple review process generally keeps the bar of apps better and puts some good discipline inside of the wider range of development, people that are coming to the Apple platform.

    But, it also, when you wanna move quickly, you can’t. The platform itself kind of enforces this four to seven-day lag in shipping.

    Paul: So, you’re now running teams that are shipping on non-Apple platforms. Did you guys make any changes to your release cadence as a result of that?

    Kevin: Yeah, so I think on Android, for example, we still kind of keep to the one-week junk of we plan a sprint. Build for a week. Release to our beta group or dog freed users, and then release to production. But Android gives you this advantage where you can do percent-based rollout.

    So, with Android, we end up releasing usually two or three times a week. So, you’ll release kind of The Release on Monday, at like 1-2%, kind of walk that out three, five, 10%. Usually at five or 10%, you can pick some new signal of crash or behavior that you wanna adjust, make a hot fix, push another release, start at 10%, 20%. Oh, you see another one you wanna tweak at 20%. And finally, from 20 or 30, you end up going all the way forward.

    Paul: Do those Android users help you on the other platforms, as well? Are they finding crashes that also exist on the other platforms?

    Kevin: Today we’re 100% native on both, so there’s no shared code between iOS and Android from the mobile app perspective. Obviously from the service, it’s 100% shared code. So, what we’ve actually done in the past, is when we launch a new feature that we think is either controversial, or has a high chance of risk, for example, when we launched iMap support.

    We launched generic iMap support on Android first, knowing that we could patch it quickly, and adjust if things went wrong, simply because there was no way to test the millions of random iMap servers in the world and get any kind of focus. When we talked to Gmail or Yahoo, or– Single point.

    Paul: That’s really interesting, actually, the idea that because the rollout, or the sort of safety features are only supported on Android, that people will end up building features there first.

    Edith: How deliberate did you have to be about putting stuff in the service, versus in the apps? Was this the lesson that you learned?

    Kevin: So, we had always thought of the service as the place where we always wanted to do as much of the logic that could break us– You just don’t wanna do it twice. You don’t wanna implement active sync twice. You don’t wanna implement an iMap stat twice.

    And it also allowed us to pick a protocol between our cloud service and the mobile app. It was optimized for mobile. So, think of the API, the shape of the API looks closer to the screen of the app. There’s a request for “Get my inbox,” and then make a change. Where, if you think of iMap, or the mirror out of email protocols, that we talk today, they don’t have that very linear type of approach of “Give me this screen. Give me that screen.”

    They’re thought of in a way where you’re going to sync all the mail down to your desktop client, and then manipulate it locally, and then sync changes back up in some either connected or disconnected state. So, the service was basically set to put complex logic, but also to allow us to get leverage, ’cause everything we could build in the service was one less thing we’d have to do twice on the client, because we picked a path that we didn’t support. We didn’t basically have a lot of shared clip or clients, essentially none, just the API.

    Paul: And it’s the same service that sits behind all the different clients?

    Kevin: Yes, so we have a single cloud service that basically powers both the iPhone client and the Android clients.

    Edith: It’s interesting, so at TripIt we had a lot of similar issues. Like, we were at one time one of the biggest, a lot of our users are Gmail.

    Kevin: Okay, awesome.

    Edith: People would sync their Gmail with us, and their calendars. Did you find there was a difference between the you or the app experience that Andoid and iOS wanted? How much did you bifurcate that?

    Kevin:

    Sure, so, a lot of people ask is like, “Hey, did you launch a feature first on Android or iOS, “or do you always have to make them “come out identical?” My answer to that is pretty simple. Most normal people carry one phone. They’re either an Android user or an iPhone user.

    So, they don’t know that you released it first on Android or iPhone, because they don’t carry both phones. So, we think of that, from a feature point of view, that when they’re done, then we ship it. Now, if we’re trying to line a press event around something, we’ll wanna make sure they’re both there before we do the press event. One may come out a day or two before the other. But, generally speaking, we don’t try to, you know, legislate you have to do it this way, ’cause it works on iPhone.

    In fact, we actually took a different approach, where we said, We wrote it one way on iPhone and then we went to Android. It wasn’t just like, “Oh go port all the screens over.” It was like, “What would this screen look like for an Android user?”

    So, we want our apps on the mobile device, and most mobile users wanna look at an app and feel like it fits in with the rest of their phone. So, the way that an Android navigation works, the way that people use the back button, or not, is very different on iPhone and Android. The fact that there’s widgets, and the way the widgets are exposed on Android, and the way that those people want to integrate deeper into the platform.

    Those kind of things we let go with the platform. Our view is write it in native code, ’cause it allows us to take advantage, and move quickly to the platform-specific features. But it also lets us tailor the experience, and the UI and UX to whatever that user base wants.

    Paul: It’s interesting that’s, that you talk about it that way, because in the old days, kind of ten years ago, when people were making multi-platform applications, there was a tendency of people to try to make it all match the same, the Firefox is an example here, or Chrome one of those tools that people use on all platforms, was designed to look the same on every machine, even though people didn’t have that machine.

    I think one of the nice things that’s happened with the mobile ecosystem is people have thrown away that concept, and they’ve made it feel actually native for each of the platforms.

    Kevin: Yeah, and I think that also goes to feature set, right? Like, you would think that the traditional app is like, “Oh, if it works this way on Mac, “or on Windows, or on the web, you would expect kind of 100% feature parity between all those platforms. And today on mobile, we go and select where you actually use it. There’s features that exist in mobile only.

    Because mobile has all this additional context of your location and where you are, and what you’re doing. Are you running? Are you driving? There’s other sensors in the devices today that allow you to build specific features that are richer. So, we always thought of a commonality.

    So, rather than sending more emails, send better email, right. Don’t say, “I’m running late.” Say, “I’m running late and I’m here,” and send a quick tap of a button to put the map and show you that, “Oh, I’m 20 minutes away,” or, “I’m on the train,” or, “I’m at the train station,” so that people can understand you don’t have to type out how many minutes it’s gonna take. You don’t know. Or, you walk in, or you drive in–

    Edith: Plus, people lie all the time. Like, “I’m on my way” has a pretty loose meaning of anywhere from “I might leave my house in 30 minutes” to–

    Kevin: And depending on the context, right? What that person’s thinking, so yeah, good point.

    For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.

     

  • To Be Continuous w/Special Guest Kris Gale: Effective Team Structure for Continuous Delivery

    To Be Continuous w/Special Guest Kris Gale: Effective Team Structure for Continuous Delivery

    In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk with Kris Gale about effective team structure for continuous delivery. This podcast is brought to you by Heavybit, a 9-month program for developer startups.

    Screen Shot 2015-11-13 at 9.33.45 AMKris is co-founder and CTO of Clover Health, a unique health insurance plan focused on driving down costs and producing improved health outcomes. Formerly, Kris was VP of Engineering at Yammer.

     

    Edith Harbaugh, CEO LaunchDarklyEdith is CEO and co-founder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.

     

    Paul Biggar, CEO CircleCIPaul is founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.

     

    [soundcloud url=”https://api.soundcloud.com/tracks/232834454?secret_token=s-3aRuL” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    Edith: So, Kris, what do you like best about continuous delivery?

    Kris: So, I like continuous delivery, because I think it’s sort of the first step of building a learning organization. It’s sort of a fundamental capability that a team needs to be able to learn collectively about what it’s doing in the world. The first step is being able to sort of ship increments of value continuously.

    The second, I think, is to be able to measure the result of any incremental change you’ve made, so you can back out of a bad path that you’re on, or sort of refine what you’re doing and know that quantitatively.

    The next step then, is to move from a sort of historical corporate planning, so this is a our six-month milestone, our two-year roadmap, our marketing commitments for Q3 of next year, into sort of an assumption that if we are shipping continuously and learning continuously because we’re measuring what we’re doing, whether it’s quantitative or qualitative measurements.

    The next step is really being able to assume that in your planning, and know you’re going to have more information in the future than you had in the past. So your plans that you made six months ago are necessarily lacking information than plans you would make today would have, so sort of institutionalize that.

    And then kind of the last part, and this is all kind of enabled by continuous delivery, the last part is building organizational fluidity.

    So as an organization, whether it’s your project planning structure, or your accountability to your work flows, are fluid, so if you learn something new, you’re not locked into, oh we have a sixth person, I don’t know, sign-up flow team, and we just discovered that we have no levers for moving engagement right now in the sign-up flows, so we’re committed to that and we’ve got this implicit prioritization that’s built into our org chart. So, to have organizational fluidity where you’re adapting the organization to what you’re learning continuously.

    Edith: Oh. You know that’s fascinating. Paul and I have talked a lot about, I think, more the mechanics of continuous delivery. We put it as kind of the last step to software org, when you’re really talking about how that changes entire organization beyond software. So, this would be a great time for you to introduce yourself, and tell us more about you.

    Kris: Yeah, so I’m Kris Gale. Most recently, I was the VP of Engineering at Yammer. I heard you talking about Yammer in a previous episode. I’d love to dig into any of that, if you’ve got questions. Currently, though, I’m the co-founder of a company called Clover Health. So, what we’re actually doing is offering health insurance for seniors, with the idea that we can fill gaps in care, and actually prevent hospitalizations.

    We’re not trying to be people’s doctors, but we’re trying to look across their consumption of health care and say, “Oh, here’s an opportunity ‘to inflect the probability with which ‘this population of people will be admitted to the hospital.” So, for example, making sure they get treatments regularly, get follow-up visits with specialists, that particular bio-markers don’t deviate outside of a specific range. And all this stuff is really interesting to me, because the health care industry as a whole doesn’t change what it does continuously.

    We’re sort of locked into what people learned in medical school, or what people learned on their first day on the job at, say a big health insurance company, or something like that. But, to be able to power workflow with software, you’re able to continuously iterate and continuously refine and optimize the intervention protocols that are leading to these prevented diseases.

    Edith: Yeah, I mean, I heard that you were pretty good at data loops, and that this is just basically using data loops for good, I think is how you say it.

    Kris: See, the way we were talking about data loops, we were actually talking a little bit before we walked up here, is this idea that there’s something you intend to do. There’s data that you collect in doing it. There’s your objective assessment of that data to determine whether you’ve added or removed value or improved something.

    And then closing the feedback loop, to get back now to actually modifying what you’re doing, and form find out new data. So, that’s sort of the data feedback loop, is from doing something to measuring it and deciding to do something differently the next time.

    The tighter you can close that feedback loop, the more opportunities you have to optimize in a short amount of time.

    Paul: So, it’s kind of what you might call the shipping loop, if you implicitly believe that data is a key part of shipping.

    Kris: Yeah, yeah, and I would say that I think I do.

    Edith: No arguments here.

    Paul: Well, I think it is interesting, because I think that a lot of people don’t actually have measurements as part of their feedback loop, and especially, I think this especially happens in a small organization, where you have a start-up that sort of comes out of a founder’s vision, or something like that.

    And so, one of the questions that I’m very interested about is, you were doing a big thing, with lots and lots of people. And now you’re doing a very small thing with, I presume, a small handful of people. How much of what you’ve learned still applies at that very small scale?

    Kris: 

    So, yeah, a lot of the project workflow strategy that we came up at Yammer was really about how do you maintain that agility, that velocity of value delivery, and ability to react to new information, and not be locked into, “Well, this is just the way we do it,” in the face of new information. At scale.

    So, Yammer’s whole process was really about, how do you take these advantages we had, when there were like four or five of us in the really early days, and not have that big bureaucratic company, “Oh, we know we should be doing something better, “but we’re not,” because of momentum, inertia, bureaucracy, all these things. Like, how to break that. So, what’s cool about starting over with a much smaller team is, we’re just there again. We’re back at that five person size.

    Edith: You’re a lot bigger than five people.

    Kris: We are now, yeah. So, like nine months ago, we were. We’re like 65 people now, already. It keeps growing really, really fast. It’s been really helpful to me to have been through the process of scaling a software engineering team and seeing what that looks like. So, it’s like, “Oh, I’ve seen this problem before, “not worried about this.”

    Paul: So, when you started, did you immediately set out organizational goals, and mission, and values, and that sort of thing that would help the company scale it?

    Kris: Yeah, no, that’s a great question. I’m obviously a really big believer in the way we approached this problem space at Yammer, but I didn’t want anything to be dogmatic. I didn’t want anything to be–

    Paul: You wanted it to grow organically.

    Kris: Yeah, I wanted it to grow organically, to a certain point, but there have definitely been times where I stepped in and I said, “Nope, I’m buying a physical board “to track our projects, “because, just the way we’ve been doing this “with sticky notes, “I know that it needs this other dimension “of information capture. “I know that we’re suffering because it’s not there.

    “So, I’m gonna actually pull the boss card and do this.” There have only been like three or four times I’ve done that in the last year, but it’s things where, like– Okay, I’ve seen where this goes off the rails, if we don’t address this right now.

    Paul: And is that a result of knowing, sort of directionally, how the company should be doing things, or setting things up in some way that you didn’t get, that you didn’t end up in these bad places?

    Or was it all sort of retroactive, and you know, a couple places where you pulled the boss card?

    Kris: Yeah, it was mostly, like, observing patterns, saying, “Okay, we made this mistake three or four times, “and I’m noticing that our process or our approach here “is divergent from a known good approach “that I’m aware of.

    “So, let’s at least–” I still wanna give the team full empowerment to iterate and own the process themselves, but let’s at least just pull the boss card, get it back on the rails, and then say, “Okay, let’s see how this works.” Just give everybody a chance to work in a way that I know to be okay for a few iterations, and then give them the keys again, and say, “Okay, take this where you want.”

    Edith: Also, Kris, what I’m dying to know now is, what are those three or four key things where you gotta play that card?

    Kris: Yeah, so, one is single project focus. Really, really important to me for software engineering teams. So, what I mean by that is, if you are on a project, that is your only accountability. You are not being split between three deliverables.

    The other is that everybody work in cross-functional teams. So, that means not giving, for example, I might tell you, “Build out the back end for this feature, and I’ll do the front end.” So, in order for us to do that, we would have to agree to the abstractions ahead of time.

    And again, I’m pretty opposed to planning, and locking in planning, because the information that we have now, when we’re trying to lock in those abstractions, is going to necessarily be incomplete.

    Edith: I love that quote, because it’s so true. I think you put it more, it’s like you always know more in the future. It seems so obvious.

    Kris: Yeah, I was actually– So, we had Parker from Pivotal Labs over at the office yesterday, and he said that there was this saying they had a Pivotal, that I really, really liked. He said somebody there would always say, and I wish I could remember who it was, would always say,

    “You will always make better decisions with more information, and you will always have more information in the future.”

    This is so much more concise than any way I’ve ever said that. But I think it exactly captures it.

    Edith:

    But, on the other hand, my old boss, Gregg Brockway the founder of TripIt, he always said, “A good enough plan today is far superior to a perfect plan tomorrow.”

    Kris: Absolutely.

    Edith: He’s like, “You just gotta start moving.”

    Kris: Yeah, but I think the key is not locking yourself into that “good enough” plan. To allow yourself the mutability of changing that plan over time, so you’re not constrained by that.

    Edith: Yeah, but that was TripIt. I mean, I think he’d seen so many, because he came out of the travel industry, where people just got paralyzed. Like completely paralyzed. “Oh, we need to plan for five years, “and what business travel will be like then.” It’s like, well, nobody knows.

    Paul: I think the kind of key point to that is that in order to get that information, you have to get moving today with whatever information you have available. Otherwise, you’re just not gonna have that future information. You’re gonna be in the same place.

    Edith: Well, look, sometimes, though, it’s painful. Sometimes you move backwards, then. Like, you don’t always move forward. Sometimes you start moving, and when you’re looking back, you’re like, “Actually we took, like, three steps back “before we started moving forward.”

    Kris: It can definitely happen.

    So to complete that thought, when we assign teams, we would always assign a cross-functional team with a business objective. So, it would never be I give you a back-end task, and I take the front-end task. It would be collectively, let’s accomplish this business goal.

    So hopefully, we’re iterating day by day on the abstractions. So, we are collectively accountable to one another, and to this business goal we’re trying to deliver. Not onto some specific set of tasks that we broke off when we first decided to assign out the work.

    So, that’s one of the other ones, is always cross-functional teams. Always single project focus within those.

    Paul: So, there’s an interesting thing that you’re talking about there. It’s not just cross-functional teams, but it’s also that those cross-functional teams get put together for a specific project, is that right?

    Kris: That’s absolutely right.

    Paul: So, are those teams teams that will have worked together in the past? Is there explicitly trying to make sure that the people know how to work together?

    Kris: Yeah, so this is one of my other core principles that I am pretty, pretty set on, which is rotation, is this idea that cross-functional teams are rotational and ephemeral. So, that would mean that maybe the three of us come together to ship a feature.

    We’re all within the same organization, but when we’re done shipping that feature, we’re gonna break up.

    You might be the back-end person that goes onto another iteration of this feature area, or maybe, you know, the two of us work on some other feature. Maybe it’s three completely different people. But constantly rotating people around, I think is really important in an organization.

    I have totally heard the arguments about, you know, you can build high trust teams if they’re stable and stay together. But I’m trying to optimize for trust across the entire organization. So, across Yammer, we had 220 people in engineering. They would all rotate through these two to 10-person cross-functional teams together.

    Which meant that you, basically within three to six months, would work with a pretty significant portion of those 220 people, and would start to build trust across the larger organization. Whereas had we just torn off four people and said, “You’re a team,” they would– And I’ve seen this throughout my career in engineering, where it’s like, “We do everything right, and that team has no idea what they’re doing.”

    Edith: It’s their API’s are bad.

    Kris: Yeah, yeah they use this weird language, or this approach, or we use this style guide, or whatever. So, it sort of forced that rotation, so that we’re all sharing best practices among one another and building, maybe lower trust, maybe not getting to that very, very, very highest level of trust among a group of four people, but across the entire organization, getting to like 40% or 60% of what’s possible versus 0% if you don’t ever work with those people.

    Paul: So, interested in structurally how that ends up, because one of the things you got when you split people up by team is you end up with a sort of microservices architecture.

    And this team works on this repo, or this project, or whatever works on this service or this API. And when you have those teams that are forming and splitting up and they’re ephemeral, and it’s only a small period of time, architecturally, what does that do to your application or your architecture, you know, across your entire product?

    Kris: Yeah, no, totally. It was actually very intentional that we ended up with that kind of architecture for working this way. For two years, we had a big monolithic Ruby on Rails code base, and for two years we said, “This is a big problem. We should break this up.”

    And everyone agreed, and everybody was on board with breaking it up, and two years later, we had never done it, because we continued to work in such a way that supported that sort of architecture.

    So, the key for us was to say, well one of, the– So, something to dig into here a little bit, is these cross-functional teams are made up of representatives of what we call functional teams. So, front-end engineering might be one.

    Paul: Right, right.

    Kris: We had one called Core Services, which I actually owned the service-oriented architecture that you’re describing. Or there’s little micro-services that collectively made up our service-oriented architecture.

    Paul: And this is sort of a matrix style, where you’re a member of the team, but you’re also a member of the back-end team, or you’re a member of the front-end, or the core services, or whatever.

    Kris: Yeah, so I might be a member of core services. My manager is one of the core services managers. But all the work I do is going to be within the context of the cross-functional team, with the front-end engineer, maybe a client’s engineer, whoever’s necessary to get end-to-end with a value proposition.

    And then, I’m always going to be a core services member, when I go into these cross-functional things, but it’s the fact that all the members belong to functional teams that makes that team cross-functional.

    For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.