Category: Doin’ DevOps

  • Don’t Make Me Code Podcast: A New Heavybit Series on Developer Experience

    Don’t Make Me Code Podcast: A New Heavybit Series on Developer Experience

    Welcome to Don’t Make Me Code, a new podcast that discusses developer experience design and the unique challenges you face when building developer facing products. Don’t Make Me Code is hosted by Steve Boak, co-founder and CPO at Opsee, and David Dollar, co-founder and CEO at Convox.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    In this first episode, Steve and David discuss the origins of the term “DX” (developer experience), they share their thoughts on what makes a great developer experience, and they talk about some of the challenges they face when designing developer-facing products.

    [soundcloud url=”https://api.soundcloud.com/tracks/242787724?secret_token=s-09pht” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]

    David: Hi, I’m David Dollar. I’m CEO and co-founder of Convox, a Heavybit company. I’ve been working in developer experience for about the last three or four years now.

    I’ve founded the DX team at Heroku, worked on a whole bunch of open source tools including Foreman, and now working on an open source platform as a service.

    Steve: I’m Steve Boak, the co-founder and head of product at Opsee. We are a monitoring company, and I’ve been working at several monitoring companies over the past six years, helping to help developers improve the experience of their products as well as find bugs in their code and improve the quality of their products.

    And though I didn’t hear the term DX for the first time until a few months ago, it’s been a passion of mine for several years, and it’s nice to finally attach a name to it. And I’m excited about this first episode because we get to unpack the term a little bit and figure out what it means.

    I’ve thought one of the most interesting parts of your background, David, was that you were part of this founding team at Heroku, that sort of created DX, and I’m interested to hear more about that.

    David: Yeah, so I guess this was maybe late 2012, maybe early 2013, right as at Heroku we started to expand, out of being just a Ruby host into being a host for really any language. That meant we also started to see more, you know our standard user before that had been sort of like, and I was ten user building rails and was very, there was sort of one happy path for that.

    As we expanded languages we had Windows users and Linux users and sort of, everybody coming together to try and build really any thing. And that started to get out of hand really fast. So, really, we formed the DX team, brought together basically the people that were in charge of the various languages at Heroku. Somebody in charge of the Ruby, and somebody in charge of Python and Node. Brought us all together into one team and really decided to try and make a consistent experience, sort of the Heroku way, across all of these languages and …

    Steve: And was it called the Developer Experience team from the beginning?

    David: It was, yeah.

    So the first time I heard the term was from Adam Wiggins. He sort of like came to me and said, “Hey, I think we should build this DX team. What do you think?” And yeah, so I mean it sort of exactly described what it is we were trying to do, which was make this nice experience for people building things on Heroku.

    Steve: And when was that? What year was that?

    David: I think it was the end of 2012, but it’s kinda fuzzy at this point. But yeah, it was a while ago.

    Steve: And it does seem to continue to be an important thing for Heroku. I, just in researching the term of it, saw that there’s a Heroku.com/dx, this is a big part of how they market themselves.

    And clearly it’s been important to Heroku for a long time. Like I said, it’s interesting to now talk about this with a name because a lot of us are doing this and have been doing it for a while but, giving it a name gives it so much more weight and it kind of marks how important it is for all of us. Mmm-hmm.

    David: It’s interest … you think of DX and what does that actually even mean? And its, there’s a certain type of user that you’re talking about now, right? When you’re talking about designing an experience for this user that likes to kind of get under the hood and break things apart and do things their own way and sort of knows what’s going on and maybe you could even consider a power user.

    Yeah, you want to make sure that they can sort of do whatever it is they’re trying to do but that the options aren’t overwhelming and that it’s not complete, just utter chaos.

    Steve: Yeah, it’s almost like this partnership that we as the makers of tools that developers then use to build tools for their customers, if they don’t like the product that they’re using that may show in the product that they’re ultimately trying to build. And so it’s this partnership where we all try to write good code and create great products together.

    And I think that’s part of what makes this different from user experience design as a whole, and why it’s interesting as a topic all unto itself.

    David: I definitely agree with that.

    The most rewarding thing about working, I guess in developer experience that I found is when people do things that you never expected them to do.

    You put all the pieces in place, you thought like, “Oh, maybe somebody will use these pieces to build X,” and it turns out that somebody comes and builds something you never even thought of. I think that that’s just really rewarding to see that.

    Steve: And it’s interesting that, I think a lot of product builders in general would probably share that sentiment in terms of the unexpected ways that people find to use their products, and the hacks that they create, and I’ve heard, even with consumer products you know, people being surprised by how the users are actually using the product in ways they never expected, but with developers that really is to another level because they may try to break your APIs.

    They’re much better at exploring the depths of your code and then creating tools that you never expected with it, which is both surprising and wonderful for us.

    David: It’s still largely about aesthetics, right? Like, you’re a good user experience, a good developer experience.

    At the end of the day is this thing easy to use? Is it pleasant to use? Are you able to figure out what’s going on? Does it make me happy to use the thing at the end of the day? So yeah, it takes a different set of things when somebody’s trying to come in and actually build something on top of your product.

    Steve: I remember in the early days of the Google Maps API just how many experiments and apps were being written with that.

    I think, partly, because of how easy and accessible they made it for developers. And I heard a talk from someone on that team talking about that the product that we all knew and loved then was not the first iteration of it and they actually had a much harder to use initial version. Do you know more about the history of that?

    David: I don’t really know.

    Steve: Yeah, but that was one of my first exposures to using another company’s API and it was a really rewarding experience. And they had all this boilerplate code that you could very easily lay down to plot pins on a map and see that.

    Especially for me as a designer it was so rewarding to get that gratifying output. The actual map on the screen with the pins that I had laid out. I think that is a really great way to measure how effective and how elegant your developer experience design is, in how quickly a developer can make something themselves.

    David: One of the barometers I use for the things that I’m building is you look for the blog-posts people are writing around your tools.

    Are they writing a blog post of like this thing was so insane I couldn’t figure out how to use it? Like here’s my giant blog post teaching you actually how to use it because I had to figure all this out. Or are they talking about how easy it made their life? Like, look how awesome this is, you should try it, too.

    Steve: Yeah, and that makes me think of the JavaScript world today and just how many frameworks are being launched, and the competing arguments for this or that. And in some ways this has really gotten good for the development community as a whole because there are so many choices now, and so many good frameworks to choose from. Almost too many, actually.

    And I think we’re starting to see some of that in the JavaScript world. Just competing ideas of how things should be done. And that almost raises a counter-argument to the value of developer experience design that when we open up too many possibilities and don’t create a set of tools, I think you described it in your Heavybit talks as like, sharply …

    David: I like to compose sharp tools. I mean it’s sort of like the Unix philosophy, right? Like, you want to make tiny tools that do one thing really well. You generally want to have some good examples or defaults, like, this is how to use these tools in combinations to make interesting things. But having those tools be sort of separate and composable is where you start to get really interesting things happening that you didn’t expect.

    Steve:

    Yeah, that each piece does its thing really well but also that when combined they can form something greater and that the whole is greater than the sum of the parts.

    David: Right. Like, a good example of that, it’s not really developer experience, but I have TTT, it’s sort of like an infinite variation of all these little components that you can put together, but they also have some default pipelines that come out of the box, you say, like, “Okay I really just want to make my email go to Google drive or something like, just like these nice defaults.”

    It’s really important, I think, when you have all of these choices to really also have the guides through them.

    Steve: Yeah. We use the Amazon APIs a lot now, and they’re an ecosystem all unto themselves. For every product Amazon offers inside of AWS there’s a unique API and many of those APIs seem to have differing views of the world and so, they in some sense they’re prolific in the amount of tooling they create for developers and they clearly have won a huge share of the market with just how much they offer, but it still feels to me like they’re a counter example.

    They seem to have APIs that a lot of us use but not APIs and tools that a lot of developers enjoy using.

    David: Even despite that, like pretty much everybody is still using their Sphinx, right?

    Steve: Yeah.

    David: So there’s something to that.

    Steve: You know, Facebook’s API or Twitter’s API I can’t speak to how good or bad those are but man, a lot of people use them. And there’s something to that, too, that the product as a whole has so much value and impact on the world that you are sort of forced into a place where you need to use it.

    And it doesn’t absolve you from the responsibility to make your tools great for your developers and your partners in this, but yeah, it’s sort of mind blowing to see how much can be done when you are hitting on a big share of the market.

    David: The AWS database is another great example of that. There’s so many options, so many ways to do things that you can, like, it’s great. You can really do anything. Like the sky’s the limit. But there is, like, finding the happy path can often times be, it’s just a lot of trial and error at the beginning.

    You end up just having to build up some expertise around that stuff and eventually you can, okay now I can understand how to design things using these tiny permanents, but yeah there’s not a really great, like I just have some code and I want to put it on the Internet.

    Steve: Yeah, and part of Convox’s business is really helping developers do this, right? That there are in fact so many possibilities to create an infrastructure in AWS, that giving a little guidance is probably a good way of helping developers through that process.

    David: Yeah it’s basically just trying to create a happy path. I think there’s a lot of choices that you don’t necessarily have to make.

    Steve: Yeah, we too, at Opsee, use AWS heavily. It’s a monitoring product designed just for Amazon web hosting. And when we were creating this product we were assuming we would see a pretty high level of consistency in how our customers are configuring their environments, but that was shot down almost immediately.

    There have been no two environments that are the same and so now we’re dealing with a wide variety of customer environments and it’s an example of just how much people can do with these tools, but how differently they’re choosing to do them.

    That though does get to this other interesting question of how we measure the quality of developer experience and how we know that we’re doing well. And we talked already a bit about time to value and how quickly developers can make tools with our products, but then I think it’s also interesting to talk about how that becomes part of how we price these tools. Like, we try to look at how much development time we can save our customers and then try to price the product based on that. Do you try to do something similar?

    David: We’re trying to think about this as a peer developer tool, so rather than, you know a lot of people try to charge for resources, things like that, we’re doing more of like charging per developer that’s using the thing. So we actually, actually creating value for developers I think is sort of the core of what we’re trying to do, so we’re approaching it from that angle.

    Steve: Yeah, and monitoring being a little bit different, we’re mostly trying to measure how much code our developers don’t have to write when they’re using this tool, and what they would have to do in a competing world, but

    I think the one thing that all Heavybit companies, certainly in all developer tools companies share is that at it’s core we’re trying to measure our value that way: How much code can we save developers from writing, and how much time can we save them?

    And that feels in many ways different from normal user experience design and consumer user experience design. That we have to look at that, the value that we’re creating for our users in order to assess our own value. So as designers of developer experience products I think it’s interesting to talk about the inspirations and the other companies that are doing this well.

    The first time I had ever really thought about developer experience as something to be measured was looking at Stripe and how … You said earlier how blog posts and recognition within the community can mean a lot. And I remember hearing lots of developers talk about the quality of Stripe’s documentation, to the point that it became synonymous with a great developer product.

    Like do you have Stripe-level docs? And so they were always a barometer for, you know, are we doing well and who do we measure ourselves against?

    And I feel like they’ve done a great job not just in creating great documentation and great APIs but in great products. I mean even their web design. They’ve taken all aspects of their design to a level that I think was not common in our world when they were doing it.

    David: Yeah definitely. I mean, we use Stripe and I just remember when I was first clicking through their web I was just like, “Wow, this is really nice.” You just have that thought to yourself while you’re using it, so yeah it’s very pleasant to use.

    Steve: And I don’t know, you know in the early days of Heroku I don’t know if you saw much of this but certainly five, six years ago if we designed a UI that was polished and really nice looking there was almost a bit of hesitation among our developer customers that we didn’t have the street cred like, “No, you can’t be good, you can’t be real because no one in our world would do this, would make a product like that.”

    And I think that’s changed a lot over the last few years, but we were almost fighting this uphill battle to try to create an elegant and beautiful user experience because people didn’t expect that.

    David: I certainly think that sentiment has changed. I think having a good design aesthetic, a sense of that, is actually like seen as a good thing in most developer circles now, as opposed to being like, maybe you don’t know what you’re talking about if you have a sense of style or something.

    Yeah, it seems to be important. People really look for it when they’re looking for tools or products and that’s also something that we tend to see celebrated.

    Steve: That’s about all we have time for today.

    If you’re interested in being a guest on the show, or if you have a DX topic you’d like us to dive into you can reach us at dmmc@heavybit.com or on Twitter @dontmakemecode. To learn more about Heavybit visit heavybit.com. And while you’re there, check out the library. It’s packed with amazing talks from founders of developer tools companies and other industry leaders.

  • To Be Continuous Podcast: How Do You Deal with Your Data?

    To Be Continuous Podcast: How Do You Deal with Your Data?

    Welcome to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. Edith Harbaugh and Paul Biggar talk with Peter van Hardenberg about about continuous delivery in the context of organizational structure, databases and pricing.

    Peter is a co-founder of the Heroku Postgres product team. For more than three years he has been responsible for designing, developing and operating the largest cluster of Postgres databases in the world. He knows how to get databases running, keep them safe and ticking fast.

    This podcast is brought to you by Heavybit, a program that helps developer startups take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/241491956?secret_token=s-oZ2T7″ params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=’166′ iframe=”true” /]

    Paul: So Peter, it’s traditional to ask you, what do you like best about continuous delivery?

    Peter: Well, I’m increasingly old and I’ve done a lot of jobs and so I remember delivering Gameboy cartridges to the market and a Gameboy cartridge is a physical device that goes into a Gameboy, hence the name, and that process involves physical manufacturing and you get one shot and so there’s a tremendous amount of basically wasted and un-fun energy that goes into making sure you never make any mistakes.

    My favorite thing about continuous delivery is the ability to make mistakes and move on with them.

    Paul: Nice.

    Edith: This would be a great time for you to introduce yourself.

    Peter: Yeah, hi, my name is Peter van Hardenberg. I’m a Heroku old-timer, I’ve been with Heroku about six years now and I co-founded the Heroku Postgres team and have built a lot of products since then.

    Edith: So it’s funny you’re talking about the Gameboy because Paul and I were just talking about that.

    Peter: About Gameboys.

    Edith: Because I said one of the things that had fueled the rise of Zynga was continuous delivery.

    Paul: That’s the first thing we found about continuous delivery that was actually bad.

    Peter: So I guess they’re probably still continuous delivering somewhere. In the great startup in the sky. They’re still around, aren’t they?

    Paul: They’re still around, they’re still public and,

    Peter: Farmville will never go away, I’m sure.

    Paul: I think they’ve moved into other lesser things.

    Peter: Developer tools.

    Paul: Imagine.

    Peter: Gamified developers. Well actually, it’s how all developer tools are becoming now.

    Edith: Github is gamified. How many chickens do you have, can you make a nice pattern out of them.

    Paul: New watches.

    Peter: It’s a bit like a game except if you’re losing the game, you lose your job.

    Paul: Or you just can’t get new jobs.

    Peter: Well, that’s actually probably more accurate, yeah. Yeah it’d be a weird company if they let you go because you didn’t have a long enough commit streak.

    Paul: On open source projects that weren’t your job.

    Peter: I like that, actually, let’s make that a game.

    Paul: We were talking in the last episode, as well, about open source funding models, or the anger over lack of open source funding models.

    Peter: That’s a tough problem to solve.

    Paul: That would be an interesting one. You get fired if you don’t have enough contributions.

    Peter: Yeah, just put it on your quarterly review, current Github streak.

    Paul: Let’s put this suggestion on Hacker News and see if people agree with it. I can pretty much picture how this is going to go.

    Peter: I try not to read the comments.

    Edith: So what do you think is liberating about the ability to make mistakes?

    Peter: Well, I mean, more important than the ability to make mistakes is the ability to try things, right, I mean we, I’m so accustomed now to being able to see in great detail because of a SaaS model, sort of the reality of what my product is in the market and then vice-versa, I can turn that around and say, well, basically live and in real time, what’s happening if we put something out there, do people care?

    Does it help with adoption? Is it suddenly throwing a bunch of errors? And if there’s a problem, you can roll back very quickly. You know we run a database product and so availability is really, really important but only for a very small core of the service. People are generally pretty understanding as long as your, sort of, problems and outages are brief.

    Small and, sort of, blast radius and quickly remediated and I think that they prefer to get features vs. stability, sort of, at the periphery. At the core, they need stability over features, so, you know if you’re a stripe, right, you need to keep the payment processing online but if some ancillary piece of your API has a problem that’s probably not as big of a deal.

    Paul: The dash part is less important—

    Peter: Right.

    Paul: than the payment going through.

    Peter: Well, and responsiveness and progress are so important, and I think because we’re in such a changing market, the ability to drive forward is at least as important as the ability to keep things stable but you can’t do both in the same place, basically, right?

    I think continuous delivery is cool because it’s separated, sort of, a slow-moving core from a fast-moving periphery.

    Paul: So when you’re creating a software product, you really want every single component of it to be continuously deliverable, which means you want to be able to migrate from the thing that you are doing now to the new thing.

    And so it’s interesting, or there’s an interesting thing about databases because databases, I think, were fundamentally not built for this model.

    Peter: They’re heavy.

    Paul: Right. We had a problem recently where we deleted an index.

    Peter: Ah, yeah.

    Paul: And, whoops, that index was in use. And we thought it wasn’t in use but it was in use.

    Peter: You should’ve been using Heroku Postgres, it would’ve told you.

    Paul: Oh so, that’s interesting. So the thing that I wanted was to be able to, you know, partially delete that or let’s run all the queries without that and then after an hour, I can actually delete it. What does Heroku Postgres offer you?

    Peter: Well so we have a number of diagnostic tools where you can see exactly how often an index is being used. In addition, next time you’re trying to delete an index, wrap that in a transaction, so say, begin and then drop the index, then do whatever and then say, rollback, and that way, you can actually test as though you didn’t have the index because Postgres MVCC model means that all that stuff is totally transactional.

    Paul: So you’re, when you say do all the stuff in between, what do you mean?

    Peter: Whatever queries you want it to run. Also, and we’re getting into minutia and then some baseball here but you can actually disable indexes specifically, sort of at a session level as well.

    Paul: Oh, interesting. This is all Postgres or Heroku?

    Peter: Yeah, that part’s just Postgres I mean, we built the tooling around it so that you can do these things, kind of, the easy way but, I’ll tell ya, if you read that 3,000-page Postgres manual, you’ll find some real gems.

    Paul: Well so part of the problem might be using a thing that’s not a real database?

    Peter: Oh, you’re not using a real database?

    Paul: We’re not—

    Peter: Oh, that’s right, I forgot.

    Paul: I mean we are using a real database, we’re using many databases but that particular one was not in a real database.

    Peter: Well you know, even baby databases grow up to be real databases someday.

    Paul: Yeah, well, fingers crossed that that particular database that starts with an M and that we shall not discuss.

    Peter: Oh, we don’t like to speak ill of anything.

    Paul: No, no not at all, especially Mongo.

    Edith: No, you said the word!

    Peter: You said the M word! So coming back to, sort of continuous delivery of every part of the stack, yeah, it’s interesting, I mean we’ve had some interesting challenges with databases in particular and I actually gave a talk at Heavybit a while ago about, kind of, how we treat databases less as part of a service in sort of the classical sense but more like a factory.

    So basically what happens is when people request databases from us, we stamp one out and then, you know, we try and keep it pretty low entropy from that point on so we have a lot of tooling and support services around the database but the actual database itself, once it’s given to a customer, we mostly try and keep our hands off it.

    Now if there’s security issues or performance issues or whatever, we’ll go in and we’ll kind of warm it up and do some work on it. But even still we mostly prefer to actually replace the database. What we do is we create a replica, bring it up to speed and then we transition the load over to it. And that’s kind of a form of continuous delivery for data services, which has been surprisingly powerful as an abstraction.

    Edith: So you’re basically, you’re treating them as cattle, not pets.

    Peter:

    Oh absolutely, it’s all about the cattle and model and when you’re farming a very large crew of livestock, you can’t really afford to give each one their own personality.

    Paul: This reminds me a lot of, I think, what started out as an IMVU way of doing data migrations, I’m sorry, of doing schema migrations without locking a whole table. For our people at home, I’ll describe this.

    So when you want to add, let’s say you want to add a new value and it has a default to a particular table in your schema. There’s a possibility of that if it’s a particularly large table, for it to be locked while that happens. At least this used to be the way, I think modern databases are much better at this now.

    Peter: I believe for some default values, it’s now the case that you don’t need to do a table lock but that might be pushed until 9.6 or Postgres, at least but it’s a common and challenging operation.

    Paul: And so what IMVU started doing was instead of adding to a schema, they said that for every table, that schema is immutable and instead they created a new table and then they migrated the data from that table so every time you want to read from the user, you’re reading from user table 38 at the moment and then you start to read from user table 39. If the user isn’t in user table 39,

    Peter: You fall back to 38.

    Paul: fall back to 38, find it, migrate it, write into 39 and then do the thing. Then you have a background process.

    Peter: If you have any listeners who are listening from the world of the Internet and would like to try implementing this technique for themselves, Postgres has something called a schema search path so you can actually, basically make each schema its own version and then you have a search path so if the table wasn’t in the newest version, it would use the last version’s search path.

    Paul: Oh, wow.

    Peter: You could set a version on your code then basically, as you make new versions of the tables and the new schema, you can kind of move them forward. This actually sounds like it would probably work. Now it’s really easy to say that from sitting around a table here in beautiful San Francisco but you have any listeners out there who do give this a try, have them send me a note.

    Paul: I was not expecting this to be a session about continuous delivery databases and the intricacies of modern versions of Postgres.

    Peter: I have to say, it’s one of the hardest things is basically, how do you deal with your data? I mean, data is sort of fundamentally heavy, I mean I think about it as having mass, which is to say, if you’ve got a few terabytes of data on a disk somewhere, just getting it out of that server and into another server is gonna take hours.

    Paul: Right.

    Edith: Right, just querying the data, loading it all into memory, that’s gonna take time and so this is really a challenge for continuous delivery and people have tried all kinds of schema versioning and replication strategies and sharding strategies but, you know, the old question, okay you deploy a new version of your code and it needs the new column but the new column’s not there yet. Okay so you move the data to the new column and then you deploy your code but then the old code doesn’t have it.

    Paul: Right, right.

    Peter:

    And these are hard problems and no one’s really done a great job of answering them. So far, sort of the state of the art is, you know, you continuously deploy the new version of your code that supports both schemas then you continuously migrate your database and then you deploy yet another version.

    Now without continuous deploy this is even harder—

    Paul: This has to be tough on a Gameboy cartridge.

    Peter: Oh yeah, let me tell you, I think we had 4k of RAM or something like that so at least it was fast. But yeah I think data and continuous delivery are both challenging interactions for sure.

    Paul: We end up at this problem in the testing world when large customers, in particular, large enterprises use the old way of testing, which is you actually use a dump of your production database as part of your test suite and one of the major challenges we get when we’re looking at older companies who are coming to the new world, is, you know, your server’s over here, my server’s over there and I want to get a 5 Gig database dump onto your server to run my tests, how do I do that?

    Peter: Right and it’s trying to, like, move a bathtub through a straw. So how do you do that? You just wait, right?

    Paul: We wait until they stop doing that and then they can become our customers.

    Edith: So do you turn customers away then?

    Paul: We don’t turn them away but we don’t really support that use case, they’re going to have a miserable experience.

    Peter: I mean there’s data migration again, right, just coming back to the physics problems. There are things that are getting better so,

    Paul: The speed of light is improving?

    Peter: the speed of light has gone up 13 percent in the last year, did you know that?

    Paul: I don’t know if you’re joking, I presume you’re not.

    Peter: It absolutely has not, yeah. They call it a constant for a reason. I suppose it could.

    Paul: I thought there might be something about, like,

    Peter: Dark matter, no but it was very convincing the way I said it, wasn’t it?

    Edith: Like this is Hindenburg’s Principle?

    Peter: It’s uncertain. So anyway, the logical replication scene is getting better. Amazon’s just launched a new service, it’s three letters long.

    Paul: Oh really?

    Peter: Yeah, uh, got it, hang on. DMS, Data Migration Service. And so through black magic and probably by not working very well, it will move all your data from one database to another in sort of a streaming online way which is really cool if it works.

    Paul: Okay.

    Peter: But, you know, moving from Oracle to MySQL

    Paul: So you can talk to this service as if it’s MySQL and it will pull it from—

    Peter: The way it works is you basically set up a migration job and if you think about it, it basically queries one database and writes to another and it’s a little bit of a hack job in the sense that you have to run, like, a desktop client if you’re moving from a local machine. It’s all very complicated and kind of baroque but,

    Paul: So it’s an Amazon service? It isn’t you’ve used Amazon before, I gather.

    Peter: But yeah, no, it’s what everybody’s always wanted which is to move their data from one place to another without having to think about it and, of course, it’s never gonna work.

    Paul: Okay.

    Peter: But, you know, it’ll help, it can help, things could be less terrible. I think ultimately that’s what got me into databases was the understanding that no matter how impossible it would be to make things good, at least we could make them a little bit less bad.

    Paul: Right.

    Paul: The soul-crushing misery of writing software could at least be slightly less crushy.

    Edith: I feel like writing software is one of the greatest joys in life because you get to create something from nothing.

    Peter: So you don’t work in databases.

    Paul: I’m not sure you work in software.

    Edith: So what are the biggest changes you’ve seen in continuous delivery?

    Peter: Oh, ubiquity, certainly. It’s rare to meet someone these days that, maybe I’m just lucky enough, I live in San Francisco.

    Edith: You hang out at Heavybit.

    Peter: And I hang out at Heavybit with a bunch of other crazy people but I guess I would say growth then, right, I mean we’re seeing large enterprises practicing continuous delivery—we’re seeing small shops practicing continuous delivery, it’s not, sort of, a bleeding-edge concept that only, sort of, a small cadre of, like, sort of, avant-garde developers are doing. It’s really, I think it’s kind of the mainstream.

    Listen to full podcast here to hear Peter talk about continuous delivery in the context of organizational structure, databases and pricing.

     

  • How Mature Is Your Automation Strategy?

    How Mature Is Your Automation Strategy?

    Working in Operations can sometimes feel like walking up a down escalator. The business puts relentless pressure on your team to overcome bottlenecks, accelerate the overall delivery process and grow the volume and frequency of deployments. You’re also faced with a growing number of service level requirements and demand for higher release velocity—all stretched across different platforms and increasingly complex deployment scenarios, on premise or hybrid.

    The answer to these challenges is IT automation. Scaling your automation not only secures continuous service delivery, it also frees your resources to focus on core activities and industrializes more service management processes, such as self-service portals for infrastructure provisioning.

    However, many organizations are still on low automation maturity levels because of various barriers, including:

    • Organizational inertia
    • Lack of process capability maturity
    • Lack of the right technology expertise
    • Mistrust and fear of getting replaced

    Do you recognize some of these barriers to automate faster, better and more in your organization?

    Here’s the problem: You can determine how much and where that IT automation is needed only after you understand the current state of your automation. Are you still at the manual stage (level 0), copying new release files in the production environment, adjusting parameters and restarting the application service via a management console using manually entered parameters, for example?

    Are you automating repetitive tasks through scripts, but need to do all the preparation work to run the script, frequently modify the scripts and script parameters based on changing context, start scripts manually, monitor their execution individually and look at the outputs each time a script runs? If so, you are on level 1—and you can do much better than that.

    Are you connecting scripts into sequences and deploying new releases using a run book automation tool (level 2), where integration with other tools in the release chain is still manual and context-specific parameters have to be set individually for each execution?

    Or have you scaled the heights of process automation (level 3), where you benefit from the automated deployment of the entire process, including automated retrieval of the right parameters at the right time and in the right context triggering of automated deployments (either using commands or driving run book automation tools), confirming completion and updating records? If so, you are doing very well and can reach the last (final) maturity level by adding dynamic composition and configuration of processes based on the real-time context.

    Understanding your automation status and barriers helps you decide how and where to invest in the next automation steps. Having a baseline means you can build a business case, apply goals and targets to your organization, and measure your success as you progress.

    An Automation Maturity Assessment Service can be applied to create this baseline, identify gaps and provide recommendations, which result in lowering the manual effort and increasing your IT automation maturity and operational efficiency. Consultants initially review your current IT infrastructure and your software delivery processes and release management processes. Next, they perform discovery interviews to gather information on core processes relevant for automation. Services are subsequently rated in terms of their automation maturity and their potential for automation. The qualitative results are then correlated with operational measurements and KPIs to understand the impacts of low maturity on service levels and competitiveness.

    The assessment might find, for example, that your development team changed its deliverable for a particular application to a Docker container, which is currently not integrated in your automation release and deployment process. Your tool already supports Docker, the assessment finds, and it can be easily integrated. The benefit to do this is assessed as ‘medium,’ while the cost is ‘low.’

    Based on the results, a business case can be created or a DevOps roadmap can be crafted.

    You already know the benefits of IT automation. By allowing your automation to mature, you open the door to even greater cost savings, increased efficiency and lower risk.

  • Can We Stop Talking About DevOps?

    Can We Stop Talking About DevOps?

    I first heard the term “DevOps” from a friend in the software business when I was working at Dell HQ in Round Rock, Texas. He recommended I read “The Phoenix Project,” by Gene Kim, Kevin Behr and George Spafford. I did read it and I found it a fascinating story, but as a longtime network administrator (to oversimplify as much as possible) and having never been involved in developing software, I wasn’t quite sure how it applied to me.

    It was enough to pique my interest, though, and drove me to volunteer at a DevOpsDays event in Austin. Within weeks I left Dell and joined a small software company, where I could have a better chance of getting my arms around this slippery pig called DevOps and more hands-on practice with the related tools and techniques.

    So imagine my surprise when a year later, the same old friend who introduced me to DevOps suggested that the end goal of a DevOps transformation should be that we stop talking about DevOps. Uh-oh, I thought, he’s lost the plot. But he made his case by pointing out that adding testers to dev teams—a typical step in any Agile transformation—doesn’t result in new dev test or agile dev teams; they remain simply “dev teams.” Once a business/group has fully internalized the Agile or DevOps values and behaviors, he argued, “working in an Agile or DevOps way” becomes simply “working.”

    This is why many DevOps promoters, myself included, have a negative reaction to seeing “DevOps” in job titles, job descriptions or department names. While it simplifies the job of identifying which job candidates have a DevOps mindset or skill set, it effectively excuses everyone else in the organization from taking responsibility for change. “Doing DevOps” becomes the job of a particular individual, team or department, instead of a philosophy internalized by the organization.

    Of course, the horse is already out of the barn on that one. Each year we see an increase in the number of businesses advertising for DevOps engineers, forming DevOps departments or listing DevOps among the required skills for a position. It doesn’t matter whether we think this is a good thing or bad thing; evolution will take its course and there’s little we can do to stop it. What we can do is observe, acknowledge and adapt.

    Sure, the dangers are real. A DevOps department can easily become just a third silo or the DevOps engineer might become the scapegoat for everything that goes wrong between development and operations. But if a DevOps engineer, VP of DevOps or DevOps department can form, contribute to and/or nurture an environment of increased empathy, faster feedback and continual growth, that’s awesome; put the focus there. Don’t spin your wheels arguing that they just “don’t get it” or “they’re doing it wrong.”

    I’ve spent the past couple years researching, promoting and attempting to practice this thing we call DevOps. While it’s widely believed that DevOps is impossible to define, I am firmly in the camp of those who believe that empathy is, to quote Jeff Sussna, “the essence of DevOps.” Just as the sales team needs to have empathy for its customers, everyone in the value chain needs to have empathy for each other. I’m aware that many people find this term too “touchy-feely,” but the business benefit of more harmonious interaction between one’s teams and between the organization and its customers is clear: The faster and more effective one is at converting solutions into real value for customers, the more successful one will be in the market. Thus, being empathetic isn’t just being a good citizen, it’s also being a good businessperson.

    Taking it a step further, someone on Twitter recently suggested that “DevOps has nothing to do with technology.” The statement seemed ridiculous on its face. After all, the term “DevOps”—literally a portmanteau formed from the words “developers” and “operations”—was coined by Patrick Dubois, a technologist intent on improving the process of building and deploying software. But as I thought about it I started to understand. As the DevOps movement has evolved, the most fundamental components its promoters have identified, from Gene Kim’s “Three Ways” (systems thinking, amplify feedback loops, culture of continual experimentation and learning), to John Willis and Damon Edwards’ CAMS (culture, automation, measurement, sharing), to Dave Zweiback’s ICE (inclusivity, complex systems, empathy), are primarily about how people work. It doesn’t matter if one is building software or running a school—DevOps principles are sound.

    I recently tweeted the observation that “anyone who attempts to define DevOps in a paragraph is trying to sell you something,” adding, “not necessarily a bad thing, just buyer beware.” This wasn’t a criticism of tool providers—many tools, from build automation to log monitoring, are essential to an effective DevOps transformation. It was a reminder that the problems and solutions DevOps is concerned with are complex and nuanced. There is no prescribed tool set or checklist of actions one can take to be successful; an effective DevOps transformation requires thoughtful, committed action across the whole range of people, practices and products one uses in their work.

    It might take months or years of practice, but maybe one day we can all stop talking about DevOps.  

  • How DevOps Helps Improve Application Security

    The DevOps trend is gathering pace, evolving from a niche to a mainstream strategy. Organizations are overcoming the barriers to successful implementation and finding real benefits in terms of speed and efficiency. But there are lingering issues to solve with regard to security. As companies look to push more changes to production more quickly, risks increase from multiple areas including integrating third-party software with known vulnerabilities, creating security defects in the code, not being able to fix the security issues that do arise and poor configuration.

    According to the 2015 Cost of Data Breach Study from IBM and the Ponemon Institute, the average total cost of a data breach for participating companies increased 23 percent in the last two years to reach $3.79 million per breach. As DevOps presses ahead, security isn’t being considered often or early enough. Maybe a more “rugged” DevOps approach is needed.

    Building in vulnerabilities

    The vast majority of data breaches last year were entirely preventable. According to Verizon’s 2015 Data Breach Investigations Report, 99.9 percent of the exploited vulnerabilities were compromised more than a year after the Common Vulnerabilities and Exposures database, which lists known information security vulnerabilities and exposures, was published.

    Developers frequently integrate open source code with third-party applications that have known vulnerabilities, providing would-be attackers with a way in. It’s difficult, time-consuming and expensive to close these loopholes after the fact.

    Expand the DevOps mindset

    We see many benefits when development and operations teams work closely together, so perhaps it’s time we applied the same logic to security and risk experts. Putting them into the DevOps mix earlier in the process could lead to an increase in empathy and understanding—and an injection of expertise at the outset that could save a lot of pain later in development.

    Security and risk professionals can work directly with developers and operations to identify and remediate potential vulnerabilities. They can paint a picture of how to decrease the attack surface, set up alerts and boost response times.

    Integrate automated security into your pipeline

    If you can embed security goals into your continuous delivery processes, it should be possible to automate a lot of your security checks. Stick to the same principles as your wider development by making frequent, small, fast improvements to security. Automation can free up your security experts enough to find underlying weaknesses and tackle them.

    The average organization receives almost 17,000 malware alerts in a typical week, but only 19 percent of those are deemed reliable, according to the Ponemon Institute, in this Malware Containment Report. This means two-thirds of security staff time is wasted because of faulty intelligence, at an average annual cost of $1.27 million. Only 41 percent of organizations have automated tools to capture intelligence and assess threats, but those that do report that 60 percent of malware containment can be handled without any human intervention.

    It’s important that security professionals work directly with your developers and operations people to integrate vulnerability tests and security scans into your continuous delivery pipeline. They’re engaged in an ongoing battle with hackers, so make sure to continually review and tweak the tests that are run.

    It also helps to have versioned definitions of all your systems, so you can readily and automatically scan for vulnerable systems when new vulnerabilities are announced.

    Don’t forget to measure

    Testing the effectiveness of your security measures is vital. Run penetration tests and employ destructive testing tools to see how well your processes stand up. Make sure you have clear remediation plans and that you can deploy quickly to roll back or forward to eliminate any vulnerabilities.

    Security is too important to be an afterthought. We know there are lots of benefits to breaking down silos—let’s extend that beyond development and operations to include security. Build it into your continuous delivery pipeline and automate where possible, and you’ll get greater value from your security experts. You’ll also turn out better-quality software.

    About the Author
    cropAndrewPhillipsAndrew Phillips is vice president, DevOps Strategy for XebiaLabs, a provider of automation software for DevOps and Continuous Delivery. He sits on the management team and drives product direction, positioning and planning.

  • To Be Continuous: Empathy in Software Development

    To Be Continuous: Empathy in Software Development

    Thanks for listening to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk with Sam Stokes about empathy in software development, how mini-services are the new microservices, and constantly updating software in a constantly updating world. Sam is currently a freelance software consultant and previously was the co-founder of Rapportive.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/237727552?secret_token=s-k6OKe” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=flase” width=”100%” height=’166′ iframe=”true” /]

    Paul: Alright, Sam, so what’s your favorite thing about continuous delivery?

    Sam: I like to think about it terms of shortening feedback loops. A lot of things that startups do can basically be looked at as doing something and then waiting to get some feedback on it, and then deciding whether to do more of that thing or to do a different thing.

    And continuous delivery is, people talk about sort of reducing the batch size and also about shipping stuff faster and clearly having some kind of automation. And continuous delivery is going to help you with both of those things.

    Paul: So is this specifically on the, sort of on the journey of a small startup towards product market fit? Is that kind of the sense you’re thinking of it as?

    Sam: I see it as applying to teams of different sizes. So for sure in a small startup you’ve got you need to iterate fast, you’ve got all these product ideas. But I think even if you’re a small team in a big company as I was recently working on, and still have a lot of the same pressures you have, product managers who want to try new ideas, you have customers who are sending in feedback all the time—some of which you want to listen to, some of which you want to tune out.

    It’s sort of, in some ways, a lot like being in a startup. At least, ideally it would be. And a lot of the problems you can run into are basically about not being able to close those loops quickly enough.

    Edith: Sam you’re someone who knows a lot about loops, so now might be a good time to introduce yourself.

    Sam: Sure, my name is Sam Stokes. Until recently I was working as a staff software engineer at LinkedIn. And right now I am, I guess a freelance software consultant, thereabout.

    Paul: I feel there’s a little bit of history missing there.

    Sam: Alright. Going back in time a little bit. So I was a co-founder of a startup called Rapportive. That was a plugin for Gmail that when somebody emailed you, would show you a social profile of that person in the right-hand side next to their email.

    So that you could get some insight into who you were talking to and put some more humanity into your communications with people—that’s sort of the way we saw it.

    Paul: That’s still the only plugin to Gmail that I use.

    Edith: I use it, too, and it’s for exactly that reason, ’cause I do a lot of customer development, and it helps so much to do, to empathize. To see that this—

    Paul: Right, right, yeah is a real person.

    Edith: You know, to see their face and you feel a connection.

    Paul: It’s interesting just with kind of customer communication. One of the tools that we use is Intercom. And once I started using Intercom it was the same sort of feeling I got the first time I used Rapportive, that like I feel that there’s a human at the other end of this and I feel I know who that human is and it just changes how you communicate with the people.

    Sam: It’s really motivating to hear and to see other examples as well. We really felt like email and sort of a lot of business communication in general can be sort of cold and formulaic.

    Yeah, just bringing that human element, even just seeing a face. I haven’t used Intercom but I’m guessing they have some similar ideas.

    Paul: Similar, when you send out messages or reply to someone it does the person’s face there.

    Sam: Right, we had a sort of plugin API in the, it was a sort of private beta version of our product and one of the things people used it for, so this would let you plug into the sidebar, people would plug it into their customer database and so they could see something like, they’d get an email from a customer it shows them how long they’ve been a customer, how much they’ve spent in their store—

    Paul: Right, right.

    Sam: What version of the app they’re running, that sort of thing.

    Edith: Was your original idea to do that sort of empathy or was that something that just you iterated into?

    Sam: The empathy was where we started. There were lots of ideas about taking all the information from all these different sources, you know public sources like, truly public sources like Google, quasi public sources like a Facebook profile if you’re friends with them on Facebook, and then private sources like stuff in your company database.

    All stuff that you would be able to search for if you could be bothered to go and do the searching but doesn’t necessarily bring itself to you while you’re talking to that person. And yeah, it was very much, what should you know when you’re talking to this person in order to make them feel like they’re interacting with a real person and an empathetic one.

    Paul: So I remember when you launched Rapportive it was like, or it felt to me like it was, an instant hit and maybe that was just the overnight success that comes after years of trying. I had a doco and what was the kind of software delivery process of that?

    Sam: Hmm, that’s a great question. I guess there’s two parts to that really. So how did it get there and what was our process that allowed us to survive it? So I guess how did we get there, it was a certain amount of luck in that process as there probably always is. We built our prototype and it was a pretty minimal prototype.

    You know, it had the functionality that was like this awful demo website up. The only reason there was a website up was we were trying to raise a seed round at the time, we were applying to OIC. Y Combinator requires that you have a demo site that’s public. They don’t like it to be password-protected. And so we had this thing up where you could install it and we had like 20 users. One of our friends decided that he was going to email a friend of his at one of the blogs, I think it was ReadWriteWeb and say, you know, hey you should check this out.

    And his friend checked it out and it turned out that one of the demographics that really liked the product that we built was journalists because they get a lot of emails from people they don’t know and they want to sort of know an extra nugget. I guess that was one of our takeaways was like, if your startup is really useful for journalists, that’s a great way to get some early press.

    Edith: That’s what worked for TripIt because journalists travel a lot. So TripIt just got all this free press basically ’cause journalists used it, loved it, and blogged about it. Unfortunately, journalists do not do continuous delivery so much.

    Sam: So how do we survive that? ‘Cause what happened in that 24-hour segment was that we went from 20 users up to 20,000.

    Paul: Wow.

    Sam: So I guess one part of it was our CEO had made the original platform choice. His name’s Rahul Vohra, he has a computer science degree, he’s an awesome technologist, but his focus is not generally on writing the software. He’d been writing the prototype, and he was like, ‘I need the quickest thing that I can get to get this market, so I’ll put it on Heroku.’

    Which turned out to be a life-saving decision because when the 1,000 X traffic spike arrived we just added some more dynos. So that was how we didn’t burn down. What did we do after that? I guess in terms of software process we were pretty young as a company, so it was pretty much hack away, read all the tweets. But there was a lot of, what you might call customer development more than software development at the time. So we pretty quickly decided that we were going to try and respond to every single tweet and every single email and so on to like try and live this idea of a good customer experience.

    And I think that helped to sort of amplify the viral effect of that press spike because people were sort of retweeting the things we were saying and getting a generally good feeling about us.

    And so a lot of those tweets were like, ‘Oh I wish I did this,’ or ‘I wish I did that,’ or ‘I wish you could see Twitter profiles in there.’ So it was a little bit of sort of customer support-driven development. Which obviously doesn’t scale but it can be a pretty good way to build some goodwill early on.

    And I think a lot of our users were sort of quite passionate about using us, ’cause they felt like they got that personalized attention.

    Paul: So did the, when you scaled the dynos did it just work?

    Sam: It worked perfectly until we hit the rate limit for the API that we were getting all our data from.

    Paul: The LinkedIn API?

    Sam: At the time we were using a service called RapLeaf.

    Paul: Oh yeah, yeah, I remember them.

    Sam: It wasn’t their main product, but they provided an API which … So basically the thing that Rapportive does, which is take an email address and produce social profiles about that email address. At the time that was entirely just UI wrapped around RapLeaf. We ended up building sort of our own version of that.

    And so we’d been using their API on a sort of demo basis. Because again, 20 users. Yeah, we immediately had to learn about things like request queues, so that our app wasn’t just falling over waiting for their API, and then frantically Biz Dev-ing to try and increase our rate limit, while at the same time figuring out the best ways to get around the rate limit without upsetting the Biz Dev deal. Yeah, we didn’t really have much of a sort of continuous delivery process at the time.

    But I guess not as many people were talking about it—this was 2010. I think the first blog post had come out from Iron View and so on, but I think it wasn’t as, you know, there weren’t full startups supporting it. What we did have was the Heroku deployment model, which was basically like within five seconds of writing the code you can deploy it to production.

    Which if you squint at it, kind of is continuous delivery because you’re saying, all of the bits of your deployment that you have are automated. We didn’t really have many tests at the time, so the tests we had were automated ’cause there weren’t any. Our deployment was absolutely, you do it get push, you don’t have to like move any servers around or anything like that.

    Paul: Seems like the testing model around, or the continuous delivery model around a Gmail plugin is probably going to be a fairly difficult process anyway.

    Sam: We never really had a good answer to, you know, what happens when Gmail changes their UI. The surprising part was it didn’t happen that often. We thought it would be breaking every week and we’d need all kinds of canary accounts and end-to-end testing to monitor that kind of thing. In the end, it happened rarely enough and we noticed quickly enough that we could usually just fix it on the spot. So even when we got better at testing and that sort of cycle of things.

    Paul: We had a problem or I guess solution with GitHub. So GitHub used to be super flaky around two, two and a half years ago, when they were still on Rackspace. And whenever something would go wrong with GitHub we would find out immediately. All of our, all the developments would start failing and so before they knew that there was a problem, we knew there was a problem.

    And we would often like, declare on Twitter, oh we think GitHub’s having a bit of an issue and two minutes later their status would tweet us and then we would retweet their tweet.

    Edith: So they were using you for their free testing.

    Paul: Well, our customers at least.

    Edith: No I mean you were acting as their tester and telling them that—

    Paul: Well, I mean it was very, you know, automated on our but we didn’t have necessarily automated fixes at the start. It kind of happened every, you know, two or three months and they would swear this is the last time and—

    Edith: It’s always the last time.

    Paul: And in the end it happened like six or eight times and by that time we had built a thing that allowed, you know, certain fixes, but it wouldn’t be quite, you know, fully automated process that you would expect if it was happening all the time ’cause it always seemed like it was gonna be the last time.

    Sam: And it’s probably always happening for a slightly different reason or in a slightly different way. It’s actually, it’s an interesting question to me in the subject of continuous delivery in general. People always talk about every time you commit you run your tests, every time you commit, you deploy.

    But that’s actually only one of the kinds of events that you’re dealing with in a software life cycle, right, when something’s actually running. I mean, sure you’re changing the code but also the world’s changing around you. And I feel like you always end up needing a little bit of both—you need something that’s gonna trigger when you change it, but you also need sort of an eye on everything you depend on. And, of course, even figuring out what all those things are can be interesting.

    Paul:

    So I wrote a blog post about this on the Heavybit Blog, I think a while back, and it was about third-party APIs you know, how do you test third-party APIs. And the conclusion that I came to was that, so if you run the tests as part of your test suite, then if something breaks, your test suite will be broken, but also some things will just like randomly break in production.

    And then of course you can’t deploy because your test suite is broken. So you’re in kind of the worst of both worlds. And so what I considered is, is that you need something which, you know, is just permanently monitoring. Like whatever API things you do constantly hits your personal accounts or your test accounts for those things, and then stub it out in your test suite.

    And that way you kind of get that best of both worlds, your test suite doesn’t fall over and you get an actual monitoring event when the third-party service goes down in production.

    Sam: I can tie this to an experience from LinkedIn, actually. So LinkedIn has the model where there’s a staging environment and a production environment. And the staging environment is—I don’t know if it’s technically called a staging environment because it’s not exactly production-like, but what they actually call it is early integration. Because it’s the environment in which teams integrate with other teams.

    And so the idea is you deploy your stuff into EI, early integration, as soon as you can so that if any other team depends on you or vice versa, you can find out there before the thing hits prod. But what you’ve effectively got in EI is this constantly changing environment full of dependencies and you have exactly this problem and different teams which using different solutions to, well, let’s write some integration tests which actually hit the dependencies and then we’ll run them in our test suite when we deploy to EI or when in the build step which—

    Paul: I can’t tell as you’re describing this whether this is a good thing or bad thing.

    Edith: Yeah.

    Sam: I think there wasn’t a clear conclusion, but my conclusion is that it doesn’t work for the same reason that you’re saying.

    You need to know that if your test suite fails, it fails because of something you did. And then you need something else that can fail because of something someone else did.

    But if you’re ever unsure which of things it is, then the signal is lost for both systems.

    Be sure to listen to the full episode. While you’re there check out Heavybit’s library, home to great educational talks from other developer company founders and industry leaders.

  • Self Forgiveness

    Charlie Breaks the Build
    “How to apologize when you have broken the nightly build.” — http://bit.ly/1REOAOz

    — ROELBOB

  • Chronos, Kairos, & Janus: It’s Time to Seek Balance

    Chronos, Kairos, & Janus: It’s Time to Seek Balance

    We conflate idealism with naïveté and pessimism with ‘just being realistic’. But to deny the existence of perfection is to deny the evidence of our own lives.

    Was 2015 a win or a loss?

    It can be tough calling the winners and losers. The mathematics department I studied in had a habit of calling out the As in the class or saying nothing at all. One professor considered us lucky as his alma mater called out the Fs when handing back papers. It’s interesting how information presented from different perspectives can have such a dramatic impact on the range of emotions it invokes.

    A postmortem is an activity that focuses on the time line leading up to a failure. Etsy has done an excellent job of encouraging blameless postmortems in industry. It’s a matter of when things are going to fail, not if failure will occur, when we’re dealing with complex systems.

    Retrospectives are a similar meeting held at the end of a development sprint where team members try to identify actions they can take to have more successful sprints in the future. Looking back on 2015 I see the need to balance our successes with our failures. When we’re focused on optimization we’re too quick to celebrate our successes. When we’re focused on stability we tend to only see the failures and ignore the wins as we come across them. When we’re well balanced it can be hard to tell the difference between our postmortems and our retrospectives, but we definitely have both.

    No matter how you feel about it 2015 is quickly slipping through our fingers. Chronos & Kairos represent time as we typically think of it, mechanically, and the opportunistic time to take action, respectively. I happen to be a fan of Janus, the god of all beginnings and endings, as he was said to exist through all space and time simultaneously. While I do not have that type of experience or foresight I am capable of looking at this past year as I anticipate the next and try to adjust.

    I can remember students that had trouble making passing grades that would be excited about Cs. I can remember other students that would be disappointed if they made a B and utterly devastated if they failed a test. Odds are you’ve been both of these students as our performance is just as dynamic as the courses we’re faced with over time. While we celebrate As it’s really a sign that a student isn’t being challenged. An F is a sign that they need to slow down or maybe back up a bit, and C students represent those that are being challenged by the curriculum and are successfully making forward progress.

    Cultural Balance

    In early November I gave a talk called “Precision, Accuracy, and Debugging the Bimodal Model” at the devOps Summit during Cloud Expo in Santa Clara. My focus was on the granularity in which we measure things and our tendency to get caught up in sorting things as right or wrong, good or bad, new or old, and into other extremes instead of simply acknowledging them as being different. While boolean categorization can be accurate in some contexts there’s often circumstances that require the precision provided by defining a spectrum between two extremes. These intermediate states often increase our odds of implementing successful change as they provide gradual steps to follow instead of expecting a single instantaneous jump to occur.

    When undeniable failure occurs we find where to learn by following a time line of small successes. At some point our expectations will not accurately line up with the reality we’re faced with and this is where we need to take a closer look.

    I was able to catch Devops Days Silicon Valley while I was in Santa Clara. The conversations I had where intended to gather more feedback around culture hacking and how culture influences the artifacts an organization produces. The theme was “bridging devOps cultures”, which made it easy to get feedback on some models I’ve been working on as it was a general topic of conversation at the event.

    Matter of fact, concerns that cultural change is ultimately what is needed for sincere advancement echoed throughout various sectors of the science and technology industries I found myself in this year. Devops Days Silicon Valley made it a theme, but cultural change seemed to be the hallway track topic at most of 2015’s events. Dave Snowden‘s cynefin framework was also present at most events I attended this year. I was introduced to the model during Jabe Bloom’s talk in NYC. Both J. Paul Reed and myself mentioned it at devOps Summit, but Jason Hand went into it in more detail with his talk, “Flaming Poo and the Human Response” at DevOps Days Silicon Valley.

    Lean Agile Scotland had a lot of discussion around culture and cynefin as well. It was even in the carpet. This worked for me since I made the trip in an effort to contrast the cultural differences in tech communities across US cities I had visited with an international group. Outside of the presentations culture was a common topic among corridor conversations. Lean Agile Scotland had folks from all over the world in attendance. It’s often the little differences, like calling it a corridor instead of a hallway, that we tend to find amusing.

    Conway’s Law links the artifacts an organization produces to its communication structure, which ultimately is an artifact of the organization’s culture itself. I’m starting to look at culture as the context of an organization with respect to how it communicates. Jabe Bloom and Chris Matts both gave presentations on culture in Edinburgh but the videos have not been posted yet. Here’s a picture of Jabe providing a definition of culture during Chris’s keynote.

    culture

    I was really delighted with Chris’s keynote. It was well rehearsed and entertaining but it also provided a missing piece to a vector based model I’ve been developing in an effort to more consistently benchmark organizations. I had hit a wall as I couldn’t see how I could quantify something like culture without injecting my personal bias into the metric, but what Chris shared about Hofstede’s Cultural Dimensions easily drops into my model and satisfies the requirement.

    I’ve received a lot of feedback on my model between Pittsburgh, Edinburgh, and Silicon Valley and it’s all been very encouraging. I’ve seen initial doubt turn to optimism and have had lots of people offer to help me extend my efforts and research this further, which I intend to be a focus for me in 2016. I do look forward to benchmarking organizations next year in an effort to find more paths between safe game states in industry.

    Looking back at the communication I described earlier with professors handing back graded tests, it is clear to see that how we learn is influenced by the emotions that are emphasized when we communicate. Are we focused on failure or are we focused on success? Does the culture we’re in value stability or optimization more? Does your team have postmortems, retrospectives, or both?

    Katherine Kirk advocated “Why Team Happiness Can be the Worst Thing To Aim For”. Insight is enveloped in emotion and equanimity is the tool we need to access it. As emotions are fleeting we should not pursue them. Instead, by anchoring ourselves in equanimity we are prepared for the waves of different emotions that will come and allow them to pass without judgement.

    Richard Sheridan, author of Joy Inc and CEO of Menlo Innovations, offered a different perspective with his keynote on how to “Build A Workplace People Love – Just add Joy“. He describes an open, high trust community that’s centered on learning. This means an environment that’s comfortable with failure when it happens and embraces the opportunity to learn from it. To pursue an emotion may be folly, but when culture is so harbored on stability that the failure ultimately manifests in the human components of the system it is only natural to drastically course correct towards different bearings.

    To Err is Human

    Jonathan Disher‘s talk, “DevOps Army of One” at devopsdays Silicon Valley was an excellent blend of technical and cultural information. We automate in an effort to gain some room for balance but that space can quickly be filled with other work instead.

    Asking for certain kinds of help, like more team members to offset workload, can bring leadership to question an individual’s level of commitment. Fear of retribution may discourage us from taking the personal time we need even if we don’t have adequate depth on the team, but eventually something will give. This is why burn out rate is high in our industry.

    The intention of mapping is to clearly challenge our assumptions. This helps us to realize we’re off course as soon as we checking our bearings. Wardley Mapping is a value chain mapping approach developed by Simon Wardley, appropriately enough. His mapping is quite practical and provides an interesting contrast to Edward Tufte‘s techniques, another data visualization and mapping expert I was pleased to meet this year.

    It is unfortunate that the time needed to give the level of attention required for this type of insight is often a considered a luxury in corporate cultures. Thanks to automation and adaptive visualization frameworks the standards are slowly rising. Will Evans‘s presentation from Lean Agile Scotland provides a solid blend of Wardley Mapping and design as he explains how new practices like these enter our workplaces.

    A Culture of Learning

    As 2015 enters its final moments I feel successful in finding consensus that the cause of our failure is a cultural one. I believe we cannot improve what we do not measure, and therefore my plans for 2016 are to further study, model, and visualize organizations and their equilibrium states.

    I cannot help but form a hypothesis on how we fix the problem before I’ve gained the context to make a reasonable attempt, but mapping is intended to help us challenge our assumptions. I’m assuming the solution lies in our actions as individuals and we must want to improve ourselves if we expect to see improvement around us.

    Often the final state we are striving for is literally not achievable. John Willis introduced me to this concept of “True North” from Toyota Kata, by Mike Rother, and suggested I read it myself to learn more. We’re currently reading Toyota Kata in a devOps book club we started, now that we’ve finished reading Designing Delivery by Jeff Sussna.

    The book club is intended to help us discuss ideas as we move through a book. I’ve been surprised to find how asynchronously a book club can operate. Everyone reading the same book at the same pace is a more typical book club approach, but given we live in different time zones and keep busy schedules we’re adapting to a model that’s a better fit. If you’ve read Designing Delivery, Toyota Kata, or plan to read either book and would like someone to discuss them with feel free to ping me on twitter.

    The improvement kata from Rother’s book maps the scientific method into Plan, Do, Check, and Act. Instead of focusing on processes we should focus on enabling people to improve themselves. We improve when we learn and we learn when we communicate. If your culture does not foster communication can if foster learning? If you’re not open and honest with yourself, how does that effect your environment?

    I could go into more detail about my intentions for next year and may do so at a later date, but my plans are of no use to you as we are different people. I would encourage you to gain awareness of your own situation by challenging the assumptions that have bounded your operation over the past year. You have had successes and failures and both deserve attention. Reviewing your outcomes with equanimity will help balance your actions, and hopefully lead you into a successful 2016. Happy New Year.