devopsy.kinsta.cloud

Category: Blogs

  • Puppet Labs Launches Annual State of DevOps Survey

    Puppet Labs Launches Annual State of DevOps Survey

    Puppet Labs has announced its 5th annual State of DevOps Survey. In the past five years, we’ve surveyed more than 20,000 technical professionals around the world, spanning all industries and companies of all sizes. We’ve captured a tremendous amount of data about how IT performance, culture, technical practices and organizational performance intersect.

    What sets our survey apart from the many other surveys out there is that it’s science-based. Yes, there is a science to creating surveys and analyzing data, and there’s also an art to translating all of that amazing data into something that’s useful to the industry at large.

    This year, we’ve partnered with DevOps Research and Assessment (DORA), founded by our friends Gene Kim, Jez Humble and Nicole Forsgren. If you’ve been following the DevOps conversation for a while, their names should be familiar. Together, our goal is to help advance the DevOps community and help technology organizations improve their practices through rigorous scientific research.

    Through our analysis, we’ve been able to quantify IT performance. We know that high-performing IT teams deploy code 30 times more frequently, with 60 times fewer failures, and they recover 168 times faster. We also know the factors that predict IT performance—technical practices such as continuous delivery, cultural norms and lean management—which in turn predict organizational performance.

    This year’s survey seeks to dive deeper into each of these areas, while also shedding light on new areas of research, including:

    • ROI of IT performance. What can you expect from your investment in DevOps? Our goal is to understand how throughput, stability and quality affect the bottom line.

    • Security and compliance. What role do security and compliance play in a DevOps environment? Some people see security as a roadblock to DevOps, but we want to investigate if it has to be that way.

    • Quality. So you’re deploying code faster than ever. What’s the quality of that code?

    • Containers. How do containers affect IT performance? Are they making IT’s job easier?

    • Lean management. Last year we found that lean management practices such as limiting work in process and creating and maintaining visual displays showing key quality and productivity metrics predict IT performance. This year’s survey looks at additional lean management practices to help us better understand what it really means for a technology organization to go lean.

    The survey seeks answers to questions we get all the time and we believe they reflect the broader concerns of the DevOps community. Your experience matters and we hope you’ll share it by taking the 2016 DevOps Survey. It should only take about 20 minutes, and when you complete the survey, you’ll be entered to win some great prizes (see the official rules), including :

    Please take the survey and share it with your friends and peers. Together, we can make IT better for everyone.

    About the Author/Alanna Brown

    unnamedAlanna Brown heads up DevOps research and go-to-marketing strategies at Puppet Labs. She also works closely with Puppet Labs product owners and engineers to develop a roadmap and support for Puppet on Windows.

  • The DevOps Equation: Agility + Empathy = Quality

    The DevOps Equation: Agility + Empathy = Quality

    I had the chance to catch up with Jeff Sussna (@JeffSussna), founder and principal of Ingineering.IT, ahead of his keynote address on continuous design, scheduled for DevOps Days Atlanta, April 26-27.  He discussed the importance of designing for service, responding to the unexpected, and the importance of building empathy across teams.  

    Derek Weeks (@weekstweets): Today I’m really happy that we have Jeff Sussna joining us for the latest in this series. Jeff, why don’t you introduce yourself?

    Sussna: Sure. Thanks for having me. I’m an independent consultant. I’ve been around, since rocks were young or the 80s, whichever is older. My particular background is that I have built systems and led organizations across the entire development, QA and operations spectrum. So, I’m a little unique in that respect.

    My current focus is on helping IT organizations and digital businesses really bring together their ops and design thinking so they can not only go faster and not only be more resilient, but provide more customer value at the same time.

    Weeks: Let’s talk about that a little. How is continuous design—a subject you have written a lot about—coming into play? How can it help us be more resilient.

    Sussna: Just to explain a little about what we mean by continuous design. It’s basically bringing together the entire organization and a unifying design in operations. It’s taking what we are doing in DevOps and expanding it to include product, design and support. It lets the whole organization really have an agile conversation with a market and with its customers.

    The issue that we face is that we live in a service world now. The point is no longer delivering products like cars coming off an assembly line. It’s much more about responding to our customers and having relationships with them. This is happening in a world that is more and more complex while less and less is in your control. We can’t figure everything out in advance. We have to have the ability to respond to the unexpected. We have to be able to do that at every level.

    “This is happening in a world that is more and more complex, while less and less is in your control.”

    I will give you a couple of examples of things that are perfect illustrations of this:

    I recently wished a friend of mine happy birthday. The reason I did that was that because there was a notice on my calendar that said it was her birthday. It was wrong. It wasn’t her birthday. I had no idea where that notice came from. Did it come from Skype? Did it come from LinkedIn? Did it come from email? I had no idea.

    We face that in IT, too. Systems are becoming more complex. We develop microservices. We use public and private clouds at multiple levels. We’re no longer in a situation where we can control things. We have to be able to respond and respond at all levels.

    When we make design and operations unified and continuous, we are able to say, “Where did we think we were going? Where would we need to go? and what do we do next?”

    What I talk about in my book, in my workshops, and all my work is that this needs to happen at every level. It’s not just a matter of designing for failure inside the data center or inside the infrastructure. We also have to do it at other levels.

    Here is another example:

    I was trying to get paid recently by a large enterprise client in Europe. They were trying to wire money internationally and my bank wasn’t really set up for it and they were having problems. They just kind of gave up. I kept getting these messages saying, “Sorry, we tried paying you and it didn’t work.” That was the end of it.

    I started to get frustrated and my client within the enterprise started to get frustrated to the point where they sent a scathing email to our AP department which they CC’d me on, saying, “Why can’t we take the lead to get his guy paid?”

    That’s an approach that is based on resilience in the sense of assuming that something can go wrong and figuring out what he can do to fix it when it can go wrong. As opposed to the traditional approach where we’ve been trained to push certain buttons and take certain actions. If those actions don’t work the way they are supposed to, well, I guess that is someone else’s problem.

     

    https://www.youtube.com/watch?v=ISSMHmwIMtA

     

    Weeks: This brings to light some recent conversations I was having with Paula Thrasher at CSRA. She was talking about developing systems and how to make them more resilient. She is using pictures as part of her conversations with people to help visual their work. What work steps are there? What work is flowing? What is communicating with what?

    Are you taking similar approaches with continuous design in how you work with clients to pursue it?

    Sussna: Well, I try to introduce them to the principles and I also try to introduce them to a set of practices which actually comes out of Mark Burgess‘s work on promise theory. It is really a conceptual model for how we think about success and failure and how do we think about service. For example, Amazon helps you get everything you need in your life from shaving cream to batteries to snow tires to books as quickly and easily as possible. In order to keep that promise they make a whole variety of other promises. Some of which filter all the way down to a promise to be able to go over and make a change to production as quickly as possible.

    Once you take that mindset. The interesting thing about a promise and the reason we use that word is because it is very intuitive. We all know what a promise is. We all make promises to each other every day. On the other hand, when we think about it in terms of service, IT, or whatever it may seem a little odd. The thing about promises is that on the one hand we are making commitments to do things and on the other hand promises sometimes get broken. That is actually good because it forces us to think. If I promised to pay my vendor. What do i do if I’m not succeeding in paying my vendor?

    The thing about promises is that on the one hand we are making commitments to do things and on the other hand promises sometimes get broken. That is actually good because it forces us to think.

    It’s sort of flipping our mind. Again, we are doing this in DevOps world where you start thinking about things like mean time to repair (MTTR) instead of mean time between failure (MTBF). We shift from how do we create a situation to how do we create a situation from where nothing will ever break—which becomes less and less possible. When then need to shift to a situation where we can respond more and more quickly.

    We go to executives and say, “We don’t want to worry about how often stuff breaks anymore,” and they freak out. We have to teach them and we have to teach ourselves that we can’t prevent things from breaking. Instead, let’s be able to fix them so quickly that maybe no one notices. My focus is on really helping people shift their mindset from thinking about ourselves to thinking about who we are helping. And from thinking about what we are doing to thinking about how well it succeeds and how well it overcomes problems.

    We have to teach them and we have to teach ourselves that we can’t prevent things from breaking. Instead let’s be able to fix them so quickly that maybe no one notices.

    Weeks: You talked about delivering on promises as quickly as possible. How does continuous design account for doing things as quickly as possible but making sure the quality is there at the same time?

    Sussna: In two ways—one way is that when you do things in small increments you are able to find problems more quickly. This is classic; it comes out of lean. It comes out of integration. It’s pretty basic stuff for our community now that it’s much easier to find and fix a bug five minutes after you change a code than five weeks after you change the code.

    When you put things like Linux, agile, DevOps, and design all together, they all focus on incremental feedback loops where you discover the gap between what you think you’re accomplishing and what you’re actually accomplishing very quickly. Secondly, when you take this all to the way to the marketplace and you make it part of the relationship with your customer, you can reduce a tremendous amount of waste.

    The way to think about continuous design is: it’s all about steering. Think about a boat and you’re trying to sail across a bay. You can’t just point the boat at the other side and say go because the winds are changing and you’re going to be able to respond to things that you can’t predict. If you sail for 20 minutes then discover you had pointed in the wrong direction, you’re going to be way off-course. You’re then going to generate a tremendous amount of energy getting back on course.

    What continuous design says is just an extension of continuous integration and continuous delivery taken all the way though the business. When we steer continuously we steer efficiently.

    What continuous design is really just an extension of continuous integration and continuous delivery taken all the way though the business. When we steer continuously we steer efficiently.

    Weeks: I know you’re going to be keynoting DevOps Days Atlanta coming up in April. Chris Corriere, one of the guys on the organizing committee, tells me that you’re leading a full day workshop on continuous design before that. Maybe you could tell some of the people that are attending DevOps Days Atlanta why they should be there a day earlier for your workshop and what things you’re going to address during that time.

    Sussna: To some degree what I am going to address is everything we have been talking about for the last 10 minutes or so—relative to DevOps. I think the key thing is, and we need to think about it. We need to think about more than just going fast. One of the big misunderstandings that I see with DevOps is this notion that it is primarily about delivering application changes to production faster. That’s only half of it. We really need to think about from the perspective of providing service.

     One of the big misunderstandings that I see with DevOps is this notion that it is primarily about delivering application changes to production faster. That’s only half of it. We really need to think about from the perspective of providing service.

    You know what we are talking about is essentially developers and operations people providing better service to each other. As a result, they can deliver better service to the business.

    Where we talk about microservices. You know that word service is in there for a reason. If all you do is you kind of shatter your organization and shatter your architecture into a bunch of little pieces, you still have to figure out how those pieces come back together into something coherent. The answer to that is taking a really deep service-oriented approach. That is very much what my workshop is about. What does that word even mean and how do we do it?

    Weeks: You mentioned thinking about interactions as delivering better service to one another within the organization. How does empathy come into your thinking around DevOps and Continuous Design?

    Sussna: I’d like to say agility plus empathy equals quality. What I mean by that, in particularly empathy, I think it’s important to take a minute and clarify. When we talk about empathy we are not talking about wallowing in someone else’s pain. We’re not even necessarily talking about things on an emotion or feeling level. It’s very simply the ability to see things as if from an other’s perspective. I will give you a prime example of where that comes into play and how it’s important and how it helps.

    Agility + Empathy = Quality

    When I work with clients, I tend to look at their process using their organization from the product beginning to the support end. Often times I see support getting the short end of the stick. I see places where they get phone calls from customers with features that they don’t even know exist yet. They sometimes get talked down to by development or product or even operations. They say things like, “Well you’re just supposed to support the stuff. You’re the young people. You’re the less experienced people.”

    When we have the ability to see things from their perspective, we can think about how support is actually experiencing our delivery process. If we don’t think about them when we go through the process of delivering new features, they will be lost. Them being lost will show when customers call for help. That will significantly affect customer satisfaction and then even beyond.

    The reason we need to empathize with support is so we can empathize with the customer. Often times when we build things and we don’t think about the notion that our customers might have trouble with it. On the one hand we have the illusion that if we design it and we build it really well, they won’t have trouble. We can’t perfectly predict who they are, what they need or what they will encounter. There’s an inevitability to the idea that they are going to need some amount of help.

    Weeks: I think that is probably the best description of empathy within a DevOps environment that I have heard anyone talk about. Jeff it has been a great conversation. Thank you very much. I really appreciate it and we look forward to seeing you at DevOps Days Atlanta coming up.

    Sussna: Thanks. It has been great talking to you and I am really looking forward to the conference. It will be fun.

    If you want to learn more about continuous design from Jeff, be sure to join us at DevOps Days Atlanta. I will also be at the event giving an Ignite Talk, titled, “Real Heroes Draw Pictures,” focusing on lessons learned from the 31 DevOps and continuous delivery reference architectures I assembled.

  • IT DevOps: One Size Does Not Fit All

    IT DevOps: One Size Does Not Fit All

    As absurd as this may seem, some in IT may be longing for the good old days of the mainframe. After all, the hulking machine, with its dumb terminals and thousands of punch cards, gave IT complete control. No citizen developers or business analysts allowed.

    Things got more complicated with distributed desktop computing and its different operating systems and requirements. Server-based web apps simplified things once again, but now mobile has made development more complex than ever. Instead of just updating the server, you also now have to update each mobile device—creating apps that work on each and getting changes out to the entire mobile population.

    What’s IT to do? Applied properly, DevOps can help.

    The Evolution of DevOps

    Coming on the heels of agile development—a term first coined in 2001—is DevOps, which was first popularized in Belgium in 2009. Today, 15 years into agile development and six years into DevOps, it is becoming clear that these intertwined terms are quite broad, encompassing attitude and approach, platforms, tools and solutions. Agile development, with its Scrum meetings and quick turnaround cycles, did speed development but made deployment more difficult, because many organizations just couldn’t support fast cycle deployment.

    DevOps, with its platforms, tools and practices emphasizing collaboration between IT and development, targets application delivery with continuous testing, ongoing feature development and ongoing maintenance to deliver apps faster. While the intention is well and good, the DevOps process must be task-specific to succeed.

    Today’s development environments can be broken down into three categories: low-code, deployment-centric apps and, in a class by itself, mobile. Delivering apps in these three environments involves a broad range of skills and tools. Therefore, the approach and degree to which DevOps is utilized must vary as well—from virtually no DevOps in low-code environments to extended DevOps in mobile.

    Low-Code = Zero Ops

    In cloud-based, application performance-as-a-service (aPaaS), low-code environments, DevOps can take a back seat. With the vendor managing, maintaining and controlling the deployment environment, the citizen developer or business analyst is empowered to work on GUI development and other features and functions with little involvement from IT or development. In these environments, power users can easily push out low-code apps. A strong DevOps presence here is not necessary because the separate environments typically required to propagate apps does not exist. The business analyst simply makes the changes and pushes them through to production.

    Here, IT is best utilized as gatekeepers, getting involved only when needed. We characterize these environments as zero ops—the less DevOps the better. This leaves business users, IT and development free to focus on their strategic areas and core competencies, funneling resources appropriately and enhancing productivity across the board.

    Deployment-Centric = Controlled Ops

    High-transaction, highly scalable, business-critical global apps fall into the deployment-centric category. The goal of DevOps in these code-intensive environments is to free developers from deployment issues, enabling them to focus on their key competency of writing code. Well-executed DevOps can effectively move apps from development to deployment, enabling tight collaboration between development and IT. In this environment, a well-managed DevOps process can make it easy for the DevOps team to deploy the apps and to manage the deployment moving forward, scaling apps automatically with rules that scale automatically based on usage.

    That is why we characterize DevOps in these environments as controlled ops, a collaboration tightly controlled by development and IT.

    Mobile = Extended Ops

    From development to deployment, maintenance, monitoring and analysis, mobile has introduced new levels of complexity to application delivery. Developers have iOS, Windows and Android to deal with, not to mention multiple versions within those environments. Then there is the development environment itself—web, native or hybrid. Compounding it further are the devices—countless phones and tablets. And deployment options—public app stores, private app stores, file-sharing services. Finally, the end user is engaging the device in ways never before imagined, placing the need to build responsive and compelling user experiences front and center.

    This environment calls for extended ops—a highly collaborative environment where development, deployment, maintenance, support and business stakeholders are all integral to an app’s success. In mobile, everyone needs to have a place at the DevOps table, as mobile apps are literally iterative apps—constantly on the move, morphing and changing, driven primarily by end-user feedback. The ability to make seamless updates and user-driven changes at record speed can spell an app’s success or failure. The collaboration, platforms and tools DevOps offers in this environment have never been so critical, as business users now must be involved in the design and development of the app and in analyzing how the app is used, which will dictate changes moving forward. Successful, extended ops welcomes business users, but does not involve them in the DevOps process itself—that’s for IT and development.

    In Summary

    DevOps can have an impact on the entire application delivery cycle. From improving deployment frequency—which can lead to faster time-to-market—to shortening lead times for new releases with lower failure rates, the goal of DevOps is to maximize the predictability, efficiency, security and maintainability of the simplest mobile app to the most complex transactional system. The key to successful DevOps is applying the right level of collaboration and employing the right tools to ensure that DevOps improves quality and productivity rather than hinders it.

    About the Author/Mark Troester

    image001Mark Troester is the Vice President of Solutions Marketing at Progress Software. Mark has extensive experience in bringing application development and data integration products to market. Mark previously led product marketing efforts at Sonatype, SAS and Progress DataDirect. Before moving into marketing, Mark worked as a developer and developer manager for start-ups and enterprises alike.
    LinkedIn Google+

  • To Be Continuous Podcast: The Clash in Open Source

    To Be Continuous Podcast: The Clash in Open Source

    Welcome to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this episode, Edith and Paul talk about the clash in OSS between developers looking to get paid and software companies looking for things for free.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/244279037?secret_token=s-oeBCP” params=”auto_play=false&hide_related=true&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=’166′ iframe=”true” /]

    Edith: So Paul, what do you like about open source?

    Paul: So I guess that depends on whether we’re talking about writing open source or using open sources as part of a company. Writing open source, I like just having all my code out there.

    And I’ll do that as often as possible. Anytime I start a little bit of code, a little library, I’m looking to open-source it and have a readme and have it out there for other people to use. I had a very long discussion, which I eventually won, about the …

    Edith: I like the way you think of all discussions as win or loss.

    Paul: Well, yeah, yeah. It was more on the argumentative side of discussion. But with the other co-authors of the PHP compiler I wrote, which was originally GPL. And I argued very strongly that I actually don’t care if people give back, I just want people using my software. So to me, open source is a way of having people using my software.

    Edith: That’s really interesting. So, I liked what you said at the beginning about, like, you liked open source ’cause it meant getting stuff out sooner. ‘Cause I’m a big fan of that.

    Like, I always say you should just publish stuff. Like, my co-founder John said something really good at our team meeting last week. He’s like, if it’s 90 percent done and you don’t publish it, it’s actually 0 percent done.

    Paul: It’s 0 percent done, yeah.

    Edith: Yeah, and that’s a hard lesson, ’cause you always try for perfection lately.

    Paul: Right, right, right. Well, I think there’s also some really hard downsides about that. So when you publish something before it’s ready, before you really —

    Edith: Well, ready.

    Paul: Well, there’s ready ready, but let’s say ready is, let’s say you publish it before you clarify what the software is to do, or the values or the mission of the software.

    And then you get contributors who are saying, ‘Oh, this is great, but I really want it to do this other thing,’ and you’re like, ‘Well, that’s not exactly what I was going for here.’

    Edith: Well, I’d say most open source projects are so lucky as to get contributors at all.

    Paul: True, true, true. So I’ll give you an example. I wrote the V8 bindings for Closure. So a binding that allows you to write JavaScript and run it in Closure. And I did that for a purely selfish thing, for another, for a nascent pipeline that I was building called Stefan.

    And then people came along and they wanted to do other things with it. They’re like, ‘Okay, let’s add a binding for this other thing.’ I was like—

    Edith: Well, that’s kind of like a pivot.

    Paul: It is, it is, but I really was not interested in it.

    I’d rather someone took it over, took over the project and actually ran with it where they wanted to go, and I can stick on the old version, or I can see if the new version solves it for me, but I wasn’t interested in becoming a maintainer in this thing, I was interested in solving my itch.

    And I just happened to open-source it, because I felt it might be useful for other people.

    Edith: So Paul, I hate to say this, but I agree with you.

    Paul: No, not again.

    Edith: Not again.  No, I mean, I think what you said is exactly right, that you said, I did this for this reason, and I want somebody else to take it over now.

    I think where open source runs into issues where people are like, it’s only this purpose and you can’t do anything with it besides my original thing.

    Paul: So you think that’s a good thing, or you think that’s a problem? Or it causes problems in the community?

    Edith: I think open source is so misused for so many different purposes that it’s hard to even say. Like, so, in Europe, people say open source is good because it means free software.

    Paul: Libre or gratuit?

    Edith: I’m sorry?

    Paul: Free as in beer or free as in, what’s the other side of that?

    Edith: There’s no such thing as a free lunch, is that what you’re talking about?

    Paul: No no no, like people say open source software, it’s like free-as-in-beer software, as in software that has no cost vs. software that has no price tag vs. software that is, they use the French word libre for—

    Edith: Oh yeah, that’s what I was talking about. So I remember I went to Europe in, I think it was like 2004 or 2005, and they just had this big push for open source, ’cause they thought it meant free.

    Paul: Oh, this is like the German government and that sort of thing.

    Edith:  So literally like, so I went to this customer and they’re like, ‘So it doesn’t cost you any more to give us an extra copy.’ So why shouldn’t just everything be free?

    Paul: Oh, right, yeah, well.

    Edith: And so I think open source has been misused by many businesses who think of it as just like, hey, we get all these developers working for free.

    Paul: So I think that the main advantage to business of open source is not that it’s free, but it’s that it’s low-risk.

    Edith: Low-risk?

    Paul: Lower risk. So it removes—

    Edith: Ah, I would argue—

    Paul: Well, so, what I mean is that it removes a certain element of risk from it. So there’s, Joss Belsky wrote a blog post about how you shouldn’t use, I think it was a database that you didn’t have the source code to.

    Edith: Well, so I’ll argue the reverse, and I think open source is sometimes more risky. ‘Cause I talk to people now who are like, nobody’s maintained this open source project that I depend on.

    Paul: Right, right, right. But that vs. a closed-source version.

    Edith: Well, so the opposite risk is that you end up that you are responsible for this open source project, and you’re like, well, actually, I just wanted to use this thing. I didn’t want to do the maintaining.

    Paul: So absolutely, you don’t wanna end up the only person that, well, it’s actually fine to end up the only person which is using this thing.

    Edith: Well, no, it’s not—

    Paul: It’s not ideal.

    Edith: It’s that many people are using it, but nobody is paying money, and then you’re stuck.

    Paul: Because no one is maintaining it.

    Edith: No one is maintaining it, and you depend on it.

    Paul: Right, so, but imagine that no one maintained it and all that you could get was like a binary block. That’s the real problem. The company goes out of business. You have random database version seven, the binary, that only runs on 32-bit, that you’re maintaining, like, companies are literally maintaining COBOL—

    Fortran things from 30 years ago. It sits in a machine, you can’t touch it, you can’t fix it, you can’t apply security updates to it.

    Edith: Yeah, that’s fair. So Tim Chou, who is, he’s an awesome guy, he was early Oracle, he said that people don’t buy, when they buy software, they’re not buying it what it is today.

    Paul: Right.

    Edith: ‘Cause basically, anything that you’re buying today, you could just hire 20 engineers that could build it. What they’re buying—

    Paul: Yeah, you could just copy it.

    Edith: Yeah, what they’re buying is the promise of stability, that it will continue to be maintained. That there is a vision. And that’s actually, I went over to a big customer who won’t let us use their name yet, but they said they love LaunchDarkly because we thought every day about feature flags.

    Paul: Right, right, exactly. This is why people buy SAS, because, because people think all day about CI, about QA as a service, about queues, or whatever it is from various Heavybit companies that I just shilled for, they’re, you can probably figure out yourself which ones they are by looking at the homepage.

    Edith: Well, and Paul, by the way, thank you very much for wearing your LaunchDarkly t-shirt today. It looks very handsome.

    Paul: The, I definitely agree that the benefit of the thing is that you use it every day. But you can do that with open source as well. You can have open source software that, so, for example, our front end is open-sourced.

    Edith: Oh, I’m absolutely a fan of open source. Like, we use open source every day. Our client libraries are open source. I’m not saying that it’s, I don’t think it’s a religious thing vs. open source vs. not, I just think that many times people assume open source is something that it’s not.

    Paul: Yeah, fair enough, fair enough.

    Edith: I mean that if you have open source, that just means that developers spring out of the ground willing and happy to update your software every day.

    Paul: A mixed blessing, definitely. If it even happens.

    Edith: Like, I remember, so I remember when Sun, so I used to work on a, I used to be at Epicentric, and we had a portal server, and we were killing Sun. Absolutely killing Sun. And this was in 2003, so what they said is, ‘Oh, we’re gonna make our portal server open source.’

    Paul: Right.

    Edith: And so somehow this whole thing, we’re like, everything will be open source, and like, developers will spring out of the ground, and it’s like, well, no, developers, they don’t wake up every day like, ‘I’m gonna work on your thing.’

    Paul: Right, right. They will if it’s sexy or exciting. There’s lots of developers who are showing up at Mozilla just to contribute. And in fact, they’ve fought through incredibly large hurdles to contribute to Mozilla’s software.

    Edith: Well, that’s fascinating, ’cause I see so often the reverse, where people, so what do you think about Mozilla made people want to contribute?

    Paul: I think the fact that it was an open-source-first company, and that it was a nonprofit, and it was all about the free and the open web and that sort of thing. Also, people had had very good experiences with Firefox for, like, a decade at this point by the time I was there. Well, maybe seven years, but, you know.

    Edith: It’s fascinating, ’cause I went to dinner last night with the CEO of MySQL, and I don’t know if I should say this, but he said—

    Paul: This is Marten Mickos?

    Edith: Yep. So he gave a talk—

    Paul: He has a new startup?

    Edith: Called HackerOne?

    Paul: Oh, he joined HackerOne.

    Edith: Yeah, he’s the new CEO. So the dinner was kind of like a debutante ball for him.

    Paul: Nice, nice.

    Edith: So what he said was that somebody had asked him flat out how much of MySQL was actually from contributors. He said 1 percent.

    Paul: Seems about right.

    Edith: He said the open source was more just a branding thing. Like it wasn’t actually like a productivity thing. It was more just a branding thing.

    Paul: So you have this in Mozilla as well, that any time that there’s a really avid contributor, that Mozilla would hire them.

    Edith: Yeah, that’s what he said.

    Paul: Right, and if Mozilla didn’t hire them, very often they stopped being a contributor. Like, it was a real sort of a double-edged sword.

    So someone who did it for six months, they were doing it for the love of the game. They weren’t doing it to be hired at, at least I think for the most part, they weren’t doing it to be hired at Mozilla. And then six months in, someone’s like, well, you know, you’re doing a really great job, wouldn’t you love to work on this full-time? They’re like, yeah, that’s so exciting. They’re flown out for an interview. And then it’s like, you know, actually, I don’t think so.

    Edith: Wait, they didn’t wanna work for Mozilla, or Mozilla didn’t wanna hire them?

    Paul: Mozilla didn’t wanna hire them. I mean, this isn’t a particular case. But you know, some subset of people, Mozilla didn’t wanna hire. And then what happens? Well, you kind of don’t wanna keep contributing to the company that just told you to f*ck off.

    Edith: Yeah, I mean, it’s so true. I mean, I think there’s a very human thing about volunteering.

    Paul: Right, right.

    Edith: I volunteer, I was the moderator for a long time for the Lean Startup Circle, and somebody new came on, and they’re like, yeah, I’d love to raise some money so that we can get paid, and I’m like, they cannot afford to pay me.

    Like, I don’t do this to get paid, ’cause—

    Paul: Yeah, it’s like, it’s an altruism that people are trying to do.

    Edith: Yeah, I was contributing like 10 to 15 hours a week, and at my going rate, it’s not, it’s like, I’m not doing this to get paid, I’m doing it ’cause I like giving back.

    Paul: So there’s this thing that we were talking about earlier, about the current discussion around people getting paid in open source.

    Edith: Yeah, and I think a lot of what has fueled open source is the whole 20 percent time. Like, that Google will give their engineers 20 percent for open source, and I know that other big companies, like Twitter and so on, do that too. So these people are, in essence, getting paid.

    Paul: Right, right, there’s a, in most companies, in most startups at least, there’s an unspoken rule that you’re expected to contribute to the libraries that are in your application, and that sort of thing. If you’re spending time upstreaming paths that you’ve deployed, that’s work.

    Edith: Yeah, so I think we’re agreeing, which, I thought we agreed that we were not gonna agree. It’s funny, ’cause I’m Twitter friends with a guy I’ve never met, who’s, he started Jango, and right now what he tweets a lot about is—

    Paul: This is Jacob Kaplan-Moss?

    Edith: Yeah.

    Paul: Okay.

    Edith: That people are working their nights and weekends, and contributing to open source, and then just, people are kind of, for lack of a better word, jerks. People are like—

    Paul: People are jerks.

    Edith: Yeah, like—

    Paul: Even when you’re paid, people are jerks.

    Edith: But it’s more aggravating.

    Paul: Right, right, because you’re doing it for free, or whatever.

    Edith: Yeah, and so people will, like, they’ll be like, ‘I have this pull request and it’s been open for two hours. Why haven’t you reviewed it?’

    Paul: Right, right. I mean, we get the exact same thing. We get customers who email us, or more often it’s the people who get, who are using the free version, who send us a support request and then, 30 minutes later they’re on Twitter going, I can’t believe Circle hasn’t solved my problem yet. It’s been 30 minutes.

    Edith: What do you think drives that?

    Paul: There’s a pattern across the population in general, that people are jerks.

    Thanks for listening to this episode of To Be Continuous, brought to you by Heavybit and hosted by me, Paul Biggar of Circle CI, and Edith Harvaugh of LaunchDarkly.

    You can find the full episode of this podcast here. For more on Continuous Delivery and to hear other Heavybit podcasts, copy this link into your favorite podcast player. We’ll have show-specific RSS and iTunes subscription options soon. You can also follow To Be Continuous on Twitter at @continuouscast.

  • To Be Continuous Podcast: How Do You Deal with Your Data?

    To Be Continuous Podcast: How Do You Deal with Your Data?

    Welcome to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. Edith Harbaugh and Paul Biggar talk with Peter van Hardenberg about about continuous delivery in the context of organizational structure, databases and pricing.

    Peter is a co-founder of the Heroku Postgres product team. For more than three years he has been responsible for designing, developing and operating the largest cluster of Postgres databases in the world. He knows how to get databases running, keep them safe and ticking fast.

    This podcast is brought to you by Heavybit, a program that helps developer startups take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/241491956?secret_token=s-oZ2T7″ params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=’166′ iframe=”true” /]

    Paul: So Peter, it’s traditional to ask you, what do you like best about continuous delivery?

    Peter: Well, I’m increasingly old and I’ve done a lot of jobs and so I remember delivering Gameboy cartridges to the market and a Gameboy cartridge is a physical device that goes into a Gameboy, hence the name, and that process involves physical manufacturing and you get one shot and so there’s a tremendous amount of basically wasted and un-fun energy that goes into making sure you never make any mistakes.

    My favorite thing about continuous delivery is the ability to make mistakes and move on with them.

    Paul: Nice.

    Edith: This would be a great time for you to introduce yourself.

    Peter: Yeah, hi, my name is Peter van Hardenberg. I’m a Heroku old-timer, I’ve been with Heroku about six years now and I co-founded the Heroku Postgres team and have built a lot of products since then.

    Edith: So it’s funny you’re talking about the Gameboy because Paul and I were just talking about that.

    Peter: About Gameboys.

    Edith: Because I said one of the things that had fueled the rise of Zynga was continuous delivery.

    Paul: That’s the first thing we found about continuous delivery that was actually bad.

    Peter: So I guess they’re probably still continuous delivering somewhere. In the great startup in the sky. They’re still around, aren’t they?

    Paul: They’re still around, they’re still public and,

    Peter: Farmville will never go away, I’m sure.

    Paul: I think they’ve moved into other lesser things.

    Peter: Developer tools.

    Paul: Imagine.

    Peter: Gamified developers. Well actually, it’s how all developer tools are becoming now.

    Edith: Github is gamified. How many chickens do you have, can you make a nice pattern out of them.

    Paul: New watches.

    Peter: It’s a bit like a game except if you’re losing the game, you lose your job.

    Paul: Or you just can’t get new jobs.

    Peter: Well, that’s actually probably more accurate, yeah. Yeah it’d be a weird company if they let you go because you didn’t have a long enough commit streak.

    Paul: On open source projects that weren’t your job.

    Peter: I like that, actually, let’s make that a game.

    Paul: We were talking in the last episode, as well, about open source funding models, or the anger over lack of open source funding models.

    Peter: That’s a tough problem to solve.

    Paul: That would be an interesting one. You get fired if you don’t have enough contributions.

    Peter: Yeah, just put it on your quarterly review, current Github streak.

    Paul: Let’s put this suggestion on Hacker News and see if people agree with it. I can pretty much picture how this is going to go.

    Peter: I try not to read the comments.

    Edith: So what do you think is liberating about the ability to make mistakes?

    Peter: Well, I mean, more important than the ability to make mistakes is the ability to try things, right, I mean we, I’m so accustomed now to being able to see in great detail because of a SaaS model, sort of the reality of what my product is in the market and then vice-versa, I can turn that around and say, well, basically live and in real time, what’s happening if we put something out there, do people care?

    Does it help with adoption? Is it suddenly throwing a bunch of errors? And if there’s a problem, you can roll back very quickly. You know we run a database product and so availability is really, really important but only for a very small core of the service. People are generally pretty understanding as long as your, sort of, problems and outages are brief.

    Small and, sort of, blast radius and quickly remediated and I think that they prefer to get features vs. stability, sort of, at the periphery. At the core, they need stability over features, so, you know if you’re a stripe, right, you need to keep the payment processing online but if some ancillary piece of your API has a problem that’s probably not as big of a deal.

    Paul: The dash part is less important—

    Peter: Right.

    Paul: than the payment going through.

    Peter: Well, and responsiveness and progress are so important, and I think because we’re in such a changing market, the ability to drive forward is at least as important as the ability to keep things stable but you can’t do both in the same place, basically, right?

    I think continuous delivery is cool because it’s separated, sort of, a slow-moving core from a fast-moving periphery.

    Paul: So when you’re creating a software product, you really want every single component of it to be continuously deliverable, which means you want to be able to migrate from the thing that you are doing now to the new thing.

    And so it’s interesting, or there’s an interesting thing about databases because databases, I think, were fundamentally not built for this model.

    Peter: They’re heavy.

    Paul: Right. We had a problem recently where we deleted an index.

    Peter: Ah, yeah.

    Paul: And, whoops, that index was in use. And we thought it wasn’t in use but it was in use.

    Peter: You should’ve been using Heroku Postgres, it would’ve told you.

    Paul: Oh so, that’s interesting. So the thing that I wanted was to be able to, you know, partially delete that or let’s run all the queries without that and then after an hour, I can actually delete it. What does Heroku Postgres offer you?

    Peter: Well so we have a number of diagnostic tools where you can see exactly how often an index is being used. In addition, next time you’re trying to delete an index, wrap that in a transaction, so say, begin and then drop the index, then do whatever and then say, rollback, and that way, you can actually test as though you didn’t have the index because Postgres MVCC model means that all that stuff is totally transactional.

    Paul: So you’re, when you say do all the stuff in between, what do you mean?

    Peter: Whatever queries you want it to run. Also, and we’re getting into minutia and then some baseball here but you can actually disable indexes specifically, sort of at a session level as well.

    Paul: Oh, interesting. This is all Postgres or Heroku?

    Peter: Yeah, that part’s just Postgres I mean, we built the tooling around it so that you can do these things, kind of, the easy way but, I’ll tell ya, if you read that 3,000-page Postgres manual, you’ll find some real gems.

    Paul: Well so part of the problem might be using a thing that’s not a real database?

    Peter: Oh, you’re not using a real database?

    Paul: We’re not—

    Peter: Oh, that’s right, I forgot.

    Paul: I mean we are using a real database, we’re using many databases but that particular one was not in a real database.

    Peter: Well you know, even baby databases grow up to be real databases someday.

    Paul: Yeah, well, fingers crossed that that particular database that starts with an M and that we shall not discuss.

    Peter: Oh, we don’t like to speak ill of anything.

    Paul: No, no not at all, especially Mongo.

    Edith: No, you said the word!

    Peter: You said the M word! So coming back to, sort of continuous delivery of every part of the stack, yeah, it’s interesting, I mean we’ve had some interesting challenges with databases in particular and I actually gave a talk at Heavybit a while ago about, kind of, how we treat databases less as part of a service in sort of the classical sense but more like a factory.

    So basically what happens is when people request databases from us, we stamp one out and then, you know, we try and keep it pretty low entropy from that point on so we have a lot of tooling and support services around the database but the actual database itself, once it’s given to a customer, we mostly try and keep our hands off it.

    Now if there’s security issues or performance issues or whatever, we’ll go in and we’ll kind of warm it up and do some work on it. But even still we mostly prefer to actually replace the database. What we do is we create a replica, bring it up to speed and then we transition the load over to it. And that’s kind of a form of continuous delivery for data services, which has been surprisingly powerful as an abstraction.

    Edith: So you’re basically, you’re treating them as cattle, not pets.

    Peter:

    Oh absolutely, it’s all about the cattle and model and when you’re farming a very large crew of livestock, you can’t really afford to give each one their own personality.

    Paul: This reminds me a lot of, I think, what started out as an IMVU way of doing data migrations, I’m sorry, of doing schema migrations without locking a whole table. For our people at home, I’ll describe this.

    So when you want to add, let’s say you want to add a new value and it has a default to a particular table in your schema. There’s a possibility of that if it’s a particularly large table, for it to be locked while that happens. At least this used to be the way, I think modern databases are much better at this now.

    Peter: I believe for some default values, it’s now the case that you don’t need to do a table lock but that might be pushed until 9.6 or Postgres, at least but it’s a common and challenging operation.

    Paul: And so what IMVU started doing was instead of adding to a schema, they said that for every table, that schema is immutable and instead they created a new table and then they migrated the data from that table so every time you want to read from the user, you’re reading from user table 38 at the moment and then you start to read from user table 39. If the user isn’t in user table 39,

    Peter: You fall back to 38.

    Paul: fall back to 38, find it, migrate it, write into 39 and then do the thing. Then you have a background process.

    Peter: If you have any listeners who are listening from the world of the Internet and would like to try implementing this technique for themselves, Postgres has something called a schema search path so you can actually, basically make each schema its own version and then you have a search path so if the table wasn’t in the newest version, it would use the last version’s search path.

    Paul: Oh, wow.

    Peter: You could set a version on your code then basically, as you make new versions of the tables and the new schema, you can kind of move them forward. This actually sounds like it would probably work. Now it’s really easy to say that from sitting around a table here in beautiful San Francisco but you have any listeners out there who do give this a try, have them send me a note.

    Paul: I was not expecting this to be a session about continuous delivery databases and the intricacies of modern versions of Postgres.

    Peter: I have to say, it’s one of the hardest things is basically, how do you deal with your data? I mean, data is sort of fundamentally heavy, I mean I think about it as having mass, which is to say, if you’ve got a few terabytes of data on a disk somewhere, just getting it out of that server and into another server is gonna take hours.

    Paul: Right.

    Edith: Right, just querying the data, loading it all into memory, that’s gonna take time and so this is really a challenge for continuous delivery and people have tried all kinds of schema versioning and replication strategies and sharding strategies but, you know, the old question, okay you deploy a new version of your code and it needs the new column but the new column’s not there yet. Okay so you move the data to the new column and then you deploy your code but then the old code doesn’t have it.

    Paul: Right, right.

    Peter:

    And these are hard problems and no one’s really done a great job of answering them. So far, sort of the state of the art is, you know, you continuously deploy the new version of your code that supports both schemas then you continuously migrate your database and then you deploy yet another version.

    Now without continuous deploy this is even harder—

    Paul: This has to be tough on a Gameboy cartridge.

    Peter: Oh yeah, let me tell you, I think we had 4k of RAM or something like that so at least it was fast. But yeah I think data and continuous delivery are both challenging interactions for sure.

    Paul: We end up at this problem in the testing world when large customers, in particular, large enterprises use the old way of testing, which is you actually use a dump of your production database as part of your test suite and one of the major challenges we get when we’re looking at older companies who are coming to the new world, is, you know, your server’s over here, my server’s over there and I want to get a 5 Gig database dump onto your server to run my tests, how do I do that?

    Peter: Right and it’s trying to, like, move a bathtub through a straw. So how do you do that? You just wait, right?

    Paul: We wait until they stop doing that and then they can become our customers.

    Edith: So do you turn customers away then?

    Paul: We don’t turn them away but we don’t really support that use case, they’re going to have a miserable experience.

    Peter: I mean there’s data migration again, right, just coming back to the physics problems. There are things that are getting better so,

    Paul: The speed of light is improving?

    Peter: the speed of light has gone up 13 percent in the last year, did you know that?

    Paul: I don’t know if you’re joking, I presume you’re not.

    Peter: It absolutely has not, yeah. They call it a constant for a reason. I suppose it could.

    Paul: I thought there might be something about, like,

    Peter: Dark matter, no but it was very convincing the way I said it, wasn’t it?

    Edith: Like this is Hindenburg’s Principle?

    Peter: It’s uncertain. So anyway, the logical replication scene is getting better. Amazon’s just launched a new service, it’s three letters long.

    Paul: Oh really?

    Peter: Yeah, uh, got it, hang on. DMS, Data Migration Service. And so through black magic and probably by not working very well, it will move all your data from one database to another in sort of a streaming online way which is really cool if it works.

    Paul: Okay.

    Peter: But, you know, moving from Oracle to MySQL

    Paul: So you can talk to this service as if it’s MySQL and it will pull it from—

    Peter: The way it works is you basically set up a migration job and if you think about it, it basically queries one database and writes to another and it’s a little bit of a hack job in the sense that you have to run, like, a desktop client if you’re moving from a local machine. It’s all very complicated and kind of baroque but,

    Paul: So it’s an Amazon service? It isn’t you’ve used Amazon before, I gather.

    Peter: But yeah, no, it’s what everybody’s always wanted which is to move their data from one place to another without having to think about it and, of course, it’s never gonna work.

    Paul: Okay.

    Peter: But, you know, it’ll help, it can help, things could be less terrible. I think ultimately that’s what got me into databases was the understanding that no matter how impossible it would be to make things good, at least we could make them a little bit less bad.

    Paul: Right.

    Paul: The soul-crushing misery of writing software could at least be slightly less crushy.

    Edith: I feel like writing software is one of the greatest joys in life because you get to create something from nothing.

    Peter: So you don’t work in databases.

    Paul: I’m not sure you work in software.

    Edith: So what are the biggest changes you’ve seen in continuous delivery?

    Peter: Oh, ubiquity, certainly. It’s rare to meet someone these days that, maybe I’m just lucky enough, I live in San Francisco.

    Edith: You hang out at Heavybit.

    Peter: And I hang out at Heavybit with a bunch of other crazy people but I guess I would say growth then, right, I mean we’re seeing large enterprises practicing continuous delivery—we’re seeing small shops practicing continuous delivery, it’s not, sort of, a bleeding-edge concept that only, sort of, a small cadre of, like, sort of, avant-garde developers are doing. It’s really, I think it’s kind of the mainstream.

    Listen to full podcast here to hear Peter talk about continuous delivery in the context of organizational structure, databases and pricing.

     

  • Technical Debt

    Technical Debt
    “Debt can build to a point before it needs to be addressed. But as many new to credit cards can attest, ignoring the debt building up won’t make it go away. — Jimmy Bogard, bit.ly/1VNh1ct

    — ROELBOB

  • To Be Continuous: Empathy in Software Development

    To Be Continuous: Empathy in Software Development

    Thanks for listening to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk with Sam Stokes about empathy in software development, how mini-services are the new microservices, and constantly updating software in a constantly updating world. Sam is currently a freelance software consultant and previously was the co-founder of Rapportive.

    This podcast is brought to you by Heavybit, a program that helps developer-focused companies take their product to market.

    [soundcloud url=”https://api.soundcloud.com/tracks/237727552?secret_token=s-k6OKe” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=flase” width=”100%” height=’166′ iframe=”true” /]

    Paul: Alright, Sam, so what’s your favorite thing about continuous delivery?

    Sam: I like to think about it terms of shortening feedback loops. A lot of things that startups do can basically be looked at as doing something and then waiting to get some feedback on it, and then deciding whether to do more of that thing or to do a different thing.

    And continuous delivery is, people talk about sort of reducing the batch size and also about shipping stuff faster and clearly having some kind of automation. And continuous delivery is going to help you with both of those things.

    Paul: So is this specifically on the, sort of on the journey of a small startup towards product market fit? Is that kind of the sense you’re thinking of it as?

    Sam: I see it as applying to teams of different sizes. So for sure in a small startup you’ve got you need to iterate fast, you’ve got all these product ideas. But I think even if you’re a small team in a big company as I was recently working on, and still have a lot of the same pressures you have, product managers who want to try new ideas, you have customers who are sending in feedback all the time—some of which you want to listen to, some of which you want to tune out.

    It’s sort of, in some ways, a lot like being in a startup. At least, ideally it would be. And a lot of the problems you can run into are basically about not being able to close those loops quickly enough.

    Edith: Sam you’re someone who knows a lot about loops, so now might be a good time to introduce yourself.

    Sam: Sure, my name is Sam Stokes. Until recently I was working as a staff software engineer at LinkedIn. And right now I am, I guess a freelance software consultant, thereabout.

    Paul: I feel there’s a little bit of history missing there.

    Sam: Alright. Going back in time a little bit. So I was a co-founder of a startup called Rapportive. That was a plugin for Gmail that when somebody emailed you, would show you a social profile of that person in the right-hand side next to their email.

    So that you could get some insight into who you were talking to and put some more humanity into your communications with people—that’s sort of the way we saw it.

    Paul: That’s still the only plugin to Gmail that I use.

    Edith: I use it, too, and it’s for exactly that reason, ’cause I do a lot of customer development, and it helps so much to do, to empathize. To see that this—

    Paul: Right, right, yeah is a real person.

    Edith: You know, to see their face and you feel a connection.

    Paul: It’s interesting just with kind of customer communication. One of the tools that we use is Intercom. And once I started using Intercom it was the same sort of feeling I got the first time I used Rapportive, that like I feel that there’s a human at the other end of this and I feel I know who that human is and it just changes how you communicate with the people.

    Sam: It’s really motivating to hear and to see other examples as well. We really felt like email and sort of a lot of business communication in general can be sort of cold and formulaic.

    Yeah, just bringing that human element, even just seeing a face. I haven’t used Intercom but I’m guessing they have some similar ideas.

    Paul: Similar, when you send out messages or reply to someone it does the person’s face there.

    Sam: Right, we had a sort of plugin API in the, it was a sort of private beta version of our product and one of the things people used it for, so this would let you plug into the sidebar, people would plug it into their customer database and so they could see something like, they’d get an email from a customer it shows them how long they’ve been a customer, how much they’ve spent in their store—

    Paul: Right, right.

    Sam: What version of the app they’re running, that sort of thing.

    Edith: Was your original idea to do that sort of empathy or was that something that just you iterated into?

    Sam: The empathy was where we started. There were lots of ideas about taking all the information from all these different sources, you know public sources like, truly public sources like Google, quasi public sources like a Facebook profile if you’re friends with them on Facebook, and then private sources like stuff in your company database.

    All stuff that you would be able to search for if you could be bothered to go and do the searching but doesn’t necessarily bring itself to you while you’re talking to that person. And yeah, it was very much, what should you know when you’re talking to this person in order to make them feel like they’re interacting with a real person and an empathetic one.

    Paul: So I remember when you launched Rapportive it was like, or it felt to me like it was, an instant hit and maybe that was just the overnight success that comes after years of trying. I had a doco and what was the kind of software delivery process of that?

    Sam: Hmm, that’s a great question. I guess there’s two parts to that really. So how did it get there and what was our process that allowed us to survive it? So I guess how did we get there, it was a certain amount of luck in that process as there probably always is. We built our prototype and it was a pretty minimal prototype.

    You know, it had the functionality that was like this awful demo website up. The only reason there was a website up was we were trying to raise a seed round at the time, we were applying to OIC. Y Combinator requires that you have a demo site that’s public. They don’t like it to be password-protected. And so we had this thing up where you could install it and we had like 20 users. One of our friends decided that he was going to email a friend of his at one of the blogs, I think it was ReadWriteWeb and say, you know, hey you should check this out.

    And his friend checked it out and it turned out that one of the demographics that really liked the product that we built was journalists because they get a lot of emails from people they don’t know and they want to sort of know an extra nugget. I guess that was one of our takeaways was like, if your startup is really useful for journalists, that’s a great way to get some early press.

    Edith: That’s what worked for TripIt because journalists travel a lot. So TripIt just got all this free press basically ’cause journalists used it, loved it, and blogged about it. Unfortunately, journalists do not do continuous delivery so much.

    Sam: So how do we survive that? ‘Cause what happened in that 24-hour segment was that we went from 20 users up to 20,000.

    Paul: Wow.

    Sam: So I guess one part of it was our CEO had made the original platform choice. His name’s Rahul Vohra, he has a computer science degree, he’s an awesome technologist, but his focus is not generally on writing the software. He’d been writing the prototype, and he was like, ‘I need the quickest thing that I can get to get this market, so I’ll put it on Heroku.’

    Which turned out to be a life-saving decision because when the 1,000 X traffic spike arrived we just added some more dynos. So that was how we didn’t burn down. What did we do after that? I guess in terms of software process we were pretty young as a company, so it was pretty much hack away, read all the tweets. But there was a lot of, what you might call customer development more than software development at the time. So we pretty quickly decided that we were going to try and respond to every single tweet and every single email and so on to like try and live this idea of a good customer experience.

    And I think that helped to sort of amplify the viral effect of that press spike because people were sort of retweeting the things we were saying and getting a generally good feeling about us.

    And so a lot of those tweets were like, ‘Oh I wish I did this,’ or ‘I wish I did that,’ or ‘I wish you could see Twitter profiles in there.’ So it was a little bit of sort of customer support-driven development. Which obviously doesn’t scale but it can be a pretty good way to build some goodwill early on.

    And I think a lot of our users were sort of quite passionate about using us, ’cause they felt like they got that personalized attention.

    Paul: So did the, when you scaled the dynos did it just work?

    Sam: It worked perfectly until we hit the rate limit for the API that we were getting all our data from.

    Paul: The LinkedIn API?

    Sam: At the time we were using a service called RapLeaf.

    Paul: Oh yeah, yeah, I remember them.

    Sam: It wasn’t their main product, but they provided an API which … So basically the thing that Rapportive does, which is take an email address and produce social profiles about that email address. At the time that was entirely just UI wrapped around RapLeaf. We ended up building sort of our own version of that.

    And so we’d been using their API on a sort of demo basis. Because again, 20 users. Yeah, we immediately had to learn about things like request queues, so that our app wasn’t just falling over waiting for their API, and then frantically Biz Dev-ing to try and increase our rate limit, while at the same time figuring out the best ways to get around the rate limit without upsetting the Biz Dev deal. Yeah, we didn’t really have much of a sort of continuous delivery process at the time.

    But I guess not as many people were talking about it—this was 2010. I think the first blog post had come out from Iron View and so on, but I think it wasn’t as, you know, there weren’t full startups supporting it. What we did have was the Heroku deployment model, which was basically like within five seconds of writing the code you can deploy it to production.

    Which if you squint at it, kind of is continuous delivery because you’re saying, all of the bits of your deployment that you have are automated. We didn’t really have many tests at the time, so the tests we had were automated ’cause there weren’t any. Our deployment was absolutely, you do it get push, you don’t have to like move any servers around or anything like that.

    Paul: Seems like the testing model around, or the continuous delivery model around a Gmail plugin is probably going to be a fairly difficult process anyway.

    Sam: We never really had a good answer to, you know, what happens when Gmail changes their UI. The surprising part was it didn’t happen that often. We thought it would be breaking every week and we’d need all kinds of canary accounts and end-to-end testing to monitor that kind of thing. In the end, it happened rarely enough and we noticed quickly enough that we could usually just fix it on the spot. So even when we got better at testing and that sort of cycle of things.

    Paul: We had a problem or I guess solution with GitHub. So GitHub used to be super flaky around two, two and a half years ago, when they were still on Rackspace. And whenever something would go wrong with GitHub we would find out immediately. All of our, all the developments would start failing and so before they knew that there was a problem, we knew there was a problem.

    And we would often like, declare on Twitter, oh we think GitHub’s having a bit of an issue and two minutes later their status would tweet us and then we would retweet their tweet.

    Edith: So they were using you for their free testing.

    Paul: Well, our customers at least.

    Edith: No I mean you were acting as their tester and telling them that—

    Paul: Well, I mean it was very, you know, automated on our but we didn’t have necessarily automated fixes at the start. It kind of happened every, you know, two or three months and they would swear this is the last time and—

    Edith: It’s always the last time.

    Paul: And in the end it happened like six or eight times and by that time we had built a thing that allowed, you know, certain fixes, but it wouldn’t be quite, you know, fully automated process that you would expect if it was happening all the time ’cause it always seemed like it was gonna be the last time.

    Sam: And it’s probably always happening for a slightly different reason or in a slightly different way. It’s actually, it’s an interesting question to me in the subject of continuous delivery in general. People always talk about every time you commit you run your tests, every time you commit, you deploy.

    But that’s actually only one of the kinds of events that you’re dealing with in a software life cycle, right, when something’s actually running. I mean, sure you’re changing the code but also the world’s changing around you. And I feel like you always end up needing a little bit of both—you need something that’s gonna trigger when you change it, but you also need sort of an eye on everything you depend on. And, of course, even figuring out what all those things are can be interesting.

    Paul:

    So I wrote a blog post about this on the Heavybit Blog, I think a while back, and it was about third-party APIs you know, how do you test third-party APIs. And the conclusion that I came to was that, so if you run the tests as part of your test suite, then if something breaks, your test suite will be broken, but also some things will just like randomly break in production.

    And then of course you can’t deploy because your test suite is broken. So you’re in kind of the worst of both worlds. And so what I considered is, is that you need something which, you know, is just permanently monitoring. Like whatever API things you do constantly hits your personal accounts or your test accounts for those things, and then stub it out in your test suite.

    And that way you kind of get that best of both worlds, your test suite doesn’t fall over and you get an actual monitoring event when the third-party service goes down in production.

    Sam: I can tie this to an experience from LinkedIn, actually. So LinkedIn has the model where there’s a staging environment and a production environment. And the staging environment is—I don’t know if it’s technically called a staging environment because it’s not exactly production-like, but what they actually call it is early integration. Because it’s the environment in which teams integrate with other teams.

    And so the idea is you deploy your stuff into EI, early integration, as soon as you can so that if any other team depends on you or vice versa, you can find out there before the thing hits prod. But what you’ve effectively got in EI is this constantly changing environment full of dependencies and you have exactly this problem and different teams which using different solutions to, well, let’s write some integration tests which actually hit the dependencies and then we’ll run them in our test suite when we deploy to EI or when in the build step which—

    Paul: I can’t tell as you’re describing this whether this is a good thing or bad thing.

    Edith: Yeah.

    Sam: I think there wasn’t a clear conclusion, but my conclusion is that it doesn’t work for the same reason that you’re saying.

    You need to know that if your test suite fails, it fails because of something you did. And then you need something else that can fail because of something someone else did.

    But if you’re ever unsure which of things it is, then the signal is lost for both systems.

    Be sure to listen to the full episode. While you’re there check out Heavybit’s library, home to great educational talks from other developer company founders and industry leaders.

  • A Guide to Public Cloud Security Tools

    A Guide to Public Cloud Security Tools

    Why is an arrangement of servers—which are constructed of hard metal, tend to run hot and weigh thousands of pounds—called a “cloud”? It only makes sense from the standpoint of an engineering diagram, in which data travels by an undefined pathway from origin to endpoint. In that respect, the cloud refers to the randomized packet transfer protocol that underlies modern computing.

    Although no one can know with certainty where the term “the cloud” originated, it’s clear what it represents. “The cloud is a metaphor for the Internet,” Cloud Camp co-founder Reuven Cohen told the MIT Technology Review in 2011.”It’s a rebranding of the Internet.” But for those with a security-first mindset, that metaphor is a problem. The Internet is a dangerous place that is filled with malicious people who enjoy misappropriating or damaging data. Storing private company and customer data anywhere on the Internet is a potentially hazardous course of action without the proper security controls firmly in place.

    Private vs. Public Cloud
    Private clouds are exclusive to a single enterprise—such as a enterprise company’s internal storage system. They are usually easier to secure but not as flexible in terms of rapid scalability. Private clouds also tend to have higher capital and operational costs. Public clouds, on the other hand, are operated by third parties that usually offer the cloud as service. Some of the most recognizable names are Amazon Web Services, Microsoft Azure and Rackspace in addition to the Google Cloud Platform and IBM’s cloud service.

    Private Cloud Security: Riskier Than You Think
    As Roger A. Grimes notes at InfoWorld, both private and public clouds come with risks. Private clouds may seem safer at first because they are under the enterprise’s tight control, but one of the biggest drawbacks is that many companies cannot afford the same level of security as Amazon or IBM. Even a company’s own employees are risk factors—they might do malicious acts or cause unintentional accidents. Private cloud networks rarely have geographic failover options, so all of its valuable data is more at risk. Public clouds, on the other hand, are supported by data centers all around the world—often making them more secure and efficient.

    Public Cloud Security: Still Facing Challenges
    Still, businesses typically consider these three specific security concerns when deciding whether to use public clouds:

    • Multitenancy: Sharing a cloud with other businesses—especially competitors—is the top concern of many corporate executives. Companies in public clouds must be confident that neither their private data nor their customer data will leak into other users in the space.
    • Virtualization: Exploits are grabbing headlines with increasing frequency. As identities across devices have become more fluid, access and authentication systems are having to work harder to approve the right users.
    • Ownership: Many public cloud providers are silent on the issue of ownership in their user agreements—leaving open the possibility that they could try, for example, to profit from the storage of the data.

    There are many ways to approach these challenges. The U.S. National Highway Traffic Safety Administration, for example, found a way to make the public cloud safe and accessible in less than one month by using some of the latest cloud security tools. The rapidly growing industry of cloud security firms and related software—such as those listed below—is evidence of the fact that cloud security is entering a new era. Businesses looking into a public cloud solutions may want to explore these options as additional ways to protect their data.

    The Top Public Cloud Security Resources

    • AppRiver – looks at messaging security for SaaS-based e-mail and web tools
    • Awareness Technologies – brings its SaaS-based DLP model to analyze mobile and the cloud
    • Barracuda Web Security Service – offers malware protection, URL filtering and application control
    • Bitglass – acts as a cloud-access security broker to secure applications and mobile devices
    • Bitium – handles identity and access management for BYOD and BYOA
    • BitSight Technologies – analyzes data on security behaviors and rates companies on security effectiveness
    • Centrify – centers on identity management across devices and applications
    • CipherCloud – handles encrypting or tokenizing data directly at a business gateway
    • Dome9 – examines firewall rules, IP address tables and ports to look for unusual web traffic
    • Evident.io – provides cloud security in partnership with AWS
    • ForgeRock – protects enterprise, cloud, social and mobile applications by identity access management
    • HyTrust – delivers access control, policy enforcement, hypervisor hardening and logging
    • IntraLinks – protects critical content and enables client control over data
    • Kismet – passively traces cloud machines while leaving no logs or traceable packets
    • Logz.io – users can create proactive alerts on selected events and relevant dashboards to aggregate and view data trends and monitor security threats including password brute force detection, access control and network access
    • Metasploit – takes a cloud IP address and tests penetration to assure that security is in place
    • MyPermissions – sends out alerts whenever apps or services try to access personal data
    • Nessus – operates as an open source vulnerability assessment tool
    • Nmap: the Network Mapper – tests penetration by scanning networks for congestion and latency
    • Netskope – discovers any cloud apps and shadow IT used on your network
    • Okta – manages logins across all cloud applications including Google Apps, Salesforce, Workday, Box, SAP, Oracle and Office 365
    • Proofpoint – focuses specifically on email to protect inbound and outbound data
    • Qualys – scans any and all used web apps for vulnerabilities in SaaS, IaaS and PaaS tools
    • SilverSky – offers email monitoring and network protection for HIPAA and PCI compliance
    • Skyhigh Networks – discovers, analyzes and secures cloud apps with logs from existing firewalls, proxies and gateways
    • SnoopWall – flags and stops access to high-risk data ports such as webcams, microphones, GPS and USB
    • WhiteHat Security – mimics threats to avoid coding vulnerabilities on websites
    • Vaultive – encrypts any data going from a network to an application
    • Zscaler – monitors all the traffic that comes in and out of your network along with protecting iOS and Android devices

    Public cloud security is only going to become more important with each passing year. Just look at these data points from Forbes reporter Louis Columbus:

    • Worldwide spending on infrastructure as a service (IaaS) is expected to reach $16.5 billion in 2015, an increase of almost 33 percent from the prior year
    • Cloud applications will account for 90 percent of global mobile data traffic by 2019, compared to 81 percent at the end of 2014

    So, with these trends in mind, how do you plan to secure your public cloud? I invite your thoughts in the comments below.

  • Multi-Tiered Escalation Policies are Paramount

    Multi-Tiered Escalation Policies are Paramount

    As any team charged with keeping systems up and running will tell you, having one person on-call isn’t enough. What happens if your on-call engineer sleeps through his alert? What happens if his phone’s battery dies without him knowing, or if he gets an alert at an inconvenient time, like when stuck on a bus or in traffic? It will happen.

    In today’s always-on world, companies can’t afford for that to happen without a backup plan. One or more people should always be waiting in the wings, ready to spring into action if a primary on-call is unable to perform his or her duties to the best of his or her abilities at any given time.

    Engineers who perform backup duty don’t need to be as “on-call” as a primary engineer — no pagers needed here — but what backup engineers lack in readiness they can make up for in numbers. It often makes sense to have multiple backups or an entire team of backups.

    Operationally mature companies often group the currently-on-call engineer and all of his or her backups into an escalation policy. Escalation policies set the order in which each person is alerted and the delays between alerts. While there are many ways to organize these escalation policies, here are some of the patterns most often used:

    Primaries Aren’t Enough; Determine a Secondary, Too
    Most companies have a primary on-call engineer, usually determined by a calendar that designates when members of an ops team are on call.

    Operationally mature companies take their on-call practice one step further and supplement this rotation with a rotation of secondary on-call engineers. Why a secondary rotation instead of just paging the whole team when the primary falls through? If a large team is paged, you get a ‘tragedy of the commons’ effect— it’s urgent that somebody respond to the issue, but not clear how urgent the issue is that each individual person respond. Having a designated secondary removes that confusion.

    The secondary rotation is set up to shadow the primary rotation and generally has the same members as the primary rotation. The difference is that the order of the engineers in the secondary rotation is usually offset from that of primary rotation— if your primary and secondary are the same person, you’re going to have a bad time. For example, if a primary rotation contains Alex, Bob, then Charlie, then a secondary rotation may contain Bob, Charlie, then Alex in that order. Companies often find that if last week’s primary is this week’s secondary, the it helps make sure there’s somebody with recent context who can help if the primary runs into trouble. It is becoming more commonplace for organizations to have both a primary and secondary on-call rotation; but, in an environment where customers expect zero downtime, secondary rotations are no longer an optional tool.

    Enable a Strong Support System
    Larger companies and companies with significant numbers of operations tasks often deploy a separate support team that handles all basic operations tasks before the organization elevates the tasks to an on-call engineer. This team — the first tier support team — usually knows how to handle all of the smaller, repetitive problems that crop up often and have clear resolution procedures. These first-tier teams often offer their support to multiple engineering teams and are placed first in an escalation policy.

    If first-tier teams cannot resolve the issue, escalation policies typically dictate that primary then secondary on-call engineers must complete the task. However, there is one more support network operationally mature organizations must implement: management. Management is ultimately responsible for a company’s systems, and needs to be in the know when severe problems happen— both to hold their teams accountable and to muster additional support. At PagerDuty many of our EPs have a rotating manager schedule at the top of multiple escalation policies, and many of our customers use this pattern as well.

    Put Your Tiered Teams to Work
    In practice, an organization policy for a hypothetical “Database Ops” team might look like this:

    • Assign the incident to the user who is on-call in the First-Tier Ops Team schedule
    • Assign the incident to the user who is on-call in the Primary DBA schedule
    • Assign the incident to the user who is on-call in the Secondary DBA schedule
    • Assign the incident to the team lead
    • Assign the incident to the dev manager

    This escalation policy would unfold over time, with timeouts of between 10 and 30 minutes before each subsequent escalation, depending on a team’s needs.

    Implementing an escalation policy that involves multiple tiers of resolution ensures that devops teams address all alerts — whether business-critical or minor — in a timely manner. Limited downtime is paramount in today’s fast-pace business environment. Customers expect their applications to run smoothly, or they expect to switch providers if devops teams cannot resolve issues fast enough. Organizations will be much less likely to see their processes fail, and much more likely to see happy customers, if they implement a multi-tiered escalation policy.

  • DevOps Chat: 7 habits of successful DevOps

    DevOps Chat: 7 habits of successful DevOps

    I recently had a chance to speak again with Sam Guckenheimer of Microsoft about his experience in leading the transition for Microsoft Visual Studio to Visual Studio online. In doing so, Sam and his team developed the 7 habits of successful DevOps.  You can listen to my interview below or read along below that. Also if you want to download the Microsoft paper on the 7 habits of successful DevOps you can do so here.
    [soundcloud url=”https://api.soundcloud.com/tracks/234949072″ params=”color=ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false” width=”100%” height=”166″ iframe=”true” /]

    Hi, everyone—Alan Shimel, DevOps.com, here for another DevOps Leadership Chat, and I’m joined today by Sam Guckenheimer of Microsoft. Sam, welcome.

    Interviewee:                 Hey, Alan. How are you doing?

    Interviewer:                 I’m doing well. It’s good to speak to you again. The last time we spoke was at the DevOps Enterprise Summit out in San Francisco.

    Interviewee:                 Yep, that was a lot of fun. That was a great conference.

    Interviewer:                 Yes, and you know what? I think—well, you contributed to making it great, Sam. It was a great presentation you made, talking about, kind of documenting the journey of Visual Studio from a COTS, you know, traditional kind of software model to a SaaS based program, Visual Studio Online. And, you know, I think that’s something that a lot of our listeners may be interested in, and how do they turn their current software offerings into SaaS based. And along the way, of course, you’ve come up with a, I guess you call it the Seven Successful Habits of DevOps?

    Interviewee:                 Yeah, so we think about the seven habits that we’ve applied as we’ve done our transformation. We started around 2010, we were in the, if you will, the pinnacle or the leader’s quadrants on all the analyst assessments as a provider of software for on-prem use in what was then called ALM—application life cycle management. And we said, “Hey, we need to become a great SaaS provider as well.”

    So we think of things in terms of these seven habits that we need to provide. Three of them you can think of as sort of Agile on steroids or second decade Agile. That’s team autonomy and enterprise alignment, is number one. We work in scrum teams or feature crews, think about 8 to 10 engineers and a product owner. They autonomously pull items from a common product backlog, and at the same time, they align to enterprise goals on a six month planning cycle, we call this a semester or a season, where we know, in that time frame, what are the needles we’re trying to move for the business? What are the things we’re trying to do at the epic level that count, and Sprintly we measure, how are we doing against that, and the things that each feature crew is doing are counting against that. They may move forward, they may move backward, they may decide, based on the data they’re collecting to do more or do something else. But you know, that’s in their control based on the things we’ve agreed for the six months.

    The second habit we have is a rigorous management of technical debt. We’ve known about technical debt for a long time, but when you’re running a service, you really can’t afford to let any debt creep in, and you have to stay clean all the time. So we make sure that we’re constantly monitoring both what’s happening in the production service and what’s happening in the development process. So if, for example, there is anything on the live site going wrong, we attend to it immediately. If there’s anything happening in the development process, like a feature crew is letting its bug count start to ride up, they have to stop immediately, fix the bugs, and can’t do any new work until they get them down to zero, so we stay clean on tech debt as we’re going.

    The third habit is a focus on the flow of customer value. We’ve talked about this since early days of Agile. Now, with SaaS, we can measure it. So we have great telemetry in the service, and we look at how people use our software, both quantitatively as we’re going forward; that is, we actually observe what gets used, what doesn’t get used, what the scenarios are, and we reach out to customers qualitatively, where we engage with the high usage accounts and we say, you know, “Let’s talk to them.” And we encourage our engineers to buddy up and reach out and say, you know, what’s gone well, what’s gone not so well, what would you like us to do that you don’t see? Are there things you wish for that we could be doing for you, and so on.

    So we reach out to customers, we listen to them through things like UserVoice and we observe their behavior through telemetry on a continual basis. That’s all kind of Agile on steroids.

    Interviewer:                 Mm-hmm.

    Interviewee:                 You know, those are the first three. Then you get to the stuff that’s really new with devops. The first is, of those, number four is what I’d call hypothesis driven development or a backlog that gets groomed with learning. This is what Eric Ries, in Lean Startup, calls “build, measure, learn.” So, in other words, the idea is that the backlog, instead of being the just, you know, what an omniscient product owner things is right, it’s—we recognize that it’s a set of hypotheses or beliefs. We think that these things will improve our service, but we’re gonna treat them as hypotheses and we’re gonna turn those hypotheses into experiments, and we’re gonna collect data against those experiments so that we can substantiate or diminish the hypothesis, and that data will be evidence gathered in production. That’s the fifth practice—everything is evidence that we need in order to build up or draw down against those hypotheses. We look, we use instrumentation on everything, we use telemetry on everything, and we measure it against what’s important.

    So, for example, if we think something’s gonna be a better sign of experience, we will run the new against the old in parallel, and we may, if we have multiple front doors, like in our case we have a web experience and an experience through the IDE, the integrated development environment, we’ll run those in parallel. So that would be a four cell test, you know? New and old, web and IDE, and we’ll see what the evidence shows us about how much better the new is than the old, and hopefully the new is better; it might not be. Hopefully we’re right, but sometimes we’re wrong. In any event, it’s the data that will tell us, and it’s a “bring your data” world.

    So, evidence gathered in production is our fifth habit, and that is made possible by our sixth, which is what we call a production first mindset, or what Satya, our CEO, calls a live site culture. The idea of a live site culture is that the site status is always first priority, and with site status, you have two goals. If there is a live site incident, you want to, as quickly as possible, remediate so that customers aren’t affected, and you want to drive to a root cause analysis. Practice the 5 Whys like Toyota, get to the root cause, and identify the actions that you need to take in order to prevent a recurrence of such an incident or something similar, and those root cause actions that you take, you need to complete in one or two sprints so that you’re hardening the service as you go learning from the live site incidents so that you’re always getting better.

    That’s number six, and finally, number seven is that we manage infrastructure as a flexible resource. We can do this because we use the public cloud—of course, in our case, that’s Azure. We work as a customer of Azure just like our external customers would. We take things a little earlier, but otherwise, we’re a customer when we need more resources for development or for load testing or to spin up more labs, we just do it.

    And we continue to expand the number of data centers we run, and we now run our Visual Studio Team Services in eight data centers around the world; that’s out of 24 regions that are currently active in Azure, and we fully automate the deployment, but we start each Sprintly deployment with a canary instance, which is where we work. It’s our own one of those stamps, and the one where our closest early adopter customers are so that we know from the canary if there’s anything wrong with the deployment, and if there is, we then restart at the canary before we roll out to the subsequent deployment rings. And that practice of canary-ing, going big with load testing is part of the deployment pipeline, and then using the automation to roll out in subsequent rings is something that we’ve used to harden the service and let us get to a global scale on a regular basis.

    Interviewer:                 Got it. So, Sam, these are obviously lessons learned the hard way, right, [Laughter] by doing and paying the idiot tax and doing that.

    Interviewee:                 Yes.

    Interviewer:                 For our leaders who are listening to you now, where to begin? Where to start? Obviously you can’t do all seven at once. Some of them are dependent on others. Where do they begin?

    Interviewee:                 Well, I think that the first thing is to make sure you have Agility down. I think you cannot do this if you are a command and control organization that doesn’t really let teams learn, and to recognize that you do need to measure your time to learning along with your other measurements, you also need to totally automate your release pipeline.

    So, the notion that you can have manual steps somehow between a development pushing code or checking in and the release of that is a thing of the past. You know, start with continuous integration and then quickly go to continuous deployment and put the telemetry in place so that you can see the evidence from production that gives you the feedback on everything that’s getting deployed.

                                        If you don’t have those things in place, the rest will crumble.

    Interviewer:                 Yeah, agreed.

    Interviewee:                 Sam, we unfortunately are coming up near the end of our time. I want to ask one more question of you, though, and it’s along the same vein. So, you’ve laid out the steps here for us. How did you get your team’s buy-in on this—or did you?

    Interviewer:                 Well, we certainly did. I don’t think it was very hard. I think that we all wanted to move to this world. I think that there was a broad recognition that we needed to become a first class SaaS provider. I think that there were some additional quirks in our case of being a great SaaS provider while we continue to be a great provider on-prem software, and maintaining that engineering discipline.

    The teams were all for it. I think the leadership needed to recognize that we were investing in new practices and that we needed to maintain the investment in engineering systems and engineering practices so we have a rule of thumb that about 20 percent of our investment goes toward non-customer visible engineering, things like live site work, and we don’t quarrel about that. If we need to do live site improvements, we do them.

    So I would say there was a little bit more work on management to accept the change in practices than there was on the ground with the engineering teams who were sort of chomping at the bit to get going.

    Interviewer:                 Well, there often is in devops, right? You have that “from the bottom up” and getting true top down sponsorship and cover is a problem, right? Not a problem that cannot be overcome, but an issue that we’ve explored previously in our leadership suite here.

    Interviewee:                 Yeah. Yeah, it helps that our, right now, our CEO comes from this world, Satya Nadella, so he is pushing everyone in the company to make the change and to learn the new practices. There were some folks in the middle who were not comfortable, necessary, but you know, they’ve come around.

    Interviewer:                 As they often do.

     

    Interviewer:                 Sam, anyway, we are probably over our time and I apologize, but thank you very much for sharing your Seven Habits for Successful DevOps. Continued success with your Microsoft Visual Online and the rest of your endeavors at Microsoft, and we hope to have you on again some time in the future and we can talk more about leading the transformation at companies around the globe with these new ways of doing things.

    Interviewee:                 Thanks, Alan. I’ll do one commercial, which, which is that there’s a URL, microsoft.com/devops, where you can download an e-book about our journey and a devops self-assessment if you want to read more.

    Interviewer:                 You know what, we’ll include that link in the article, too, in the transcript so people can get that. Sam Guckenheimer, Microsoft—thank you so much for being our guest today.

    Interviewee:                 Thank you very much, Alan.

    Interviewer:                 Okay, this is Alan Shimel for DevOps.com. Thanks, and have a great day.