Tag: continous delivery
-

Do You Continuously Deliver Your CD Pipeline?
If your job is to build and maintain the continuous delivery (CD) pipeline for your organization, do you feel you can’t keep up with your development counterparts in innovation? Everyone is raving about the new cool features in your company’s product. Who deserves the credit? People who wrote the code, right? It’s not that simple. Churning out new features requires a sound process and solid infrastructure. A well-designed CD pipeline is crucial to the developers’ success. It mitigates the inherent risks in a fast-paced, frequent iteration regime now prevalent in agile software shops. On the flip side, as software gets more and more complex and the appetite for a faster time to market grows, opportunities for innovation are in abundance for DevOps teams.
Multi-speed IT
So you’ve got two tactical teams with significantly different risk-reward equations, working at a different pace due to distinct objectives but marching toward the same strategic goal. You’re finding yourself in a multispeed IT environment. Development is focused on innovation and quick frequent iteration cycles. DevOps’ primary focus is to ensure the stability of the CD pipeline and keep a precise track record of what takes place inside the pipeline every step of the way. How do you make sure to stay ahead of the technological curve while minimizing risk?
Based on my experience on both sides of the DevOps isle, I recommend the following four steps to successfully build and maintain a continuous delivery pipeline:
1. Automate, Automate, Automate!
My personal motivations for rolling up my sleeves and scripting a task are:
- It became an annoying, mindless routine;
- It’s no longer a one-time thing and isn’t going away;
- My time can be spent on more important things; or
- I make mistakes and then waste time fixing them.
As with any type of code, organize the scripts into a sound logical structure easy to navigate and version-control it. You can always revise the code or completely rewrite it as the needs evolve.
2. Treat Automation Code as a First-Class Citizen
That’s right! As a practitioner, I learned the hard way that automation starts innocently small and simple. Then the requirements for more automated functionality begin to pile up rather quickly and things quickly become really messy. So write high-quality code from the get-go and reassess the design and implementation every step of the way. Remember that the visibility of your code is very high.
Apply the same standards to your code as your peer development teams do to theirs. Write tests—unit, integration, functional, end-to-end—as appropriate. Use the same CD pipeline to automatically build, package, deploy and test your automation code. This will give your pipeline another round of validation, your code will be of a much better quality, and your team’s credibility and reputation will soar.
3. Containerize the CD Pipeline Components
In the past, because the CD pipeline components such as build servers were manually provisioned, they often became the single points of failure, which made it hard to plan and roll out the modifications to homegrown, open source and commercial tools. It wasn’t unusual for some components to fall behind the latest release by a full year or more. In the meantime, a year’s worth of updates and patches could address a litany of issues, but the need for a maintenance window, usually on a weekend, and the uncertainty of the outcome often would outweigh the sought-after benefits. Containerizing the CD pipeline component can be the solution. Running a build server in a container, for example, will make it a lot easier to create an exact copy of the production version in a test environment and iron out the upgrade process more thoroughly ahead of the upgrade.
4. Test Commercial Tools
Commercial and open source projects test their software, of course. However, it could make a whole lot of sense for your organization to create your own integration, end-to-end and performance tests for the software you don’t write—especially if the software has a plug-in architecture like Jenkins or UrbanCode Deploy and you use community or homegrown plugins. Commercial tools typically don’t provide any guarantees that those plugins will work after the upgrade. Running your own automated tests that exercise the functionality of the tool in your environment and on your specific data gives the opportunity to validate updates as they become available and identify any regressions proactively.
The objectives of DevOps are not only to identify and eliminate the existing bottleneck in the organization’s continuous delivery process, but also to look out for any future potential bottlenecks and proactively address them before they become real issues.
So your team is DevOps. Your product is a CD pipeline. How do you continuously improve the CD pipeline for your product?
If want to learn more about how to effectively build and maintain your continuous delivery pipeline, please check out the newly released 2nd edition of the “Application Deployment and Release for Dummies” eBook.
About the author / Alex Seriy

Alex is an accomplished technologist and DevOps evangelist with over 25 years of experience under the belt. He is an automation enthusiast and usability guru, passionate about uncovering new use cases and innovating applications for platforms, frameworks and tools. At IBM, Alex is a subject matter expert in Continuous Integration, Delivery and Testing. He currently focuses on integration testing in the API Economy through service virtualization. Prior to joining IBM, Alex served as the Senior DevOps Architect at TIAA-CREF for 4 years. Before that, he was a consultant specializing in SCM, Release Engineering, QA and DevOps for over 10 years.
-

DevOps is a Risky Business
I spend a considerable amount of time with clients discussing DevOps and the potential value it can bring to them, but there is one theme that always resonates: the ability of a DevOps approach to eliminate risk. While many of us understand the benefits in terms of being faster (improved productivity), cheaper (reduced cost) and better (improved quality), I find that a conversation based around risk, especially when talking to the business or department heads, usually gets straight to their concerns in a language they understand.
IBM’s definition of DevOps is not simply about connecting development and operations; it is fundamentally about providing a capability that: 1) allows lines of business to connect with their customers (which may be internal employees) quickly, and 2) facilitates rapid feedback from their customers so relevant adjustments can be made. DevOps, therefore, embraces the philosophy of iterative (and incremental) development and delivery, whereby solutions are delivered through a series of incremental releases.

By ensuring all aspects of a solution (requirements, designs, code, executables, tests, etc.) evolve over time as each increment is completed, DevOps is fundamentally focused on addressing risks including:
- The risk of not knowing the requirements (have you ever tried to write down and freeze the requirements for a mobile app?)
- The risk of not knowing the solution (especially when it’s new, you’re defining a new architecture or you’re embracing new technologies)
- The risk of not knowing the effort (especially when the solution is novel for the organization)
- The risk of not delivering on time (when the pace of the team, for example, is an unknown)
- The risk of not providing the right level of quality (and the impact this may have on customer sentiment)
- The risk of adopting inappropriate practices (meaning that the approach being taken by the team is not working for them)
- The risk of having an ineffective delivery environment (including automation through the use of delivery tools)
I could go on. There are so many risks that can be addressed simply by delivering working software iteratively, that it’s surprising how many organizations continue to fall back to a “waterfall” mentality—even in high-risk environments where much is unknown.
The tack I’ve taken here originates from my time at Rational Software Corp. (which IBM acquired in 2003), where we would emphasize the need to deliver software iteratively and progress was measured based on the results achieved (i.e., working software) rather that the documents produced. My favorite graphic from that time is the one shown below, which shows how a modern approach changes the risk profile associated with a delivery project, since it focuses on removing risk as early as possible in the life cycle.

In summary, incremental and continuous delivery evident in a DevOps approach allows us to call out risk early and determine its resolution. DevOps often is used along with “lean startup” thinking (and its build-measure-learn cycle) and monikers such as “fail fast” since these are, at their heart, also focused on addressing risk.
Take the DevOps self-assessment to find out where are you on your DevOps journey
About the Author / Peter Eeles

Peter Eeles is an Executive IT Architect in the DevOps Global Tiger Team, IBM Hybrid Cloud, where he helps organizations improve their IT delivery capability in order to drive tangible business value. He has a particular interest in the adoption of architecture, agile, DevOps and cloud as innovation enablers. In the IT industry since 1985, Peter comes from a delivery background and was previously chief architect of IBM Rational’s Worldwide Solution Delivery organization, where he led many large-scale transformations in a variety of industries. Peter has written many papers and several books, including “The Process of Software Architecting” (2009), “Building J2EE Applications with the Rational Unified Process” (2002), and “Building Business Objects” (1998). He runs his own website at www.architecting.co.uk.
-

Webinar: How major enterprises are bringing their database Continuous Delivery up-to-date
Leading organizations are increasingly adopting continuous delivery, yet they often find it challenging to support database release automation, while ensuring database updates are delivered rapidly and securely.
These challenges have led a number of fortune 500 companies to database downtime caused by out-of-process updates, code overrides and other database glitches.
Attend this webinar and discover how some of the leading organizations addressed these challenges through understanding:
- The different requirements of application-code & database-code from a source control point-of-view
- Why their database was often left behind
- How they were able to use baseline aware analysis to overcome database updates challenges
- How they automated their database Continuous Delivery, from development to build to deploy
Recorded 3/22/16
Recording
Slides
[seoslides embed_id=”21adc07df1b9″ script_src=”https://devopsy.kinsta.cloud/embed-script/14078/14134/” overview_src=”https://devopsy.kinsta.cloud/slides/14078/” site_src=”https://devopsy.kinsta.cloud” site_title=”devopsy.kinsta.cloud” title=” ” /] -

Why Financial Companies Should Embrace Automated Testing
The drive to deliver new functionality to customers faster is being felt across every industry, and especially so for financial markets that are under pressure from new market entrants. Adopting agile development and continuous delivery can help financial institutions gain an edge and beat competitors to market, but it’s vital that quality is not forgotten.
Financial institutions are accelerating to update software and release new products in days or weeks instead of months. That means software must be tested rigorously and quickly. Manual testing simply cannot compete with automated testing when it comes to speed, accuracy and the ability to perform functions repeatedly over time. However, it can be challenging to implement successfully.
The Customer is Always Right
Customer power is forcing banks to improve customer experience. Accustomed to slick, efficient website experiences from such leaders as Amazon and Google, customers now are demanding similar experiences from their banks and credit card companies. Millennials expect easier access to banking, precipitating a shift to self-service, which presents all kinds of new IT challenges for banks.
When customers don’t like their online banking experience, they will simply choose another provider. A full 18 percent of millennials switched their primary bank in the last 12 months, according to a recent Accenture survey. When failed or flawed software gets into the customer domain, the fallout can be very damaging and expensive. Think: lost sales, disgruntled customers, bad PR and a massive price tag to fix the problems.
Test Automation a Solid Foundation
Introducing test automation isn’t just about finding defects more quickly and efficiently, though it will do that. It’s also about shifting the burden of regression testing and freeing manual testers to focus on issues such as exploratory testing, usability and penetration testing, where they can add the most value.
To implement automated testing successfully, financial institutions must devise a long-term plan that dovetails with their business objectives. Having a plan—and sticking to it—outweighs the importance of getting C-level support and winning a huge budget for automated testing. Without a plan, the most well-intentioned objectives can get rejected by senior management, stalled by other forces and, ultimately, forgotten.
Automated Testing Misconceptions
The plan should include a robust dismissal of two popular misconceptions about automated testing. The first—and bigger—one is that automated testing will produce instant cost savings, as fewer resources (mainly of the personnel kind) will be needed to perform testing.
Over time, automation will cut costs. But cost should never be the main factor when considering automated testing. Applying test automation greatly outweighs all the negatives of manual testing (time-consuming), but it also opens new doors for more sophisticated testing and, hence, the competitive advantage.
The second misconception is that automated testing is a one-time only exercise. Reality proves that automation is always going to be an ongoing process. Automated testing should be treated as (yet) another development activity: You write the production software and you write the software that tests the production software.
Overcoming the Challenges
Financial institutions face a threat from newcomers to the market, unencumbered by legacy system baggage and physical branches. They are offering disruptive products and new business models. How can existing financial institutions compete?
It obviously makes sense to include the business and marketing teams in the development process to develop new solutions for customers. Extend the DevOps mindset to break down those silos. A united team that includes test and marketing alongside developers can ensure that automated tests are well-designed and focused on business aims.
Pulling together to design and test new ideas can enable financial institutions to reinvent themselves and innovate. Embedding test automation in everything mitigates risk and increases velocity and quality. It can deliver the data businesses need to make the right next move.
Long-Term Vision, Short-Term Plans
Fast feedback is paramount to make the right business decisions, but financial institutions must cater to existing legacy systems and integrate new software deliveries from a range of vendors. Without test automation in place, it’s simply not possible to integrate deliveries, test them and reject them as quickly as possible if they are wrong.
The reality is that organizations must be able to quickly introduce new tools and technologies, because agility and innovation go hand in hand, but the impact on legacy systems must be repeatedly gauged. Test management tooling should be fluid and extensible to cater to this diversity. To accurately measure quality and risk in real time, test automation must be dynamic and flexible.
Failure to embrace automated testing will leave traditional financial institutions exposed and in danger of being swept away by the new wave of lean, agile, IT-focused competitors.
About the Author/Viktor Clerc
Viktor Clerc is product manager for testing applications at XebiaLabs, a provider of automation software for DevOps and Continuous Delivery helping companies to accelerate the delivery of new software. You may reach him at vclerc@xebia.com. -

To Be Continuous Podcast: How Do You Deal with Your Data?
Welcome to To Be Continuous, a show about Continuous Delivery and software development hosted by Paul Biggar, Founder of CircleCI, and Edith Harbaugh, CEO of LaunchDarkly. Edith Harbaugh and Paul Biggar talk with Peter van Hardenberg about about continuous delivery in the context of organizational structure, databases and pricing.
Peter is a co-founder of the Heroku Postgres product team. For more than three years he has been responsible for designing, developing and operating the largest cluster of Postgres databases in the world. He knows how to get databases running, keep them safe and ticking fast.
This podcast is brought to you by Heavybit, a program that helps developer startups take their product to market.
[soundcloud url=”https://api.soundcloud.com/tracks/241491956?secret_token=s-oZ2T7″ params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=’166′ iframe=”true” /]
Paul: So Peter, it’s traditional to ask you, what do you like best about continuous delivery?
Peter: Well, I’m increasingly old and I’ve done a lot of jobs and so I remember delivering Gameboy cartridges to the market and a Gameboy cartridge is a physical device that goes into a Gameboy, hence the name, and that process involves physical manufacturing and you get one shot and so there’s a tremendous amount of basically wasted and un-fun energy that goes into making sure you never make any mistakes.
My favorite thing about continuous delivery is the ability to make mistakes and move on with them.
Paul: Nice.
Edith: This would be a great time for you to introduce yourself.
Peter: Yeah, hi, my name is Peter van Hardenberg. I’m a Heroku old-timer, I’ve been with Heroku about six years now and I co-founded the Heroku Postgres team and have built a lot of products since then.
Edith: So it’s funny you’re talking about the Gameboy because Paul and I were just talking about that.
Peter: About Gameboys.
Edith: Because I said one of the things that had fueled the rise of Zynga was continuous delivery.
Paul: That’s the first thing we found about continuous delivery that was actually bad.
Peter: So I guess they’re probably still continuous delivering somewhere. In the great startup in the sky. They’re still around, aren’t they?
Paul: They’re still around, they’re still public and,
Peter: Farmville will never go away, I’m sure.
Paul: I think they’ve moved into other lesser things.
Peter: Developer tools.
Paul: Imagine.
Peter: Gamified developers. Well actually, it’s how all developer tools are becoming now.
Edith: Github is gamified. How many chickens do you have, can you make a nice pattern out of them.
Paul: New watches.
Peter: It’s a bit like a game except if you’re losing the game, you lose your job.
Paul: Or you just can’t get new jobs.
Peter: Well, that’s actually probably more accurate, yeah. Yeah it’d be a weird company if they let you go because you didn’t have a long enough commit streak.
Paul: On open source projects that weren’t your job.
Peter: I like that, actually, let’s make that a game.
Paul: We were talking in the last episode, as well, about open source funding models, or the anger over lack of open source funding models.
Peter: That’s a tough problem to solve.
Paul: That would be an interesting one. You get fired if you don’t have enough contributions.
Peter: Yeah, just put it on your quarterly review, current Github streak.
Paul: Let’s put this suggestion on Hacker News and see if people agree with it. I can pretty much picture how this is going to go.
Peter: I try not to read the comments.
Edith: So what do you think is liberating about the ability to make mistakes?
Peter: Well, I mean, more important than the ability to make mistakes is the ability to try things, right, I mean we, I’m so accustomed now to being able to see in great detail because of a SaaS model, sort of the reality of what my product is in the market and then vice-versa, I can turn that around and say, well, basically live and in real time, what’s happening if we put something out there, do people care?
Does it help with adoption? Is it suddenly throwing a bunch of errors? And if there’s a problem, you can roll back very quickly. You know we run a database product and so availability is really, really important but only for a very small core of the service. People are generally pretty understanding as long as your, sort of, problems and outages are brief.
Small and, sort of, blast radius and quickly remediated and I think that they prefer to get features vs. stability, sort of, at the periphery. At the core, they need stability over features, so, you know if you’re a stripe, right, you need to keep the payment processing online but if some ancillary piece of your API has a problem that’s probably not as big of a deal.
Paul: The dash part is less important—
Peter: Right.
Paul: than the payment going through.
Peter: Well, and responsiveness and progress are so important, and I think because we’re in such a changing market, the ability to drive forward is at least as important as the ability to keep things stable but you can’t do both in the same place, basically, right?
I think continuous delivery is cool because it’s separated, sort of, a slow-moving core from a fast-moving periphery.
Paul: So when you’re creating a software product, you really want every single component of it to be continuously deliverable, which means you want to be able to migrate from the thing that you are doing now to the new thing.
And so it’s interesting, or there’s an interesting thing about databases because databases, I think, were fundamentally not built for this model.
Peter: They’re heavy.
Paul: Right. We had a problem recently where we deleted an index.
Peter: Ah, yeah.
Paul: And, whoops, that index was in use. And we thought it wasn’t in use but it was in use.
Peter: You should’ve been using Heroku Postgres, it would’ve told you.
Paul: Oh so, that’s interesting. So the thing that I wanted was to be able to, you know, partially delete that or let’s run all the queries without that and then after an hour, I can actually delete it. What does Heroku Postgres offer you?
Peter: Well so we have a number of diagnostic tools where you can see exactly how often an index is being used. In addition, next time you’re trying to delete an index, wrap that in a transaction, so say, begin and then drop the index, then do whatever and then say, rollback, and that way, you can actually test as though you didn’t have the index because Postgres MVCC model means that all that stuff is totally transactional.
Paul: So you’re, when you say do all the stuff in between, what do you mean?
Peter: Whatever queries you want it to run. Also, and we’re getting into minutia and then some baseball here but you can actually disable indexes specifically, sort of at a session level as well.
Paul: Oh, interesting. This is all Postgres or Heroku?
Peter: Yeah, that part’s just Postgres I mean, we built the tooling around it so that you can do these things, kind of, the easy way but, I’ll tell ya, if you read that 3,000-page Postgres manual, you’ll find some real gems.
Paul: Well so part of the problem might be using a thing that’s not a real database?
Peter: Oh, you’re not using a real database?
Paul: We’re not—
Peter: Oh, that’s right, I forgot.
Paul: I mean we are using a real database, we’re using many databases but that particular one was not in a real database.
Peter: Well you know, even baby databases grow up to be real databases someday.
Paul: Yeah, well, fingers crossed that that particular database that starts with an M and that we shall not discuss.
Peter: Oh, we don’t like to speak ill of anything.
Paul: No, no not at all, especially Mongo.
Edith: No, you said the word!
Peter: You said the M word! So coming back to, sort of continuous delivery of every part of the stack, yeah, it’s interesting, I mean we’ve had some interesting challenges with databases in particular and I actually gave a talk at Heavybit a while ago about, kind of, how we treat databases less as part of a service in sort of the classical sense but more like a factory.
So basically what happens is when people request databases from us, we stamp one out and then, you know, we try and keep it pretty low entropy from that point on so we have a lot of tooling and support services around the database but the actual database itself, once it’s given to a customer, we mostly try and keep our hands off it.
Now if there’s security issues or performance issues or whatever, we’ll go in and we’ll kind of warm it up and do some work on it. But even still we mostly prefer to actually replace the database. What we do is we create a replica, bring it up to speed and then we transition the load over to it. And that’s kind of a form of continuous delivery for data services, which has been surprisingly powerful as an abstraction.
Edith: So you’re basically, you’re treating them as cattle, not pets.
Peter:
Oh absolutely, it’s all about the cattle and model and when you’re farming a very large crew of livestock, you can’t really afford to give each one their own personality.
Paul: This reminds me a lot of, I think, what started out as an IMVU way of doing data migrations, I’m sorry, of doing schema migrations without locking a whole table. For our people at home, I’ll describe this.
So when you want to add, let’s say you want to add a new value and it has a default to a particular table in your schema. There’s a possibility of that if it’s a particularly large table, for it to be locked while that happens. At least this used to be the way, I think modern databases are much better at this now.
Peter: I believe for some default values, it’s now the case that you don’t need to do a table lock but that might be pushed until 9.6 or Postgres, at least but it’s a common and challenging operation.
Paul: And so what IMVU started doing was instead of adding to a schema, they said that for every table, that schema is immutable and instead they created a new table and then they migrated the data from that table so every time you want to read from the user, you’re reading from user table 38 at the moment and then you start to read from user table 39. If the user isn’t in user table 39,
Peter: You fall back to 38.
Paul: fall back to 38, find it, migrate it, write into 39 and then do the thing. Then you have a background process.
Peter: If you have any listeners who are listening from the world of the Internet and would like to try implementing this technique for themselves, Postgres has something called a schema search path so you can actually, basically make each schema its own version and then you have a search path so if the table wasn’t in the newest version, it would use the last version’s search path.
Paul: Oh, wow.
Peter: You could set a version on your code then basically, as you make new versions of the tables and the new schema, you can kind of move them forward. This actually sounds like it would probably work. Now it’s really easy to say that from sitting around a table here in beautiful San Francisco but you have any listeners out there who do give this a try, have them send me a note.
Paul: I was not expecting this to be a session about continuous delivery databases and the intricacies of modern versions of Postgres.
Peter: I have to say, it’s one of the hardest things is basically, how do you deal with your data? I mean, data is sort of fundamentally heavy, I mean I think about it as having mass, which is to say, if you’ve got a few terabytes of data on a disk somewhere, just getting it out of that server and into another server is gonna take hours.
Paul: Right.
Edith: Right, just querying the data, loading it all into memory, that’s gonna take time and so this is really a challenge for continuous delivery and people have tried all kinds of schema versioning and replication strategies and sharding strategies but, you know, the old question, okay you deploy a new version of your code and it needs the new column but the new column’s not there yet. Okay so you move the data to the new column and then you deploy your code but then the old code doesn’t have it.
Paul: Right, right.
Peter:
And these are hard problems and no one’s really done a great job of answering them. So far, sort of the state of the art is, you know, you continuously deploy the new version of your code that supports both schemas then you continuously migrate your database and then you deploy yet another version.
Now without continuous deploy this is even harder—
Paul: This has to be tough on a Gameboy cartridge.
Peter: Oh yeah, let me tell you, I think we had 4k of RAM or something like that so at least it was fast. But yeah I think data and continuous delivery are both challenging interactions for sure.
Paul: We end up at this problem in the testing world when large customers, in particular, large enterprises use the old way of testing, which is you actually use a dump of your production database as part of your test suite and one of the major challenges we get when we’re looking at older companies who are coming to the new world, is, you know, your server’s over here, my server’s over there and I want to get a 5 Gig database dump onto your server to run my tests, how do I do that?
Peter: Right and it’s trying to, like, move a bathtub through a straw. So how do you do that? You just wait, right?
Paul: We wait until they stop doing that and then they can become our customers.
Edith: So do you turn customers away then?
Paul: We don’t turn them away but we don’t really support that use case, they’re going to have a miserable experience.
Peter: I mean there’s data migration again, right, just coming back to the physics problems. There are things that are getting better so,
Paul: The speed of light is improving?
Peter: the speed of light has gone up 13 percent in the last year, did you know that?
Paul: I don’t know if you’re joking, I presume you’re not.
Peter: It absolutely has not, yeah. They call it a constant for a reason. I suppose it could.
Paul: I thought there might be something about, like,
Peter: Dark matter, no but it was very convincing the way I said it, wasn’t it?
Edith: Like this is Hindenburg’s Principle?
Peter: It’s uncertain. So anyway, the logical replication scene is getting better. Amazon’s just launched a new service, it’s three letters long.
Paul: Oh really?
Peter: Yeah, uh, got it, hang on. DMS, Data Migration Service. And so through black magic and probably by not working very well, it will move all your data from one database to another in sort of a streaming online way which is really cool if it works.
Paul: Okay.
Peter: But, you know, moving from Oracle to MySQL
Paul: So you can talk to this service as if it’s MySQL and it will pull it from—
Peter: The way it works is you basically set up a migration job and if you think about it, it basically queries one database and writes to another and it’s a little bit of a hack job in the sense that you have to run, like, a desktop client if you’re moving from a local machine. It’s all very complicated and kind of baroque but,
Paul: So it’s an Amazon service? It isn’t you’ve used Amazon before, I gather.
Peter: But yeah, no, it’s what everybody’s always wanted which is to move their data from one place to another without having to think about it and, of course, it’s never gonna work.
Paul: Okay.
Peter: But, you know, it’ll help, it can help, things could be less terrible. I think ultimately that’s what got me into databases was the understanding that no matter how impossible it would be to make things good, at least we could make them a little bit less bad.
Paul: Right.
Paul: The soul-crushing misery of writing software could at least be slightly less crushy.
Edith: I feel like writing software is one of the greatest joys in life because you get to create something from nothing.
Peter: So you don’t work in databases.
Paul: I’m not sure you work in software.
Edith: So what are the biggest changes you’ve seen in continuous delivery?
Peter: Oh, ubiquity, certainly. It’s rare to meet someone these days that, maybe I’m just lucky enough, I live in San Francisco.
Edith: You hang out at Heavybit.
Peter: And I hang out at Heavybit with a bunch of other crazy people but I guess I would say growth then, right, I mean we’re seeing large enterprises practicing continuous delivery—we’re seeing small shops practicing continuous delivery, it’s not, sort of, a bleeding-edge concept that only, sort of, a small cadre of, like, sort of, avant-garde developers are doing. It’s really, I think it’s kind of the mainstream.
Listen to full podcast here to hear Peter talk about continuous delivery in the context of organizational structure, databases and pricing.
-

To Be Continuous w/Special Guest Kris Gale: Effective Team Structure for Continuous Delivery
In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk with Kris Gale about effective team structure for continuous delivery. This podcast is brought to you by Heavybit, a 9-month program for developer startups.
Kris is co-founder and CTO of Clover Health, a unique health insurance plan focused on driving down costs and producing improved health outcomes. Formerly, Kris was VP of Engineering at Yammer.
Edith is CEO and co-founder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.
Paul is founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.
[soundcloud url=”https://api.soundcloud.com/tracks/232834454?secret_token=s-3aRuL” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]
Edith: So, Kris, what do you like best about continuous delivery?
Kris: So, I like continuous delivery, because I think it’s sort of the first step of building a learning organization. It’s sort of a fundamental capability that a team needs to be able to learn collectively about what it’s doing in the world. The first step is being able to sort of ship increments of value continuously.
The second, I think, is to be able to measure the result of any incremental change you’ve made, so you can back out of a bad path that you’re on, or sort of refine what you’re doing and know that quantitatively.
The next step then, is to move from a sort of historical corporate planning, so this is a our six-month milestone, our two-year roadmap, our marketing commitments for Q3 of next year, into sort of an assumption that if we are shipping continuously and learning continuously because we’re measuring what we’re doing, whether it’s quantitative or qualitative measurements.
The next step is really being able to assume that in your planning, and know you’re going to have more information in the future than you had in the past. So your plans that you made six months ago are necessarily lacking information than plans you would make today would have, so sort of institutionalize that.
And then kind of the last part, and this is all kind of enabled by continuous delivery, the last part is building organizational fluidity.
So as an organization, whether it’s your project planning structure, or your accountability to your work flows, are fluid, so if you learn something new, you’re not locked into, oh we have a sixth person, I don’t know, sign-up flow team, and we just discovered that we have no levers for moving engagement right now in the sign-up flows, so we’re committed to that and we’ve got this implicit prioritization that’s built into our org chart. So, to have organizational fluidity where you’re adapting the organization to what you’re learning continuously.
Edith: Oh. You know that’s fascinating. Paul and I have talked a lot about, I think, more the mechanics of continuous delivery. We put it as kind of the last step to software org, when you’re really talking about how that changes entire organization beyond software. So, this would be a great time for you to introduce yourself, and tell us more about you.
Kris: Yeah, so I’m Kris Gale. Most recently, I was the VP of Engineering at Yammer. I heard you talking about Yammer in a previous episode. I’d love to dig into any of that, if you’ve got questions. Currently, though, I’m the co-founder of a company called Clover Health. So, what we’re actually doing is offering health insurance for seniors, with the idea that we can fill gaps in care, and actually prevent hospitalizations.
We’re not trying to be people’s doctors, but we’re trying to look across their consumption of health care and say, “Oh, here’s an opportunity ‘to inflect the probability with which ‘this population of people will be admitted to the hospital.” So, for example, making sure they get treatments regularly, get follow-up visits with specialists, that particular bio-markers don’t deviate outside of a specific range. And all this stuff is really interesting to me, because the health care industry as a whole doesn’t change what it does continuously.
We’re sort of locked into what people learned in medical school, or what people learned on their first day on the job at, say a big health insurance company, or something like that. But, to be able to power workflow with software, you’re able to continuously iterate and continuously refine and optimize the intervention protocols that are leading to these prevented diseases.
Edith: Yeah, I mean, I heard that you were pretty good at data loops, and that this is just basically using data loops for good, I think is how you say it.
Kris: See, the way we were talking about data loops, we were actually talking a little bit before we walked up here, is this idea that there’s something you intend to do. There’s data that you collect in doing it. There’s your objective assessment of that data to determine whether you’ve added or removed value or improved something.
And then closing the feedback loop, to get back now to actually modifying what you’re doing, and form find out new data. So, that’s sort of the data feedback loop, is from doing something to measuring it and deciding to do something differently the next time.
The tighter you can close that feedback loop, the more opportunities you have to optimize in a short amount of time.
Paul: So, it’s kind of what you might call the shipping loop, if you implicitly believe that data is a key part of shipping.
Kris: Yeah, yeah, and I would say that I think I do.
Edith: No arguments here.
Paul: Well, I think it is interesting, because I think that a lot of people don’t actually have measurements as part of their feedback loop, and especially, I think this especially happens in a small organization, where you have a start-up that sort of comes out of a founder’s vision, or something like that.
And so, one of the questions that I’m very interested about is, you were doing a big thing, with lots and lots of people. And now you’re doing a very small thing with, I presume, a small handful of people. How much of what you’ve learned still applies at that very small scale?
Kris:
So, yeah, a lot of the project workflow strategy that we came up at Yammer was really about how do you maintain that agility, that velocity of value delivery, and ability to react to new information, and not be locked into, “Well, this is just the way we do it,” in the face of new information. At scale.
So, Yammer’s whole process was really about, how do you take these advantages we had, when there were like four or five of us in the really early days, and not have that big bureaucratic company, “Oh, we know we should be doing something better, “but we’re not,” because of momentum, inertia, bureaucracy, all these things. Like, how to break that. So, what’s cool about starting over with a much smaller team is, we’re just there again. We’re back at that five person size.
Edith: You’re a lot bigger than five people.
Kris: We are now, yeah. So, like nine months ago, we were. We’re like 65 people now, already. It keeps growing really, really fast. It’s been really helpful to me to have been through the process of scaling a software engineering team and seeing what that looks like. So, it’s like, “Oh, I’ve seen this problem before, “not worried about this.”
Paul: So, when you started, did you immediately set out organizational goals, and mission, and values, and that sort of thing that would help the company scale it?
Kris: Yeah, no, that’s a great question. I’m obviously a really big believer in the way we approached this problem space at Yammer, but I didn’t want anything to be dogmatic. I didn’t want anything to be–
Paul: You wanted it to grow organically.
Kris: Yeah, I wanted it to grow organically, to a certain point, but there have definitely been times where I stepped in and I said, “Nope, I’m buying a physical board “to track our projects, “because, just the way we’ve been doing this “with sticky notes, “I know that it needs this other dimension “of information capture. “I know that we’re suffering because it’s not there.
“So, I’m gonna actually pull the boss card and do this.” There have only been like three or four times I’ve done that in the last year, but it’s things where, like– Okay, I’ve seen where this goes off the rails, if we don’t address this right now.
Paul: And is that a result of knowing, sort of directionally, how the company should be doing things, or setting things up in some way that you didn’t get, that you didn’t end up in these bad places?
Or was it all sort of retroactive, and you know, a couple places where you pulled the boss card?
Kris: Yeah, it was mostly, like, observing patterns, saying, “Okay, we made this mistake three or four times, “and I’m noticing that our process or our approach here “is divergent from a known good approach “that I’m aware of.
“So, let’s at least–” I still wanna give the team full empowerment to iterate and own the process themselves, but let’s at least just pull the boss card, get it back on the rails, and then say, “Okay, let’s see how this works.” Just give everybody a chance to work in a way that I know to be okay for a few iterations, and then give them the keys again, and say, “Okay, take this where you want.”
Edith: Also, Kris, what I’m dying to know now is, what are those three or four key things where you gotta play that card?
Kris: Yeah, so, one is single project focus. Really, really important to me for software engineering teams. So, what I mean by that is, if you are on a project, that is your only accountability. You are not being split between three deliverables.
The other is that everybody work in cross-functional teams. So, that means not giving, for example, I might tell you, “Build out the back end for this feature, and I’ll do the front end.” So, in order for us to do that, we would have to agree to the abstractions ahead of time.
And again, I’m pretty opposed to planning, and locking in planning, because the information that we have now, when we’re trying to lock in those abstractions, is going to necessarily be incomplete.
Edith: I love that quote, because it’s so true. I think you put it more, it’s like you always know more in the future. It seems so obvious.
Kris: Yeah, I was actually– So, we had Parker from Pivotal Labs over at the office yesterday, and he said that there was this saying they had a Pivotal, that I really, really liked. He said somebody there would always say, and I wish I could remember who it was, would always say,
“You will always make better decisions with more information, and you will always have more information in the future.”
This is so much more concise than any way I’ve ever said that. But I think it exactly captures it.
Edith:
But, on the other hand, my old boss, Gregg Brockway the founder of TripIt, he always said, “A good enough plan today is far superior to a perfect plan tomorrow.”
Kris: Absolutely.
Edith: He’s like, “You just gotta start moving.”
Kris: Yeah, but I think the key is not locking yourself into that “good enough” plan. To allow yourself the mutability of changing that plan over time, so you’re not constrained by that.
Edith: Yeah, but that was TripIt. I mean, I think he’d seen so many, because he came out of the travel industry, where people just got paralyzed. Like completely paralyzed. “Oh, we need to plan for five years, “and what business travel will be like then.” It’s like, well, nobody knows.
Paul: I think the kind of key point to that is that in order to get that information, you have to get moving today with whatever information you have available. Otherwise, you’re just not gonna have that future information. You’re gonna be in the same place.
Edith: Well, look, sometimes, though, it’s painful. Sometimes you move backwards, then. Like, you don’t always move forward. Sometimes you start moving, and when you’re looking back, you’re like, “Actually we took, like, three steps back “before we started moving forward.”
Kris: It can definitely happen.
So to complete that thought, when we assign teams, we would always assign a cross-functional team with a business objective. So, it would never be I give you a back-end task, and I take the front-end task. It would be collectively, let’s accomplish this business goal.
So hopefully, we’re iterating day by day on the abstractions. So, we are collectively accountable to one another, and to this business goal we’re trying to deliver. Not onto some specific set of tasks that we broke off when we first decided to assign out the work.
So, that’s one of the other ones, is always cross-functional teams. Always single project focus within those.
Paul: So, there’s an interesting thing that you’re talking about there. It’s not just cross-functional teams, but it’s also that those cross-functional teams get put together for a specific project, is that right?
Kris: That’s absolutely right.
Paul: So, are those teams teams that will have worked together in the past? Is there explicitly trying to make sure that the people know how to work together?
Kris: Yeah, so this is one of my other core principles that I am pretty, pretty set on, which is rotation, is this idea that cross-functional teams are rotational and ephemeral. So, that would mean that maybe the three of us come together to ship a feature.
We’re all within the same organization, but when we’re done shipping that feature, we’re gonna break up.
You might be the back-end person that goes onto another iteration of this feature area, or maybe, you know, the two of us work on some other feature. Maybe it’s three completely different people. But constantly rotating people around, I think is really important in an organization.
I have totally heard the arguments about, you know, you can build high trust teams if they’re stable and stay together. But I’m trying to optimize for trust across the entire organization. So, across Yammer, we had 220 people in engineering. They would all rotate through these two to 10-person cross-functional teams together.
Which meant that you, basically within three to six months, would work with a pretty significant portion of those 220 people, and would start to build trust across the larger organization. Whereas had we just torn off four people and said, “You’re a team,” they would– And I’ve seen this throughout my career in engineering, where it’s like, “We do everything right, and that team has no idea what they’re doing.”
Edith: It’s their API’s are bad.
Kris: Yeah, yeah they use this weird language, or this approach, or we use this style guide, or whatever. So, it sort of forced that rotation, so that we’re all sharing best practices among one another and building, maybe lower trust, maybe not getting to that very, very, very highest level of trust among a group of four people, but across the entire organization, getting to like 40% or 60% of what’s possible versus 0% if you don’t ever work with those people.
Paul: So, interested in structurally how that ends up, because one of the things you got when you split people up by team is you end up with a sort of microservices architecture.
And this team works on this repo, or this project, or whatever works on this service or this API. And when you have those teams that are forming and splitting up and they’re ephemeral, and it’s only a small period of time, architecturally, what does that do to your application or your architecture, you know, across your entire product?
Kris: Yeah, no, totally. It was actually very intentional that we ended up with that kind of architecture for working this way. For two years, we had a big monolithic Ruby on Rails code base, and for two years we said, “This is a big problem. We should break this up.”
And everyone agreed, and everybody was on board with breaking it up, and two years later, we had never done it, because we continued to work in such a way that supported that sort of architecture.
So, the key for us was to say, well one of, the– So, something to dig into here a little bit, is these cross-functional teams are made up of representatives of what we call functional teams. So, front-end engineering might be one.
Paul: Right, right.
Kris: We had one called Core Services, which I actually owned the service-oriented architecture that you’re describing. Or there’s little micro-services that collectively made up our service-oriented architecture.
Paul: And this is sort of a matrix style, where you’re a member of the team, but you’re also a member of the back-end team, or you’re a member of the front-end, or the core services, or whatever.
Kris: Yeah, so I might be a member of core services. My manager is one of the core services managers. But all the work I do is going to be within the context of the cross-functional team, with the front-end engineer, maybe a client’s engineer, whoever’s necessary to get end-to-end with a value proposition.
And then, I’m always going to be a core services member, when I go into these cross-functional things, but it’s the fact that all the members belong to functional teams that makes that team cross-functional.
For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.
