As longtime readers of my blogs will know, one of my hobbies—when not writing about or playing with technology—is modeling. There is a similarity between quality tech work and quality modeling, but the brain power involved in modeling is much less, leaving me time to think about tech issues or listen to audiobooks (when I’m not listening to music, anyway).
In modeling, as in tech, there is a process. The process varies based upon what you’re working on, but is largely the same, with some steps slightly different or skipped altogether and others taking longer. At the moment, I’m building the first day of Gettysburg from the American Civil War, but I simply enjoy modeling—and wargaming with the models, but that’s a separate story for a hobby-focused blog.
The thing is, that process is nailed down, and to the extent it can be, is automated. One step leads reasonably to the next, and while one step is completing on a given model (glue or paint drying, etc.), there is an opportunity to advance a separate project.
Sounds a lot like DevOps, doesn’t it?
The thing is, every once in a while, you will get something that is so different, it totally interferes with the carefully planned flow. My current primary project is a little over 300 figures. While one unit is getting the final two coats of finish (that are basically the same or every model), the next unit is getting primed. Assembly of all the figures came first, so there is only the priming step before painting can commence.
Piper Millin and Lord Lovat cross Pegasus Bridge.
But here is a work-in-progress picture of Pegasus Bridge. This came as a complete set, and took a large chunk of time to get together, find the correct water for, get placed on foundation boards, etc. This is outside the normal process of assemble/paint/base figures and vehicles. The set is beautiful, and even this picture, which was taken well before completion and only to study color and weathering effects versus reality, shows how truly nice the model was. But it totally disrupted the process flow.
DevOps and Multisystem Installations
That’s what complex system installations such as Hadoop or OpenStack do. You’re installing across multiple systems, configuring them for load-sharing and communication, and doing it by hand is a long process that interrupts the normal DevOps flow. Crafting scripts to install them is possible but highly complex and frustrating, simply because of the complexity and interrelationship of the nodes in the system.
Having installed both by hand, and installed both with tools designed to make it easier, I will happily tell you that yes, you will understand more if you do it by hand, and no, that immediate understanding is not necessarily worth the cost of weeks or months figuring it out and getting it installed.
You see, to operate these systems, you will need much of the knowledge you would gain hand-installing, so you’re going to learn it either way. Your choice is a simple one: Get a working system and learn by tweaking settings or swapping nodes in and out, or have no working system and learn by the frustrating process of trying to figure out what you’re supposed to do next, and why that step is failing for your specific hardware.
DevOps and Multisystem Maintenance
The only real problem is that it is painful to maintain these systems. It’s not as simple as
yum update [[packagename]]
because that would only upgrade one package on one system. But these tools span multiple systems, and even just dependencies generates a massive list for each system in the cluster. Seeing the need, most of the traditional application provisioning tools (Puppet, Ansible, et al) have built or cobbled together solutions that will get the job done. But custom solutions exist for these products, too. The problem is that, like Pegasus Bridge, as beautiful as they are they disrupt the DevOps flow. No system goes without exceptions, and I’ll go out on a limb and say that custom solutions to install and maintain solutions such as Hadoop and OpenStack are worth having yet another tool in the DevOps tool chain. The trend is multipurpose tools to be used on many projects, but the importance of a cloud or big data cluster in your IT operations makes them worth the extra tool. As long as the extra tool reduces the amount of work and the chances for human error (which, in my opinion they do—it is kind of their selling point).
That Leaves Time
These tools (a good example, though in general I won’t be talking about the competing vendors here, is the Mitaka-based release of Mirantis) allow not just installation but also ongoing management of the OpenStack cloud or big data cluster. That means there is less nitty-gritty for DevOps (or operations for that matter) teams to worry about, freeing time for working on what actually gets built on top of these powerful platforms. Cloud and big data both are just enablers for more functionality when looked at in the right light, which means more work (and often more meaningful work) for DevOps teams.
When you consider either of these technologies, check out what’s available for installation and maintenance, even if it is outside your normal DevOps tool chain. Like Pegasus Bridge, the change is likely to disrupt you for a bit, but once you have a tool and process in place, it will just be something that happened—and was going to happen with or without the tools. They just make it less painful.
While plenty of vendors tell us what they think is happening in the market, the only way to really learn is from you, our readers. So we are asking you to take a few minutes to take this important survey on the adoption of CI/CD, DevOps and job scheduling.
We will be giving a $50 Amazon gift card to one out of every 50 respondents. More importantly we will be publishing the results here on DevOps.com to share with everyone. The survey is sponsored by BMC and co-produced with DevOps.com.
Is it Strong DevOps? DevOps Security? Secure DevOps? DevSecOps? DevSecQAMktSalesOMGBBQOps?
Who cares?
Like a ton of people with an interest in information security, I was appalled when I read “The Phoenix Project.” Here was a book that portrayed DevOps, why it was required, how it could help and the benefits very well. But it served security terribly. Worse than terribly. While it did accurately portray the predominant feelings of developers and Operations toward security, it utterly failed to adequately consider the correct usage of DevOps to attain InfoSec goals. Indeed, while it is a light treatment, it certainly comes across as, “Give up on InfoSec and you’ll be happier, healthier and more popular!”
Clearly it was not a book written by information security staff.
The Point Is …
Don’t get me wrong, those who follow me know that I’m a developer that learned operations and has always found himself doing security bits, because no one else was, or because I was teamed with them (as an enterprise architect, our team was EA + Infosec, so we did a lot together). But that abiding interest in InfoSec has made it easy for me to see that the bulk of the problem between Security, Dev and Operations really is communication. An example might be in order.
When SQL injection first hit the scene in a big way, it came with simple examples and was easy (for a developer) to understand the danger. Nearly immediately, developers had worked up solutions, were sharing them and tools were being developed to scan for injection weaknesses. Things moved very quickly, because developers understood the danger and agreed that it was a big deal.
Meanwhile, using page data—hidden fields, hard-coded values (like from a list), etc.—pose the same level of risk. Copying the page, modifying those values and submitting the page—or using any of a dozen easy to use tools to mock up the data—makes them equivalent to user input, particularly in cases where they are used directly in queries. Yet even today, many many developers still use them without user-input error checking.
Why? Because they don’t believe the risk is equivalent. For SQL injection, all a bad actor has to do is hit a vulnerable webpage and enter the right (and, I might add, well-documented) characters to open themselves to free querying of the database—at least one table, possibly much more. They see that as a much greater risk than someone having to modify HTML to get the right field to have the correct data and submit it.
The fact that both attackers and tools are advanced enough to make the difference negligible doesn’t change the belief unless security convinces developers through education.
‘Faster’ has Challenges
By the same token, both agile and DevOps move fast. They are designed to move fast. Security is more concerned with “protected” than fast, and is designed that way. Security staff (and this is the real source of a lot of burnout) hires on with “Protect Our Systems” as a job title, with the subnote, “But don’t get in the way.” This is a set of core principals that are at odds before the InfoSec person even sits down to interview. Agile and DevOps aggravate this tension, because, “Wait, there is a problem here, give us until tomorrow to get details,” now violates “but don’t get in the way.” “The developers in question have a stand-up coming and really short, really tight deadlines to meet. And InfoSec is interfering—again,” will likely be added to that statement by Dev. The same is true in Ops. Stopping an entire deployment because InfoSec is worried about security of private keys inside the private network violates the “but don’t get in the way” rule.
But Solutions are Available
So how can these tensions be brought together to resolve your problems? I’ve got some suggestions (they probably aren’t in the order you would expect, so I’ll explain as I go), but honestly they are just a starting point. Entire books have been written on these topics, and more are about to be, if the state of the DevOps/InfoSec union is any indication. We can’t even agree what to call it, so we clearly aren’t agreed as to how to achieve it. Take these as what they are—suggestions—and start moving forward with what your org needs to be successful.
Automate all of the things. Yep. Go download the various OWASP security tools, or your other favorite open-source tool, or call up your preferred vendor and get the automation of security checking—from source code analysis to penetration testing—going. Do those tools have weaknesses when compared to manual security evaluation? Most of them, yes. But here’s the thing. There is a quality staffing shortage in both InfoSec and DevOps. It is time to lean a little more on those tools, even contribute to make them better. Because the security checking they can do is better than nothing, and frankly is better than spending limited resources checking by hand. And that is why this step is first on my list. You need those resources. You need them to handle the rest of the steps. Once you have a tool like Jenkins auto-running these tools, and output review being just part of the job, then you can start to consider using those freed up resources more productively, or reporting that you’ve got more testing with DevOps than you had, whichever is true.
Embrace and enhance education. Because of the rate of movement in an agile-plus-DevOps world, this must become a partnership with both Dev and Ops. Focus on helping developers and admins with the why as much as the how. People are more quick to adapt if they understand the reason for doing so. And for InfoSec to keep pace, they will need to adapt quickly.
Look for ways to automate beyond implementing simple tools and reviewing results. The more repetitive work is built into the process, the more InfoSec is available to tackle bigger, less scriptable problem-solving. I’ve been talking a lot with CloudCoreo, for example, and its auditing tools can help InfoSec understand the state of cloud deployments with both alerting and reporting. Infrastructure monitoring tools make a lot of sense in a world of frequent infrastructure changes initiated by both auto-scaling and more frequent deployments.
Shift automation of InfoSec to the DevOps team. Security is normally a relatively small function, and open positions can sit there or bring in people who need training in an organization’s tools and processes. So once automation is in place and reliable, shift responsibility to maintaining it (software updates, etc.) to the DevOps team, because it is a part of the DevOps process. Security can continue to monitor output and validate automated security testing results, but maintaining that piece of the DevOps infrastructure should not be InfoSec’s job.
Rank what’s left that’s manual. Give each an importance. If there is potential impact to the business, ask business people to help you. Then assign work based on severity rankings. We do this stuff with risk management and threat assessment; this is no different, except you have to be willing to drop low-priority things—not to let them slide because you never get to them, but to publicly say, “This is a risk we are willing to take.” Let the team know that they’re not the only ones adapting.
Will these basic starting steps solve all of InfoSec’s problems and make security never seem to “get in the way”? Of course not. Don’t aim for hanging around the campfire singing “Kumbayah”; aim for working as a team to turn out quality secure compliant code in a more predictable (and if you are an org that needs it, much faster) manner.
A Rose by Any Other Name
What, then, we should call it?
Frankly, I suggest we call it DevOps. We’re not going to add every team that is subsumed into the DevOps umbrella into the name, so just call it DevOps and have a security piece in the process.
Because it really doesn’t matter what you call it—it matters that it is compliant and protected.
Collaboration is part of the foundation of developing code in a DevOps environment. The ability to share code or leverage snippets from open-source projects enables developers to be more productive and efficient—creating better software, faster. Sharing code with others doesn’t necessarily mean you want it exposed to the public, though, which is why Atlassian recently introduced some new security controls for Bitbucket.
A platform like Bitbucket has a number of advantages in terms of collaboration and efficiency, but many organizations are reluctant (if not adamantly opposed) to putting potentially sensitive code out in the cloud. Atlassian added IP whitelisting and two-step verification to help organizations limit access to code. The IP whitelisting allows customers to restrict access to code to only designated IPs, while the two-step verification ensures it will take more than a simple password compromise for an unauthorized user to gain access to code.
Security policies obviously vary from company to company and industry to industry, but some organizations have a strict policy against working from home or require specific security controls or a secure VPN connection to allow users to connect. The IP whitelisting feature of the Bitbucket Premium plan gives companies the ability to designate which IPs can connect and limit access.
“For Limpid Logic customers, remote access and IP whitelisting are sometimes a legal requirement, especially for clients in highly regulated industries such as finance and health care. Our work often deals with sensitive intellectual property that requires limited geographic access to repos from a few specific IPs,” said Bachir El Khoury, managing director at Limpid Logic. “IP whitelisting is exactly what we need within our business and we’re thrilled to see this security feature in Bitbucket.”
The second new security feature is two-step verification. The problem with the traditional access control of a username and password is that it is easily compromised. Usernames often follow a simple and predictable pattern, and passwords have been shown in data breach after data breach to be a poor security control. They’re often far too easy to guess or crack, and even “strong” passwords are not invulnerable. That’s where a second layer of authentication can help.
Atlassian introduced the option for administrators to require two-step verification in Bitbucket. A Bitbucket blog post explains, “When you enable this option for your team, users will need to have two-step verification enabled in order to interact (view, push, clone, etc.) with your account’s private content: repositories, team settings, issue trackers, wikis, and snippets. If a user doesn’t have two-step verification enabled at the time of access, they’ll see instructions on how to enable two-step verification in the UI and continue.”
Both of the new security features are part of the Bitbucket Premium plan. The features are available as a free trial pending upcoming price changes. Once the changes are implemented, the cost for Bitbucket Premium will be $5 per user per month. You can sign up for Bitbucket here, and learn more about how to configure and use IP whitelisting and two-step verification here.
As adult learners and leaders, we tend to glean less from theory and seem to derive more from people actually describing the real-world problems they are trying to solve. Ever since the DevOps Enterprise Summit began in 2014, we’ve been adamant about providing a unique venue for the technology leaders of large, complex organizations to come together and share their DevOps transformation experiences—from how they started and what they did, to what the outcomes were and what they learned.
On the second video chat gearing up for DevOps Enterprise Summit London 2017, programming committee members Cornelia Davis (Pivotal), Jonathan Fletcher (Hiscox) and Jonathan Smart (Barclays) joined Gene Kim and spent an hour taking a “deep dive into DevOps,” to reflect on the first DevOps Enterprise Summit London last year. Panelists also shared some of their lessons learned and wisdom about their own DevOps transformation stories today and shared some of their favorite resources for leveraging best practices and principles while discussing the value of lifelong learning.
Just a reminder that Early Bird ticket pricing for DevOps Enterprise Summit London 2017 was extended to 22 March and the price is only £650+VAT. Register here >>>
All in all, it was an excellent conversation with Jonathan Smart, Cornelia Davis, Jonathan Fletcher, and Gene Kim as we all inch closer to the DevOps Enterprise Summit London this June. The goal (according to each of these people who are now on the programming committee for London) is to hopefully make the show even better than it was last year. We look forward to seeing all of you June 5-6 at the QEII Conference Centre.
Here is the video and a full transcript of the conversation below that. More links to DOES videos and interviews are down at the bottom. See you London June 5-6!
Gene: Summit 2017 in London. This conference is taking place on June 5th and June 6th. For more information on this, just go to events.itrevolution.com. I think what excites me so much about the DevOps Enterprise Summits is that there is just, no doubt in my mind, that the people who are presenting are the leaders and practitioners, who are pioneering how we’re managing and using technology in large, complex organizations. So on this Live Chat, we have Jon Smart from Barclays. We have Jonathan Fletcher from Hiscox, Cornelia Davis from Pivotal who’s also…In fact everybody is on the programming committee.
So my goal next hour is to have each one of you share what you’ve been working on, share some of your lessons learned and wisdom which is just, obviously, a fraction of what you’ll be presenting at the DevOps Enterprise Summit. So just one last reminder is that there’s an early bird ticket pricing that ends on the 8th of March, which is one week from today. So the next step is really to share what wonderful speakers that we have on the panel today. So let’s go around the table, and if each one of you could present your name, your role, company and, specifically what you presented on last year. Cornelia, why don’t we start with you?
Cornelia: Sure. Good morning, good afternoon, everyone, and, hi, Gene, and my fellow panelists. My name is Cornelia Davis. I am the Senior Director of Technology at Pivotal, and so Pivotal is a company that maybe is most well known for incubating the Open Source Cloud Foundry Project, and my role is that I work in the product organization. I work with clients and customers to take our products and use them effectively. Well, it turns out that our products are really disruptive.
It’s very different architecture and so they have to change everything. So that’s where DevOps and digital transformation comes in, is they actually have to in order to use our products effectively, they have to change organizational structures and you have to change processes. And I have this tremendous opportunity to work with very large enterprises across the globe, in then figuring out how that works for them.
Gene: It often can be very disruptive to the way we’ve done the thing for decades, absolutely.
Cornelia: Oh, yes.
Gene: Jonathan Fletcher, why don’t we start with you? I wanna go to you next and, by the way, congratulations. I hope you’re gonna tell us about your promotion, as well.
Jonathan: Yeah, so I’m Jon Fletcher. I am the interim CTO for Hiscox Insurance. Based out of offices around the world. I’ve been the CTO for a month, five minutes, it seems. So, it’s fairly new.
[Crosstalk]
Gene: Can we stand by a second? We’re having some audio issues. Let’s go to Jon Smart while we resolve that. Jon?
Jon: Hi, Jonathan Smart. So I work at Barclays. My role is Head of Development Services and I’m leading on better products faster, so better ways of working, adoption of agile dev ops, lean principles, and practices across Barclays.
Gene: Brilliant, and what did you present on last year?
Jon: Last year, I presented an Experience Report. So I gave an update of the Barclays journey across the whole organization. We started two years and two months ago, beginning in 2015, so last year, I gave an Experience Report, shared some of the lessons we’ve learned in terms of transforming a large organization. We are 130,000 people in 40 countries. We have 48 million customers.
We’ve got some main business lines such as the investment bank, retail bank, wealth, corporate. So we’re a very large organization, as I said. We’re 327-years-old, founded in 1690. Four years before the Bank of England existed so that’s a lot of legacy to change and it’s a large organization. So I shared our journey so far, and I’m looking forward to sharing an update this year, as well.
Gene: Brilliant. And I think one of the things that just left such an impression on me was just the scope of what you’ve been overseeing in terms of how many engineers and how many applications?
Jon: It’s 26,000 staff in technology and over 3000 applications.
Gene: That’s awesome, fantastic. Thank you, Jon. All right, Jonathan Fletcher, let’s go to you. Name, company role and what you presented last year, and you’re gonna to tell us about your promotion. And then we’ve gotta unmute.
Jonathan: I am the interim CTO of Hiscox Insurance. I’ve been in that position for about a month. So I’m not exactly seasoned, but I’m doing it so far. I’m really excited to be part of this and speaking in DOES in June. So I’m looking forward to that.
Gene: Awesome. And then you presented last year?
Jonathan: I did, yes. Sometimes it’s on our move to DevOps and a bit of my personal journey starting as a solution architect four or five years ago, and how I got involved, and how the whole thing started to happen, and I hope this year to carry on talking about that journey and where we are now.
Gene: Very good. Thank you. So why don’t we go to a little bit about the conference and your impressions on it? You know one of the things that many people have told me over the years is that DevOps Enterprise Summit is structured differently than most typical technology conferences that we go to. And I think many of us are, actually almost professional conference attendees. Would you mind sharing your impressions of your two days at the DevOps Enterprise Summit, London, last year? Jon, why don’t we start with you?
Jon: So what I really like about it is that it’s peer-based, so it’s people who are doing the job of transforming organizations and that’s what I like about it. It’s people sharing case studies, sharing their real experience so, you’re learning firsthand. It isn’t a series of sales pitches. It’s about people who are doing the job. Specifically in the horse is not the unicorns. I’m shy of horses, rather than the unicorns. People who are not born Agile, but are born Waterfall. So that’s what I like about it.
It has great networking opportunities, as well, with people who are going through the same journey. And also it’s interesting when you sit through a number of the case studies, you start to see the common themes emerging, and pretty much every single time, the main theme that emerges is it’s all about people. And often with DevOps people jump straight to the tooling and they don’t think about people and culture, often, but it’s all about people. So it’s great when you see these themes coming across from many different companies.
Gene: That’s awesome, and I’ll maybe just comment on that. I mean I think that’s one of the reasons that we choose very deliberately to have people share Experience Reports. Just because as adult learners, as leaders, we tend to learn less from theory. Instead, we learn from people actually describing the problem they’re trying to solve, what they did, where they started, what were the outcomes, and what did they learn. Jonathan, how about you?
Jonathan: I’d echo much of what Jon said, actually. So I like the fact that the very little of it is technical. So very people, focus, culture, process-based and I love being the least smartest person in the room. So it’s watching to learn from lots of other people. I think lots of DevOps conferences are very technical, although, they’re very at the beginning of the journey.
So being surrounded by other people that are at least where we are, if not further ahead. For me, it was a great learning opportunity and I can remember coming away reinvigorated about what we’re doing, and I guess hiring because you were pushing against people that, necessarily aren’t aligned to your way of thinking. So it’s just to get that we are doing the right thing, and we are going the right direction. That was really fantastic. So I really enjoyed those couple days.
Gene: Awesome. Cornelia.
Cornelia: Yeah, it’s all of those themes, but there’s some interesting…As you know, Gene, I work with you on the Programming Committee, have been in the U.S., and now this year in London, as well. I wasn’t at London last year, but we did something in the U.S. last year that we’re going to bring to London this year, and it’s really just an exemplar of what we’re talking about here. It’s a programmatic way of really fostering that peer-to-peer collaboration.
And so last year in the U.S., we did these lean coffees where we had this structured format where we got people together at round tables and had kind of a facilitated way of sharing ideas, and it was fantastic that the tables that I sat at, it was literally an hour-long period. And I think everybody at the table got three concrete takeaways of, “Oh, that’s a technique that I wanna try.” And so as you know, Gene, we’re talking about bringing that type of a format.
It’s not just conference sessions. It’s that type of a format to really foster that collaboration and that’s really very different from a lot of conferences these days.
Gene: Yeah, and I think both Jonathan and Jon will be joining the Program Committee starting this Friday. And I think one of our goals is, you know how do you create the conditions where those serendipitous interactions, how do you maximize the opportunities for the interactions that lead to learning, right? And part of that from a program perspective is we do both the talks, but also creating session formats like that. So one of the things that I’m just delighted about is that the conference will be featuring both Jonathan and Jon, again, sharing the continuation of the journey both at Hiscox and at Barclays.
What I love about this is it’s something that most conferences actually don’t…it’s actually quite unusual, right. But, for me, it seems like this incredible opportunity where we’re almost documentary filmmakers following these leaders, as they go through their own journey and we get to learn as they learn. So you’ll both be presenting for the second time. What can you tell us about what’s happened in your own respective journeys since you presented in 2016? Jonathan, why don’t we start with you about the Hiscox journey?
Jonathan: So last year, I talked about our starts with DevOps, and I guess the point of my talk was that I don’t think we’re really doing DevOps. We’re doing bits of Agile, and continuous delivery, but I think that’s really what DevOps is trying to solve. So there’s lots of stuff with testing in automation and application teams, but really I think the real nirvana is when we bring the Ops side into the equation. So in the last year, we’ve been doing a lot of stuff with cloud.
We’ve been doing a lot of stuff with talking around how we move to a more Agile operations team, rather than a sort of ticket-based system and doing projects in a Waterfall way. So trying to get some of those concepts that we’ve learned in an application mode into the operations teams, and the challenges of doing that, and how we get those two linked together so we can minimize and help the flow of feedback that goes through development into operations systems. This is the final bit of that journey, I guess.
Gene: And you could you talk a little bit about how the change in your role, maybe even going back, I mean how do you think the work that you did in terms of pioneering some of those practices at Hiscox, how did that lead to your change in responsibilities, and how does your change in responsibilities allow you to do more or less going forward?
Jonathan: So I started as a Solution Architect. So actually nothing to do with DevOps at all, and it was kind of just was born out of frustration. I was like, “We’ve gotta change the way we do stuff because this is daft.” So it’s coming in with a great new focus which, again, most DevOps translation started quite small which we got an application, and we did some automation around it, in terms of scale and CD and some automated testing. And then we saw these successes of that and went towards something else, and then it’s just kind of ballooned over the years. And, I guess, we’re really starting to see some of the business payback for that and that’s helped me.
I get more visibility to be seen as someone that’s trying to be more strategic, in terms of how as an insurance company we maintain a competitive advantage. And I think that this step to CTO is sort of the logical conclusion is to help to see that journey through. It’s helped me because now I’ve got a lot of guys rather than influencing up the chain over time, but other peers of mine are direct reports so that that’s gonna help…not that it’s a dictatorship. But it’s going to help my influence and ability to do that, as well as having own budgets and ability to spend some money on that.
Gene: By the way, conclusion sounds so terminal, Jonathan. Let’s not hope that’s the conclusion…haha!
[Crosstalk]
Gene: A continuation perhaps.
Jonathan: A continuation, yes!
Cornelia: I love what you said about Agile operations team instead of tickets. So instead of tickets, it’s so huge. It’s such an important part of DevOps, and it’s all about empowerment, right, but empowerment still in a safe environment.
Jonathan: Yeah, absolutely. I love that. And I think there’s a technology problem there as well. I think it’s very difficult to align the types of tooling that develop as you’re using along with the ops guys that also want to have things like, ITIL based processes and a change management database, and trying to get that transparency across the whole value stream is quite difficult. So that’s one thing I’m kind of wrestling with at the moment.
Gene: That’s great. Jon, how about you? Yeah, what has changed in your journey since you presented back in June?
Jon: So knowing why you’re doing this is important. So why is any company doing DevOps, adopting DevOps and why are active products faster with delighted customers and colleagues, better, faster, happier, safer? And so taking a broader definition of DevOps, in terms of better product, faster, safer, happier we’ve increased our adoption across the organization. As I said, we’ve been doing this across the whole firm for two years and two months.
If you go back two years and two months, it was a relatively small percentage of our change work was done with these practices and principles, and now we’ve got a fairly significant percentage of our change work now being done with Agile and DevOps and Lean principles. So in 2016, we’ve carried on that at-scale adoption, and we have done it apace. We have something like, rough order of magnitude, something like 15,000 people who are now working with Agile and Lean principles, and practices who weren’t two years ago and important to get the ball rolling.
It’s important that we do it with control in a highly regulated environment where as we know the impact of attitude is significant. And so we’ve also been focusing on control because we want both speed and control. Sometimes people might think that you can’t have both speed and control and people might think that speed is at the cost of control, but actually, it’s quite the opposite. You can do anything badly. You can do fragile or you can do Agile, and actually an Agile approach and a lean approach increases control. So we’ve been putting quite a lot of effort in 2016 into control, as to both we are balancing by speed and control.
Gene: Cornelia, you had a question for Jon.
Cornelia: Yeah, I do. So, I thought it was very interesting that you said it’s been two years and two months. So that’s a pretty specific date/time period. What was the catalyst? What was the starting point? What defines as two years, two months as opposed to roughly two years ago? Was there a specific event?
Jon: Well, maybe I’m speaking from my personal perspective. It’s roughly two years. It’s roughly two and a half years and, specifically, what any organization needs to do this successfully, it needs both top down and bottom up. So the specific event, this was a catalyst, in our case is with new leadership and new leadership saying, “Agile and lean is the way to go,” and then the natural champions in the organization, like myself, we put our hands up, “We volunteer to help move the needle.”
Gene: Okay. I mean so in other words, this signal of support from upper management is kinda what you mark as the real start of the journey, right, where it was now a sanctioned, important activity.
Jon: And it was articulated, formally as part of our strategy, yes.
Cornelia: And that is something that I see across, again because I get to see across a lot of different enterprises that are all on a similar journey and that’s not at all unusual. When we have new leadership that comes in, sometimes new leadership that’s brought in, specifically to shake things up, that is often kind of the defining moment where things start to change.
Jon: Yeah, and like I said, absolutely, mandatory criteria to be successful across and organization, you’ve got to have that top down support, as well as the bottom up community. You’re gotta have those two things.
Gene: By the way, I love the number that you put out, in terms of 15,000 engineers working with Agile principles, just to calibrate. That’s actually the size of the Google engineering organization in 2013. So it’s just amazing to see what happens when you can elevate the productivity of engineers that it’d impact that many people. Jonathan, you had a thought.
Jonathan: Yeah, Jon, I was really interested, you were talking about control. Barclays is a fairly massive beast. I was thinking with your move to Agile. How did you have that control at the scale that you’re operating in? I mean lots of Agile teams work very much, almost as a silo, themselves. They’re in back of their work how to orchestrate such a large transformation in such a large company.
Jon: The main principle has been to start with what we have now. So we clearly are operating in a highly regulated environment. We’re not starting from a clean sheet and working towards what we have. We’re starting with what we have and we’re iterating to improve control effectiveness. A starting point was a Waterfall lifecycle, and I’ve spoken about this at the last DevOps Enterprise Summit. We had NT8 mandatory artifacts, many hundreds of questions, and it was a very un-lean process, and it was evolved over time, multiple input, probably from multiple audits and multiple control findings, and so on.
But it became really catered to the lowest common denominator, but applied to everybody. So what we’ve been working on is improving the control effectiveness and leaning the process. So it’s more as a greater risk sensitivity now. So if you’re doing something which is, say, for example, an internal system, nothing to do with payments, no client data, it’s purely…I dunno…a seating map, for example, then it’s risk sensitive. So the controls that will apply to that are in line with our control framework.
Something that is to do with payments, some has client data, the control implications are much higher. The main thing is it isn’t a lowest common denominator for everybody. The thing is applying Agile principles, we’ve moved it to be more of an early and often conversation with our control professionals. So, previously in a traditional manner, you deliver something and we had a control engagement at the end, just before you’re about to put it into production.
And then that leads to another three months to six months of unplanned work, in a reactive manner is what we observed in some cases. So instead we’ve got long-lived controlled tribes, multi-disciplinary long-lived association with the customer value stream, and then you have a conversation at the beginning, and you have regular small conversations as you go, so there’s no surprise at the end. And there isn’t an end because it’s a long-lived product with regular deliverables.
Gene: That coincides with the regular audits that happened, right, the compliance reports. That makes a lot of sense.
Jon: Exactly, exactly. Exactly, the control effectiveness is stronger because you’ve got that regular interaction and the regular looking at an actual product that you’ve actually got something tangible to look at.
Gene: Just as a little side note that we have a group and, hopefully, you’ll be joining us, Jon, from MarkSoft, Capital One, and others, just really trying to help bridge that gap with compliance professionals using their terminology, using the design control effectiveness language, really to show that you know DevOps people aren’t crazy. In fact, you end up with better control effectiveness working all the time. So that’ll be a very fun interaction. I’m very excited about that deliverable.
Jonathan, one of the themes that have come up here is that wonderful things happen when you’re surrounded by fellow travelers who are facing the similar types of challenges and kind of the learning that can ensue. I actually had the privilege of observing one of those sessions that you and I were talking in, and you had a conversation with Scott Prugh who’s the VP of Development and Product Operations at CSG. They’re one of the largest bill printing companies in the United States, and the topic was how do you overcome some of the traditional challenges, how you get more traditional IT operations organizations on board. Can you share some of the challenges that you’re grappling with and some of the insights that came out of that conversation because it was absolutely amazing to observe?
Jonathan: Yeah. So in terms of our org structure, Hiscox a fairly federated business. So we have a number of British units aligned to countries around the world. Application testing teams tend to be aligned to their business units, but operations infrastructure is a group resource that delivers against all of those operations, all those departments. So what we’ve been wrestling with really is how do we or do we maintain that? Do we align infrastructure operation teams to a particular country or do we continue to measure out of a group resource? And if we do that, then how on earth do we have a backlog where we’ve got so many different stakeholders with different requirements, different projects?
And how do we prioritize amongst all those different business units and how do we still get projects done? How do we still get BAU done? I guess the questions that any Agile team has, it’s just on a much grander scale. So each business unit is operating independently. I don’t really care what some of the other business units are doing as much as it’s about their P&L. They want to first bust at the cherry. So does that mean then that BAU goes off to the side and we stop patching because there’s continued focus on projects? So there’s some of the sorta challenges we usually went over with Scott.
Gene: Yeah, mute button. And what was just some of the insights that you had or at least, maybe even validations of the problem? Specifically, how do they inform your thinking?
Jonathan: So I think what was quite interesting is listening to Scott’s talking about visibility of work in progress. And some of the frustrations there are between development operations is because both of them are almost black boxes. So a developer might say, “Look, I need an environment and it might go into a ticket system” and it goes from the server guy towards the SAN guy, and the storage guy, and it goes around in little circles.
How can we let the developers know what the ops guys are up to, and conversely, how do we let the ops guys know what developers are doing? So how do we get that work-in-progress highly visible and that, as a starting place, would be a good way of beginning to address some of the prioritization issues because they’re working off completely different backlogs. The left-hand does not know what the right hand’s doing. How do we start to make that happen? So that was a good sorta first head’s up from him, just to go and look at that problem.
Gene: All right, in fact, I think he’s spoken a lot about trying to get more and more of the ops work into the same work systems as development. So in their case, it would JIRA, and so forth, and less the remedy and so forth, right, just to have a single prioritized work unit.
Jonathan: Yes, but that’s pretty hard. And who’s gonna be a natural tendency for projects to trump everything else because that’s what a business is asking you to do and we’ve gotta change out the door? But you still need to make sure things are patched, and security’s done, and I really like that sort of conversation where you start to allocate points in a sprint, so that’s BAU and that’s projects. If you don’t do that it becomes a very difficult way to triage and prioritize stuff that’s coming because like BAU is gonna be on a loser.
Gene: Cornelia, as coming from a development background, I think you’ve observed something very similar in your journey.
Cornelia: Absolutely. And one of the things that’s really interesting that we found very effective is to take the same product-oriented mindset that the application teams have when they’re building and delivering their products, and actually have that same mindset at the operations and at the platform level. So that, instead of it being treated as just unplanned work, work that comes along, now it comes in, we have to prioritize it, they start to treat their offerings as products themselves. So that they do things like measure…they have a backlog of additional features that they wanna create.
They have release notes that say, “Hey. We’re patching the system on this day” or “You have the opportunity to apply your patches because the patches are queued up and ready to go.” And so really shifting that product mindset, all the way deep down into what’s kind of traditionally an infrastructure kind of role that tends to be a little bit more order cook-ish, and really transforming that into a menu-driven, “Here are the offerings that we have.” Of course, you can’t get rid of all of it, but that’s really, really powerful, is a product oriented mindset, even at that deep infrastructure level.
Jonathan: I think you’re entirely right. I think the mindset, yes, I think the practical application is more difficult. So if you build an environment, for example, you need a server person and a SAN person, and a storage person, and there are lots of techology-silos. So what isn’t very practical is go and take all of those resources, and go and stick them in a dev team because your Agile team’s gonna be 400-people strong all of a sudden which is not ideal. So I think I get the mindset implication or, I think, the technical implementation of that, I think is a bit harder.
Cornelia: Yep.
Gene: Jon, I can’t imagine that somehow Barclays is exempt from some of these issues, as well. I’m guessing that this has relevance, at least in some areas of the organization for you.
Jon: Yeah, absolutely. This really resonates. I think if you go back and my own personal experience, you go back about 20 years ago and, specifically, that’s when banking, we used to do all the roles. I was a developer. I did the analysis. I did the development. I did the support, and I sat next to the customer or the partner and we did everything, and over time, we’ve gone from a role specialization and the split between kinda ops, run, and build, dev. And so, yes, this absolutely resonates.
Generally, for Barclays, coming from a generally Waterfall approach prior to two years ago with islands of Agile, I’m looking at this topic, we’re looking at how we get closer to you build it, you run it. But still maintaining regulatory compliance around segregation of duties, Sarbanes-Oxley Compliance. But as we’ve discussed, the run specialists, the operational specialists, utter the feedback with the build and the dev specialists need to all be lined up behind the customer. And, obviously, what happens when you’ve got operations, you’ve got run who are working in their own world.
The feedback loop never gets back to the dev team, so developers need to feel the pain. I like the SRE Site Reliability engineering philosophy of hiring engineers into ops roles because when you put engineers into op roles, things tend to get fixed kind of at the root cause, rather than a band-aid, rather than a short-term fix. So we are absolutely this is a high priority for us, especially in 2017 to try to increase the feedback, and get more to being one team. But to Jonathan’s point here, we can’t do that with everybody in every role. So yeah, there’s going to be the product teams and there’s going to be secondary teams, and that’s absolutely something we’re trying to move the needle on.
Gene: Jonathan.
Jonathan: Yeah, I think when you start, I think, cloud for us changes that. When the infrastructure guys start to use the same processes, and tooling, and approaches that the developers do, then I think that becomes a lot easier. So I’m trying re-push infrastructure as code at the moment, and I think that is the way of really breaking it down. If infrastructure is just another software project, that becomes really easy to put into that Agile team with the application team. If you’ve got a mix of traditional, “I’ve gotta go and plug a server into a wall. Then I’ve gotta go and configure this” that doesn’t really fit in with a normal Agile team.
And you’ve got a guy was talking to me earlier, he wants to order new servers, six to eight weeks, and then we’ve gotta go and get finance approval for it and. “I think I’ll do this.” That’s not gonna fit in that tight feedback loop that Agile’s trying to work with. So, for me, it’s about transitioning some of the infrastructure skills into more software-led skills, and that’s where you’re gonna get that much closer alignment. That’s where you’re gonna get the DevOps cultural start. That’s where it is. I think trying to do in on-prem, in a manual way is kinda of a bit of a recipe for disaster.
Gene: I think the best people that really embody this is Jason Cox from Disney. His title is Senior Director of Systems Engineering at Disney and he’s talked tirelessly about how he’s tried to create, embed those ops, talent and engineers into the lines of business, into the dev teams, really helping them do things on-demand, and it’s been amazing to see his own journey over the last four years. Just a little joke here.
You talk about infrastructure is code, Jonathan, I love how Scott Prugh from CSG, he said, “The ops of that is infrastructure’s tickets” which is actually what we don’t want, right. We want those on demand. Let’s go to Jon, and one of the just tantalizing pieces that you had talked with us there at Prep Call, just before this panel was the Lean Portfolio Management work that you’ve been doing and how that’s become your newer areas of focus, and I think all our ears perked up about that.
Because of I think this realization that all of us have had over the years, is that you can have fast iteration loops within the dev test operations stream, but if you have an annual budgeting cycle and annual planning process, that really sets some very hard limits on how Agile you can be. Tell us about what you’ve been working on.
Jon: Yeah. So, as you say, we can have as much agility as we like. We can be practicing DevOps. We can have great continuance delivery, but it’s a local optimization when you look at the system, think a system’s thinking and a theory of constraints to deliver better products faster and with delighted customers and colleagues, it’s gotta be end-to-end agility. And so I think this is a natural evolutionary thing, having made some pretty progress, I think in the last couple of years, in terms of how we do what we do.
As we start to expand out in the theory of constraints, look at what the next constraint is. As a large, complex organization, we’re starting from the position of annual planning and annual budgeting, and that doesn’t give us enterprise flexibility as much as we would like. So this is a focus for 2017 for us, is to move to more of a courtly rolling wave type approach, more of a hypothesis-driven investment approach, more of a focus around outcomes, not a predictive plan. So once a quarter you said, “This investment should have achieved these outcomes. Has it?” And being able to build more agility into the system of work holistically.
Gene: Yeah, Jon, I’m dying to ask you. So I think the way I was trained there’s kind of two planning processes, right? You have the project budgets, right, which are owned by the business which tend to be…they can go up and down quite quickly. Whereas then you have the annual shared services budgets, right, where you tax every person, right, 2% of revenue or whatever, and then you a create work queue that’s prioritized, and maybe it’s not a good year now, maybe you’ll get it next year or the year after. So has your focus been primarily on the project on the business side? To what extent can you translate that to shared services?
Jon: So, that’s a good question and I think a caveat to what I said, as a public-listed company, we will still have an annual process. What we’re trying to do is we’re trying to push the courtly rolling wave, portfolio-management process as far…We’ll start at a certain point in the organization and then we’ll try to push it as far up the organization as we can. So that, basically, within our budgetary constraint whether it’s a business investment, you used the word project, whether it’s a business investment or whether it’s a shared service, irrespective of that, you treat it, your budgetary constraint, it is a constraint, and you wanna maximize the value on it.
So you would do it, you know my view is that we do it in both because, as well as about financial flexibility and being about to move money around, it’s also about de-risking delivery and getting away from the sum cost fallacy for a shared service. You know, “You said you’re gonna achieve this and, of course, this year, here’s this big chunk of money.” Naturally, “Have you achieved it? Are you going to slowly?” You know, “Can you cut the value chain? Have you delivered your value? Can you stop early? Can you pivot?” So it kinda serves multiple purposes.
Gene: Just a joke. I mean I was actually having a startling conversation with a friend of mine who was saying that they’re a publicly traded company and there was this sorta zombie operations project that they had depreciated over five years, and they couldn’t kill it because that would actually create a material change in the financial statements. But everybody knew that such a $7 million in a right world, we should just admit that it didn’t work. Kill it, so we don’t have to prop up this fallacy of a services actually running when it is not running at all. Does that resonate at all with you?
Jon: Yeah, it does. You know what we’re trying to avoid happening is exactly that kind of that sum cost fallacy and looking at a traditional approach, I’m sure we’ve all seen large initiatives which have that got multi-year sum cost fallacy that hasn’t delivered, but we’ve already spent so much money on it, we’re gonna carry on spending money on it.
Gene: Just to add color to that story. So they actually had this annual ritual where they’d have to run one transaction through it, just to prove to the compliance people that it is still operating, and so therefore, still eligible to be depreciated which is a shame. Jonathan Fletcher.
Jonathan: Jon, I love what you’re trying to do with that. It sounds fantastic. My question to you is are you forever stuck in writing businesses cases for return on investment? Does that slow the process down?
Jon: Hopefully not. You know we’re taking a lean canvas approach. So it’s not in the past in a traditional approach, you might have had to fill in a PID project initiation document which might be multiple pages long. We’re trying to keep it lean, so we’ve got a one-pager, lean canvas. You’ve got all of the information you need on that one-pager. Again, it’s conversation over contracts. That should be sufficient to have the conversation in terms of, “Do we feel this is the right way to go?” You know we might need to see more evidence. We might need to see more feedback from a proof of concept or experimentation or a customer focus group. We’re trying to keep it lean with this one-page lean canvas.
Jonathan: And is there a maximum budget that you would go with that project? Would you spend a $100 million on a one-pager?
Jon: We have a tiered approach and the tiering is the seniority of the people making the decision.
Gene: Conversations over contracts. I love it. So I’d love to know from each one of you, what are the favorite resources that you go to for finding your inspirations, practices, principles? I imagine this could be in the form of books, events, articles, research. Jonathan, would you mind sharing kinda what books are you reading these days that you’re finding most useful in your own daily work?
Jonathan: You looking for a plug, aren’t you Gene?
Gene: No.
Jonathan: All this here is homework. I think conversations of this kind of thing helps me immeasurably. So there’s this stuff you can get in theory, but it’s real world, practical, “I did this and it failed, and I could have approached it differently.” That’s where I get a load of value and I think just talking to people and getting out into the community. The DevOps community’s great at sharing information. Unlike like any other part of the [inaudible]. So I think people are always really generous with their time. So, for me, it’s not about reading papers. It’s about talking to like-minded individuals and the ones that are honest and go, “You know what? We tried that and it was a terrible mistake. Don’t go down that route.” That’s of much more benefit to me.
Gene: Great. Cornelia?
Cornelia: Yeah. I think a reason that Jonathan makes that point, and I can’t argue against that, is that we are still as an industry pretty early on in this journey. And somebody brought ITIL earlier and sometimes ITIL kinda gets a bad rap. But the reality is that ITIL is something that emerged over years, decades as kind of the codification of the best practices that we knew at the time. And when I was in college I learned about the Waterfall Methodology. That was the best practice at the time so, I’m dating myself, right?
Thirty years ago, in Computer Science Education, they taught us about the Waterfall Methodology and that was the state of the art. We created all sorts of practices around that. We created the books, the standards, and so on, and so we had a bunch of resources to go to. So when somebody was setting up an IT organization or evolving it, they had the manuals to go to and say, “Ah, here’s the codification of the best practices of the day.” Well, I think we’re still so early in the DevOps journey, not super early days because we’re seeing these fantastic success stories, but we don’t have the ITIL of the next generation yet.
So it’s hard to say that this is the go-to manual, this is the Bible. It is really about events like this, and it’s about, I, personally, I’m somebody who, when I’m on the treadmill in the morning, I watch videos, and I’m a propeller head, so I watch a lot of the tech videos, you know purely tech videos, but I watch the videos from previous events. Those things are incredibly insightful and really useful.
The DevOps Enterprise Summit, obviously, is just a fantastic event, but then there’s also what I consider more the grassroots events, like the DevOps Days which are not quite as focused on strategy, but definitely are focused on sometimes very pragmatic, very specific techniques and tools that are being leveraged. You need to bring that together with the high-level strategies. So it’s really still the community, but the good thing is that there’s lots of places to engage in that community.
Gene: Awesome. Jon, how about you?
Jon: So I think what Cornelia said that really resonates in terms of being early days, it’s not early days for Agile and Lean principles, at least in the 1980s, if not earlier, however, what we are learning is doing it to scale, and it’s been at the team level. It’s known at the team level. It’s not known at the enterprise level and so this is where a lot of the learning is. It’s, “How do you do this at scale?” And I’m not referring to scaling frameworks, necessarily. It’s more messy than that. It’s culture change. It’s behavior change. It’s involving everybody in the value stream. It’s not just technology.
Just technology is a local optimization. It’s everybody in the value stream, and we are learning, and this is a new way of working. This is a fundamentally new way of working for organizations with tech or without tech. I don’t know if this right term, but it’s like the fourth Industrial Revolution. We’re going from a tailor-ist way of working from the Industrial Revolution to not just a handful of firms working with multidisciplinary teams with long-lived products, and long-lived value chains, and fast feedback and fast learning, but entire organizations, and entire organizations trying to change how they do HR, how they audit, how they control, how they do portfolio management is a part of that, but it is a part of it.
So we’re absolutely early days. It’s a new way of working. It’s a more engaging way of working. It’s a happier way of working, and at things like the DevOps Enterprise Summit is where we come together to share on this learning. You know the question was, your question Gene was, “Favorite resources for learning…” so another really good forum which is quite similar which is around peers sharing is the SD Learning Consortium, sdlearningconsortium.org and it’s a community. It’s not commercial and we have a number of large companies sharing this journey and that’s where I get most of my learning from, as well as Twitter, as always lots of great content being tweeted.
Gene: It’s so true. I think six years ago I would have laughed at you, but just as a little side note, many people might not realize that one of my reasons for tweeting is actually too, is my mechanism recording things because, ultimately, of course, you’d actually write things down. Awesome. Jonathan Fletcher, you had a thought.
Jonathan: Just to add to exactly what you guys were saying. There’s next to where I work there’s a brand new building being put up, and I find it fascinating watching that process go on. So they’re damming in the foundations and from our floor, we can see right down to the foundations and see the guys that are working. And I often think of a parallel between a building and what we’re doing. You know IT is 20-30 years old. These guys who are building buildings since the pyramids, and they know what all the things are gonna happen, and they know the tolerances of the steel.
They know how much bricks they’ll need. They know the amount of concrete they need, and they know what order it has to go up. They know they need Planning Commission 1, 2, and 3. They don’t need to get halfway up the building, commonly knock it down and start again. And I think we should strive to get there in the future. We’re actually this is a common and understood journey and there are artifacts, and processes, and cultural norms that we can call upon, and I think this is the revolution that Jon was just talking about we’re going through now.
We’re trying to work out what these things are, and I guess I have one part which becomes routine. This is the university course of the future. I’m exactly the same as Cornelia. I can remember my sophomore engineering course and they’re talking about the silo model and Waterfall and this and that. That’s how you did it. So I think we go through that process now and I’m looking forward to that becoming more formalized and recognized, I think.
Gene: Fantastic. And love this, you know as you were reflecting on this, I was thinking about this quote that I think about all the time. As someone once said, “You’re only as smart as the average of the top five people you hang out with,” right, and so that is learning only happens when you’re hanging around people that you actually learn from, right? So that’s something I think about a lot. So we are all lifelong learners, now more than ever, at least for me.
I can’t tell you how many times, it’s difficult for me to overstate how much I’ve learned. I mean it’s just I’ve learned so much, even since the Phoenix Project came out 2013. I’d love it we’d just maybe go around the panel and just share one lesson that you’d like to share with everyone that you think it has significance and importance to everybody. Cornelia, let’s start with you.
Cornelia: Yeah, so one of the phrases that has really resonated with me as of late is this notion of radical curiosity and really just being continuously reminding myself that everything that I’ve seen so far is just what I’ve seen so far. And so I find my best days are the days where I get to talk with folks, where I get to ask the questions. I like to ask questions a whole lot more than I like to talk about things because it’s there where you get those insights. Even just this conversation for the last hour has just been the best part of my day and it’s only 8:00 o’clock in the morning for me.
So it’s really about asking questions and not falling into the trap of, “This is the way we’ve always done things.” I remember somebody asking somebody about how they do their firewall rules, how they determine what those are or how they secure something. And they said, “Well, we set the firewall rules.” And I was like, “No, no, no. I asked you how you secure things.” And it was all about the firewall rules. So it’s reminding ourselves and that we all fall into those traps of we have the solution, and we forget what the question is. I think that’s my answer.
[Crosstalk]
Gene: Yeah, that’s awesome. Jonathan Fletcher.
Jonathan: This is kind of a heavy thought for 4:00 o’clock in the afternoon. I think just that I’m really not the cleverest person in the room, and to keep an open mind about what others are telling you. Because as Cornelia says, this is a changing and evolving thing. People are reshaping IT and you know you don’t have to be the smartest person to pick up on some of this stuff. This is a movement and a cult of change that is happening, being able to set out some of these ideas and the Jez Humbles…All we’re doing is sort of following behind and listening, and picking the good bits and implement ourselves. So we’ll copy other people’s ideas, I guess, but that helps me get to where I want to get to. So just recognize that you’re not the smartest in the room, and just take the lessons that others have set out for you, and learn from them and move forward.
Gene: Perfect, love it. Yeah, one of the people I admire is Martin Fowler and I think he wrote kinda somewhere about his role, and he calls himself the best dumpster diver of other people’s ideas. I thought that was actually lovely beyond words. Jon Smart.
Jon: So in terms of a learning for me, I think in terms of culture change, an org change at scale, it’s going from, “I believe it when I see it” to “I’ll see it when I believe it.” So, you know I only really see it when I believe it and you to actuate new behaviors. So what I mean by this in terms of culture change, it isn’t about just going in there and saying, “Think this way” and trying to change the thinking. People need experience. People need to experience it, and actually you need to change the behaviors to change the thinking, to change the culture. I’ll see it when I believe it. That’s a learning for me.
Cornelia: Brilliant. That is brilliant. I love it.
Gene: I love it as kinda I’ll force it into being by sheer force of will. I love it, from Jonathan Smart. Fantastic. Well, to Jon Smart, Jonathan Fletcher, and Cornelia Davis, I’m so grateful that I spent an hour with us, and I know that your time is exceedingly valuable. You’re working on incredibly important things. I can’t tell you how much it means to me that you spent an hour with us, and for this hour, I hope that everybody else got as much value and delight as I did. So thank you, again. Now you’ll see more from us about the DevOps Enterprise Summit program as it shapes up.
Again, all three people here are part of the Programming Committee, and our goal is really to make that even better than it was last year. So I look forward to, hopefully, seeing all of you, on June 5th and 6th in London at the QEII Conference Centre. And, of course, an amazing venue in the center of Westminster and the center of London. You’ll see all the updates online at events.itrevolution.com, and again, early bird tickets end the 8th of March, so be sure to register as soon as possible. So with that, thank you so much and I look forward to seeing all of you on the Programming Committee calls or rather on the panel, and stay tuned for more updates. Thank you, again.
In this DevOps Chat we speak with Laurel Dixon-Bull of Urban Code and Frank Canihuante of Travelers about CI/CD. Frank explains to us how he improved the CI pipeline at Travelers and some of the challenges he faced with the scale of a large enterprise like Travelers. Good case study.
As usual, immediately below is the streaming audio of our conversation. Below that is the written record of our chat. Enjoy!
Alan Shimel: Hey, everyone. Alan Shimel, Editor in Chief, DevOps.com. Here for another DevOps chat. Today we have two guests on DevOps Chat. First of all we have Laurel Dickson-Bull of IBM’s UrbanCode. Laurel, welcome to DevOps Chat.
Dickson-Bull: Hey, thanks Alan. I’ll just give you a quick introduction. I’m an offering manager for UrbanCode Deploy. And I started at IBM about 14 years ago, working on internationalization and localization. I spent a few years in release management and saw the agony of software delivery. And then now working very close with UrbanCode Deploy delivery team.
Shimel: Excellent. And our second guest is Frank Canihuante. If I mispronounce it I apologize Frank. We’ll give you a chance to correct. And Frank, you’re with Travelers. Welcome to DevOps chat.
Canihuante: Hi, Alan. That was close enough as far as pronunciation. Yes. So I’m a tech lead here at Travelers. I’ve been with the company for three years and was actually hired to lead the technical aspects of setting up a new pipeline actually. Been a developer for over two decades under various roles, different types of projects. And for the past 10 years or so, I’ve been very involved with things that relate to what we now call DevOps. And always thought of that as being just doing good software engineering.
Shimel: Absolutely.
Canihuante: So this is right up my alley.
Shimel: So we spoke a little off camera or off mic before we started today. And Frank, you mentioned that you’ve been doing sort of very DevOps like things for the last 10 years, way before they started calling it DevOps. And you know you’ve been riding this wave for a while. But Frank, I wanted to take a quick moment and talk about sort of the legacy that you had at Travelers. You mentioned you joined three years ago and you were brought in to really kind of update the pipeline, the CI process if you will. Right, continuous integration. Can you give us an idea of what it was like when you first came on board?
Canihuante: Sure. You know the interesting thing about this project that is now basically setting up a pipeline. That it wasn’t really stated as such. It was about replacing the old systems. And that right away told me that we weren’t quite sure of what the actual solution was. Now, these days we think of CI, CD, even three years ago, those two words weren’t all that popular as they are now. Of course now we understand that that is the way to go these days. You set up a continuous integration and a continuous delivery set of tools. So there was a lot of confusion. There was a lot of angst because everyone knew that if you were about to change, these tools are near and dear to your day to day activities as a developer, as a QA, as a DevOps professional. There’s some disruption that’s gonna happen. So lots, lots of expectations and a little bit of apprehension as well.
Shimel: Yep. Got it. And so Frank, you came in and you know tasked with updating this. For someone who’s listening out there and says geez, that’s just where I was. Where Frank was three years ago, that’s where I am today. Can you give ’em a little advice? Where do you start?
Canihuante: Sure. So I have put pipelines together before but at a smaller scale. So my biggest concern _____ at the same time what made it very exciting for me to take on the job was how am I gonna do this in a large enterprise. Travelers has 30,000 employees, 3,000 of which are in IT. So I knew this is directly or indirectly impacting hundreds of thousands of people. So there was a lot of responsibility to go with it. So one thing that I would recommend is you have to do your homework. You have to have a lot of experience in development to be familiar with the way developers think. Because at the end of the day they are your customers. And they’re very demanding. They have things that work now, however quirky they may be, it works. You’re about to change that. You have to be sure whatever you’re gonna replace the old tools with, it’s something that you as a developer feel comfortable with. This is where the Devs part of the DevOps part comes in. And the other part also is once you have the system put together, you have to run it like a product. And this is something that sometimes we in the rush of getting something out there, people don’t perhaps think of the operational aspects of a system. But it is something that we need to have.
Canihuante: I was just saying that you know putting it together requires a set of skills that basically call on all your development experience. But you cannot forget about how are you gonna run these tools, how are you gonna operationalize them. So trying to keep your customers happy is yet another challenge.
Shimel: Yeah. Absolutely. And so Frank, ultimately you and the Travelers team wound up partnering up with IBMs UrbanCode deploy on this project, correct?
Canihuante: Yes. So we looked at several tools. We had all the main vendors of CI and CD tools.
Shimel: I mean you don’t have to name names but —
Canihuante: Okay. Sure. So we had a process in which we brought in several vendors of CI and CD tools. And we really put UrbanCode Deploy through its paces as far as the deployment tool. We had in our RFP, which we did have a formal RFP process, many, many questions that related to how it actually worked and how it was actually operated. So we had a pretty good idea before we even brought the tool in for actual POC. But having the POC is what really made a big difference in realizing what it can do. It was very different from the old tool that we replaced. And actually once we had our users in, it brought the level of angst and anxiety that they had about the new system down to the normal levels. You know one thing is to see a demo, another thing is to actually put your hands on it and actually get an application be deployed through this new tool.
Shimel: Got it. So Laurel, let me turn to you a little bit. When did you become involved with Frank and the Travelers team?
Dickson-Bull: Well, I’ve been aware of Frank and Travelers team probably for the last couple of years. I think I visited you down you in Connecticut Frank, probably a year or two ago. I have to say though most of what I know about Frank and his team are things that I’ve heard. Really strong team. They’ve actually adopted UrbanCode very quickly and have you know made really fast progress. You know to some people, two or three years may seem like a long time. But to standardize on a deployment tool and in a large organization with ten thousand developers, that’s actually really, really good progress.
Shimel: I agree. So Frank, give us an idea you know first of all do you consider this project to be successful at this point?
Canihuante: Absolutely. There are many ways to measure how a project is successful. Projects of this type. In our case, we had over 2,000 applications in the old system. And one way to measure success is to say how many do we still have in the old system after a years’ time. So out of the 2,000 plus, we got about a hundred or so that are still in the old system. And they’re on track to be actually migrated to UCD in the next few weeks or months. But in addition to that, there are these other things that actually make it successful. Just having built something in collaboration with hundreds of other people and get their heads together as to putting together a pipeline that’s gonna make their life easier down the road. That’s another way to measure success. It’s kind of hard to measure it, but you see it in the way people talk. For example, we’re no longer talking about the way we need this tool or that tool. We’re talking about two different ways to solve a particular problem. Before, we didn’t even know how to solve it. Now we have multiple ways. And even the conversations are different. It’s not about particular problems that a group may have. But it’s always thinking how can we solve this problem for everyone. How can we come up with that pattern that solves how to build and deploy applications? Not just for my group or my division, but for the entire company.
Shimel: Got it. And Laurel, does that jive with you know I’m sure you’re speaking with other organizations using UrbanCode Deploy. Is that sort of measure of success the same you know across the board? The same type of thing?
Dickson-Bull: Yeah. Well, I would say that what Frank described as you know the challenge of standardizing on a tool for the enterprise and then going through that and it really opens up the conversations. Because once you have the visibility into the deployment automation and you start having success operationalizing your software delivery. People get excited about that. And they want to you know solve problems of automation. Dev and Ops are talking to each other. And the conversations become more collaborative. So I have definitely seen that with other large companies. And you know Travelers does represent a very large segment of our customers in the financial services. You know banks and insurance make up a good chunk of our customer base.
Shimel: And you know Frank, in talking again off mic a bit you know setting up your CI pipeline. UrbanCode Deploy’s obviously a tool that you use. But you guys are also using the integration with that and Jenkins. Correct?
Canihuante: Right. Yes, that’s correct. And Jenkins is interesting too because it was already in-house so _____ reasons why people wanted it. But it comes in two flavors. We have Jenkins ______ with various versions ______ also standardizing on a enterprise version of it. And we provide solutions that now encompass not just UrbanCode Deploy but also Jenkins. And we’re also providing a standard way of solving particular types of deployments. For example, how to deploy to WAS or JBoss. How to do .Net deployments. How to do database deployments. And it gets everyone really excited about having a standard way but sometimes even with flexibility to even tweak that a bit. Because we have some groups that only do say .Net, others do only JAVA or some do a mix of both. And now we have a menu of solutions that will span more than one tool. In addition that, there are other things that we have done that we are picking up these oddball applications. Like for example, telephone applications, voice applications. Even things that _____ not thought that we were actually going to install for but we have the ability to do that such as mainframe components, desktop components. We’re all bringing it together as a family of solutions on these tools in addition to integrations with other systems outside of the tools themselves. Such as the ticketing system for example or the monitoring system.
Shimel: Got it. And Laurel, you know I’ve known about UrbanCode having integration with Jenkins for some time. Actually at Interconnect I think it was last year or the year before, Eric Minick of UrbanCode actually did a joint session with Kohsuke Kawaguchi, the founder of Jenkins and CTO of CloudBees. And so I’m wondering have you run into a lot of UrbanCode customers who are also using Jenkins?
Dickson-Bull: Oh, yeah. For sure. I would say about half of our customers are using Jenkins. And UrbanCode is you know a great extensibility platform for Jenkins, right. So we’ve got over 300 plugins that will integrate with multiple technologies. And then like Frank was saying, multiple platforms from COS and even .Net, right. We have a lot of Microsoft integrations. So, yeah. I mean Jenkins is a great partner actually in the customer environment.
Shimel: Interesting. Guys, as I mentioned to you earlier, our time here goes so quick and we’re already coming up on 15 minutes. We’re gonna have to try to wrap up. But as I mentioned, I always like to ask our guests you know can you recommend a book to our audience that you know kind of a must read or should read. And Laurel, I’m gonna let you go first. Is there a particular book you’d recommend to our audience that they might find useful?
Dickson-Bull: So in the vein of DevOps, collaboration and teamwork that’s really important. Especially in agile development. There was a tool that I came across (that helps you) figure out your strengths. And then when you work with your _____.
Shimel: You know what Laurel —
Dickson-Bull: What?
Shimel: I think we missed you when you said the name of the book. Can you repeat it?
Dickson-Bull: Okay. It’s not a book.
Shimel: Oh, okay.
Dickson-Bull: It’s actually a survey or it’s a tool that you can use with your agile teams to figure out your various strengths. And it’s called Strengths Finder and it’s published by Gallup.
Shimel: Got it. Oh, very cool. We’ll see if we can get a link to that in the notes.
Dickson-Bull: Okay.
Shimel: Frank, how about you? Anything you could recommend to our audience?
Canihuante: Yeah. There’s so many books out there. But there’s one that I really like. It’s more of an author than a particular book. His name is Steve McConnell and he’s written the famous “Code Complete.” And I believe there’s a “Code Complete 2.” But there’s one other I really like. Actually two more. It’s called Rapid Development and the other book is called “Software Project Survival Guide.” These are from late 1990s. But if you read them it’s kind of like “The Phoenix Project” but 20 years ago. The same problems. The psychology of the software development world and how all that works. Really exciting stuff.
Shimel: Excellent. Well, Frank, before we close off and Laurel, I wanted to mention Frank, you are appearing at IBM Interconnect this year?
Canihuante: Yes. I am. I’ll be talking about our Travelers story at Interconnect.
Dickson-Bull: And we’ll be having you on a panel too, right Frank?
Canihuante: Oh, that’s correct. That’s gonna be on Wednesday. I forget the date.
Dickson-Bull: Yep. We have a few of our larger customers who have had success with DevOps joining us on a panel called, “DevOps-You’ve Got This.”
Shimel: Great. Sounds Great. We look forward to seeing it. I’m actually doing a panel there too called DevOps-It Takes a Village. So we could share notes Frank, while we’re out there. And Laurel, will you be attending Interconnect as well?
Canihuante: That’s awesome.
Dickson-Bull: I will be. I’ll be leaving that panel, DevOps-You’ve Got This. And then right before the conference, we have a full day workshop with customers from UrbanCode. So I’m organizing that now and we’ll be leading that on Sunday.
Shimel: Fantastic. All right. Well, Laurel Dickson-Bull of IBM UrbanCode. Frank Canihuante of Travelers, thank you for being our guests on this episode of DevOps Chat. Look forward to seeing you both out at IBM Interconnect the week of March 19th. And your panel Laurel that you’re organizing as well as Frank, you’re talking a little bit about your Travelers experience. And good luck as you continue to you know make the pipeline and DevOps happen at Travelers Frank. Thanks for telling us about it.
Dickson-Bull: Thanks Alan.
Canihuante: Thanks Alan.
Shimel: Thanks guys. This is Alan Shimel for DevOps.com. Thanks for being our guests today and we’ll see you on a DevOps Chat soon.
Many of you may already know that Gene Kim, Jez Humble and Dr. Nicole Forsgren launched a new company called DORA. DORA stands for DevOps Research and Assessment.
Some of you probably also know that DORA and the individuals behind it have been providing a lot of the science and analysis behind the State of DevOps survey and report for a number of years now. But beyond the 3 rock star founders, what is DORA really about? What is the business model to generate revenue? I had a chance to sit down with DORA CEO Dr. Nicole Forsgren and find out for myself.
I actually caught up with Nicole when she made a pilgrimage to the DevOps Mecca of Miami. Well OK, Miami may not be a DevOps Mecca, but we do have good Cuban food. It was the Cuban food that got her. Over a delicious lunch on Calle Ocho in Little Havana, Nicole explained to me what DORA is all about.
Dr. Forsgren is CEO of the company and currently wearing multiple hats (you have to love the smell of startup in the morning). Jez Humble is now also full-time with DORA and is working away building out the backend here. Gene is also very involved, but as you can imagine, his time is pretty tight these days.
DevOps Assessments
While developing the annual State of DevOps Survey and then analyzing the data is a job in and of itself, that is really just the jumping off point for DORA. By taking the data from something like 25,000 responses over the past years, DORA has been able to create baselines and models from which they can compare how your organization compares to others who have taken the survey. They can pinpoint where you are lacking or not performing up to par, as well as where you are over performing. The entire process is built on rock solid statistical modeling and has already proven itself with several large enterprises.
DORA’s business is in providing assessments and surveys of organizations to see if you are a high performing IT organization or where you might need help and attention to become even better. When should you consider a DORA assessment?
Consider a DORA Assessment if you have problems:
• Improving technology and organizational performance
• Prioritizing improvement efforts
• Attracting and retaining talent
• Establishing your brand as a high-performing organization
How it works:
Running a DORA Assessment follows four stages:
Decompose an Organization
Identify the Lines of Business and teams to be assessed
Prepare for Assessment
Socialize the survey and prepare email distribution lists
Take the Assessment
Everyone takes a 15- to 20-min survey
Generate and Deliver Reports
Team capabilities and outcomes benchmarked against the organization
Team capabilities and outcomes benchmarked against the industry
Prioritization analysis, with capabilities arranged by impact to Technology Performance
As I mentioned earlier, the results have been pretty impressive to date. For good reason too. DORA is sitting on a treasure trove of data and Dr. Forsgren has the chops to perform the analysis that makes it invaluable. With Gene Kim and Jez Humble, this analysis can be and will be turned to helping others improve their business performance by utilizing the insights gleaned from all of that data.
It is only a matter of time until their benchmarks and modeling become the industry standard.
But wait. there’s more. Dr. Nicole also told me about some white paper and book projects on the boards at DORA. Again, analyzing the data of all of the State of DevOps reports they are going to shine a light on the ROI of DevOps (I can’t wait for that one), as well as a book by all three DORA co-founders due out this summer further explaining the science and findings behind their research.
Best of all, because Dr. Forsgren is a PHD professor, most of their work has been collaborated and peer reviewed by some of the leading people in our DevOps community.
Right now DORA maybe the best kept secret in DevOps, but I suspect before long it will be a household name for those who are striving to be part of high performing IT teams.
We are just about 2 weeks out from IBM InterConnect, which is shaping up to once again be heavy on DevOps and related technologies. Of course there are other things on the agenda besides DevOps (really?), like cognitive and Watson for instance. There will be plenty of cloud, IoT and more cutting edge technology subjects. But you could read about that on other sites. Lets stick to DevOps here.
First things first is I want to invite you to participate in a CrowdChat we are holding on March 10th at 1pm EST. CrowdChats are one of my favorite social media activities. For the duration of the event it is controlled chaos, as multi-threaded discussions break out across Twitter, Facebook and Linkedin. You just have to use the hashtag #ibminterconnect to join in the discussion.
I will be one of the hosts of this event, but I will be joined by Gene Kim, IBM’s Angel Diaz and Erick Minick and some surprise hosts that I can’t mention right now. So join in the discussion and fun on the 10th.
At InterConnect itself, there are some great keynotes lined up and many outstanding speakers from diverse backgrounds. I want to call your attention though to one session in particular.
Enterprise DevOps: It Takes a Village
Wednesday, 9:00 AM – 9:45 AM | Mandalay Bay Ballroom A | Session ID: 1346A
When talking about enterprise DevOps transformation, many people look for a silver bullet—a magic wand that will convert an old, inefficient organization into a high-performing, DevOps-enabled IT team. But in truth there is no single silver bullet. DevOps takes a village. This DevOps village, like a real village, consists of physical infrastructure, as well as members. DevOps often needs a cultural transformation that builds a true community with the same aims and goals. No one vendor has all the tools and no one tool does it all. DevOps tools must work in unison. Finally, a methodology that recognizes the communal nature of DevOps is essential to bringing it all together.
I hear the moderator for this panel session is pretty good. Some guy named Shimel ;-). Seriously, we have a strong panel and it should be a great discussion.
Other speakers to catch are Gene Kim, Sanjeev Sharma (on his new book), IBM CEO Ginni Rometty, Wayne Brady and Reshma Saujani, founder and CEO of Girls Who Code. Check out the InterConnect site for full details.
We will be posting some more DevOps Chats and interviews of some of the scheduled speakers between now and March 20th as well. So stay tuned for those.
Tomorrow, we have a webinar with Carmen DeArdo of Nationwide, Donovan Stevens, South African Revenue Service and Rachel Reinitz, IBM Distinguished Engineer, CTO IBM Bluemix Garage. The webinar is on “Lesson Learned in Scaling Enterprise DevOps”, you can register here.
The third annual “DevOps Connect: DevSecOps” held at RSA Conference 2017 shows just how far DevOps has matured in recent years from IT subculture to mainstream practice. In the presentations in this slide show, you’ll see firsthand accounts where enterprises have transformed legacy methodologies to DevOps practices, building more healthy cultures and, of course, more secure software.
[/nextpage]
[nextpage title=”Breaking Bad Equilibrium” ]
Breaking Bad Equilibrium
John Willis
In DevOps we try to identify and fix bad equilibrium. We constantly look for discontinuity in the areas of technical debt, collaboration, risk and work-life balance. In this presentation, John Willis looks at some other successful fields that address equilibrium and discontinuity in their respective fields. Willis looks at areas of behavior economics, cognitive psychology and game theory. He also has some fun with a few pop culture books, movies and game shows as examples of bad and/or Nash equilibrium.
The holy grail for software security professionals is to make their development teams treat functional and non-functional requirements as equal citizens. This becomes even more challenging in “agile.” Wouldn’t it be great if you had a quick and easy means by which you can write pertinent and actionable “security stories” and place them into the backlog of all your scrum teams so that they can get prioritized and completed along with “user stories”? The challenge, however, is making this scalable and seamless in a large enterprise with diverse sets of frameworks, application stacks and programming languages. In this talk, Raghavan shares tales from the trenches of implementing such a system—what worked, what proved challenging and the associated outcomes.
DevOps and Security: What We’ve Learned from the 2016 State of DevOps Report
Dr. Nicole Forsgren & Jez Humble
Four years and more than 20,000 survey respondents later, Forsgren and Humble have learned a lot about what makes IT and organizational performance awesome. They also have learned some things about the role that security plays in technology transformations. Their latest research includes insights into trunk-based development, lean product management and employee engagement. Watch this talk for practical takeaways that will make your teams and technology transformations even better.
Where Bits and Bytes Meet Flesh and Blood: DevOps, Cybersecurity and IoT
Joshua Corman
We’ve heard software is eating the world. Corman says software is infecting the world. Our dependence on connected technology is growing faster than our ability to secure it—in areas affecting public safety and human life. Adding millions of lines of code and connecting everything to everything else exposes cyber-physical systems to new accidents and adversaries, Corman contends in this talk. This is truly where bits and bytes meet flesh and blood.
Despite best practices, modern software development and security have allowed 100 of the Fortune 100 to lose intellectual property and sensitive information—even our governments routinely succumb to adversaries. These failure rates cannot stand with the consequences of failure being measured—not in record count—but in human lives and GDP. Paradoxically, Corman says, it may take DevOps to rise to these challenges. Rugged DevOps is finding un-obvious common ground and breakthroughs like software supply chain principles, greater visibility and response agility, and immutable infrastructure. Corman says we must be better, and provides his view of what better looks like.
The Intersection of Release Engineering and Rugged DevOps
Paul Reed
At RSAC 2016, release engineering’s role in rugged DevOps was discussed with a focus on how it relates to software delivery supply chains and the increasingly critically important topic of security and security management.
This year, J. Paul Reed explores what we’ve learned in the past year about the intersection of release engineering and rugged DevOps.
He takes take a deeper dive into specific release engineering techniques and tools that can not only start you on the path to effectively managing your software supply chain, but also pay concrete dividends in making your software’s security posture better and help you remediate issues more quickly when a security issues arises.
He also explores some nascent trends on the frontier of software delivery and security management, including the role of human factors and systems safety in the broader context of fast, sustainable and, yes, secure delivery of increasingly critical software components in our society.
Next-generation software deserves security from the start and better collaboration to effectively reduce the real risks posed by attackers. Some might say this is heresy, but given the endless trend of security breaches from traditional methods, likely not. The ideal state of security has always been to achieve continuous improvement or level 5 maturity. By this very intention, security has always been a significant factor in the production of software but relatively difficult to commoditize because of its complexity.
Using DevSecOps methods and principles, simplicity and high-fidelity controls are emerging within the software industry to help organizations forecast and react faster to attackers. Lietz’s talk provides an essential road map for security practitioners to tackle how to bring DevSecOps to their organization and avoid common pitfalls that have come from early day lessons.
[nextpage title=”DevOps in a Regulated Environment” ]
Implementing DevOps in a Regulated Environment: The Aetna Experience
Duane Schleen
One of the big challenges organizations are facing is how to introduce DevOps principles to regulated industries. Industries such as health care or financial services must adhere to stringent security and governance controls, which often make it difficult to adopt new technology stacks such as DevOps and containerization. If you have adopted an agile approach to software development and have plans for moving to DevOps, microservices and containerization, what processes do you follow to avoid compliance missteps? This talk shares Aetna’s journey of modernizing its application stacks and infrastructure.
Schleen covers which considerations went into the design and selection of the DevOps methodologies, which applications were moved to microservices and containerization first and what was observed as a result. Schleen also covers how security departments can help the business understand the advantages of DevOps and containers—removing fear rather than adding fear. Finally, Schleen details the experience of architecting security across a rapidly changing application environment, and how continuous integration and continuous monitoring can go hand in hand in delivering application agility, but also security vigilance and competency.
Listen to enough DevOps conference talks and it all starts to sound like: “deployment, deployment, deployment.” But what happens after deployment? What does DevOps mean for other traditional enterprise operations activities such as incident response, problem management and compliance?
Damon Edwards tackles these questions in this session. Here, Edwards examines what happens when the “go fast” ethos of DevOps inspired delivery teams meets the “be stable, be secure, be compliant” mandate of traditional enterprise operations organizations.
Damon also identifies DevOps-inspired principles and practices being leveraged by high-performing enterprises who are currently transforming their operations organizations.
Requirements Gathering for a Successful Rugged DevOps Implementation
Hasan Yasar
It is a must to include secure coding practices in application development life cycles to produce rugged software. Yet, each organization’s development pipeline and application is different compared to others. It is also necessary to have preparation prior to having a successful implementation of rugged DevOps. This includes organizational culture, security policy, development platform, application technical stack, operational team involvement and foremost secure coding practices. The questions are: how to assess, what to find as bottleneck, train whom on what, what to measure and, finally, how to monitor. Then, build up your customized integrated DevOps platform where you can build a rugged application along with other quality attributes such as compliances, secure testing, performance monitoring.
The burgeoning concepts of DevOps include a number of concepts that can be applied to increasing the security of developed applications. These include adding risk-based architectural design, automated security testing techniques such as fuzz testing, software penetration testing to the software development cycle or the continuous integration cycle. Applying these and other DevOps principles can have a big impact on creating an environment that is resilient and secure. In this session, Yasar explains how his company figured the right requirements and then how you can utilize this in your own organization for Rugged DevOps.
So, you’ve adopted agile software development life cycle and your DevOps team runs continuous integration/continuous deployment. But what about security? If you haven’t figured out yet how to speed up your security processes , it can easily become a bottleneck and slow down the whole software development process. Gryb shows how to avoid this.
Gryb discusses how to replace old security processes with the new processes, and how to add more security automation tools that utilize existing quality assurance test cases.
Scaling Rugged DevOps to Thousands of Applications
Tim Chase, Aaron Rinehart, Jeff Williams
Most of the talks on rugged DevOps are trivial. Write a script to hook up your scanner to Jenkins, and your WAF to your SIEM, feed the results to developers via JIRA, and claim victory. If you have multiple tools, maybe push the results into ThreadFix. But very quickly your cup will runneth over with vulnerability reports that need to be manually triaged. If you try to scale, you’ll need a dump truck and an army. If you want to really scale rugged DevOps, you need to get the humans out of the critical path.
In this talk, this trio explores how large enterprises handle this challenge by instrumenting their application portfolio, assessing and protecting applications in parallel and integrations enabling instant notification directly to stakeholders. The result is continuous protection during both development and operations.
In this DevOps Chat episode our guest is DORA co-founder, Dr. Nicole Forsgren. Dr. Nicole talks to us about her upcoming appearance at the DevOps Virtual Summit on Tuesday, March 7th at 12:30PM EST.
Additionally, Dr. Forsgren gives us some news on upcoming activities and releases coming from the DORA team. It is always a pleasure speaking with Nicole and I am sure you will find this conversation useful and a good reason to hear her and Gene Kim at the DevOps Virtual Summit, along with several other former DevOps Chat guests!
In the meantime here is the streaming audio of our conversation. The written transcript will be available shortly.
Alan Shimel: Hey, everyone, this is Alan Shimel, DevOps.com, here for another DevOps Chat. Today’s guest is a special guest for our DevOps Chat, it’s Dr. Nicole Forsgren of DORA. Dr. Nicole, welcome to DevOps Chat.
Nicole Forsgren: Thank you, and thanks so much for having me.
Alan Shimel: Thank you. Nicole, we’re gonna talk a little bit about a Virtual Summit that you’re a part of, but before we do, I wanted to give our audience a little background on DORA. No, we’re not talking about the explorer, but it is—
Nicole Forsgren:[Laughter] We do some exploring, though.
Alan Shimel: – yes, you do, you do. But seriously, for people in our—and you know, we have kind of a worldwide audience, only 38 percent of our listeners are in the U.S., and there may be people out here who aren’t familiar with DORA. So why don’t you give them a quick background?
Nicole Forsgren: Okay, sounds good. So, D ORA is a company that is myself, Nicole Forsgren, and Gene Kim, author of the famed Phoenix Project, and co-author of The Visible Ops Handbook and The DevOps Handbook, along with Jez Humble, co-author of The DevOps Handbook and Continuous Delivery and Lean Enterprise. And together, we are also the lead investigators and primary authors of the State of DevOps Reports going back a few years now—several years now.
So we have spent several years researching and understanding the key factors that drive excellence in software development and delivery across the industry, whether that’s, you know, the famed unicorns, but also enterprises. And so what DORA does is, we take everything that we’ve learned and everything that we know across our—you know, a decade each, really, so decades of understanding, and we help companies measure and assess and understand and drive improvement in their own technology transformations.
Alan Shimel: Great. And, you know, so when I always hear the DORA story, Nicole, I am—I’m a child of the ’80s and the ‘70s before that, and I think of sort of a rock super band like Asia or something.
Nicole Forsgren:[Laughter]
Alan Shimel: You know? Where we’ve got all of these great kind of musicians coming together to make great music, and that is somewhat what DORA is, where we have three great—you know, three leaders of the DevOps community kinda joining heads. But it wasn’t just as you mentioned, it’s not just with the formation of DORA that you guys are doing this. I mean, there’s—I think it’s six years of the DevOps Survey kind of, or maybe this is the sixth year now, in the [Cross talk].
Nicole Forsgren: Oh, so it’s—yeah, well, it’s four years since I’ve been involved, and then I think, like, rubbing science on things, right?
Alan Shimel: Mm-hmm.
Nicole Forsgren: And it’s been academically rigorous since that time. You know, it’s, the years of study have appeared in peer review, right? So we’ve got some fantastic data behind us. Gene and Jez have been involved, I think, five years, and then it goes back another year or two before that, right?
Alan Shimel: Yeah.
Nicole Forsgren: And then Gene and Jez have been studying the space, I think, going back a decade, right? And then I’ve been doing academic research in this space since ’07, so yeah, we’re going on at least a decade, each.
Alan Shimel: Wow. That’s a lot of time kind of studying the subject matter and a lot of expertise in there. And I think that’s why, you know, personally, why I get excited to see what comes out of this, right, and what insights come out of the survey, of course. But DORA is more than just about conducting the survey, making sure it’s academically rigorous, making sure the numbers are, you know, bear witness to what they are, that they are, in fact, reliable and show things.
Nicole Forsgren: Right.
Alan Shimel: You’re actually taking that data and helping organizations as a result of the analysis of the data. So it’s not just analysis for analysis’ sake, it’s, how do you make your organization better by taking what we’ve learned?
Nicole Forsgren: Right. Exactly. So we do have, you know, The State of DevOps Report that comes out every year, and we make that publicly available, we make that free. We partner up with the team at Puppet, right, and we do have a report that comes out free every year, and we’ve got an ROI white paper that’s gonna be coming out in the next few weeks. So we do everything we can to discover new information and make that free every year. But what DORA also does is, we have a product now that helps organizations measure where they are, benchmark where they are against the industry, understand what they should do, what levers they should turn to drive performance improvements and then prioritize those improvements and those capabilities so that they can really drive improvement most efficiently, and then track those improvements, so they can really see the value come out of that improvement.
I love it. It’s super exciting, it’s super fascinating, because like I said, we do this because we love it, and because we wanna make things better. You know, we wanna have an impact on the industry. We wanna see our peers, you know, get better. Who was it, I always forget who said this, but you know, survival isn’t mandatory. [Laughter]
Alan Shimel: Yeah.
Nicole Forsgren: You know what I mean? Companies—like, you don’t have to do this. You can die, right? You can fail. Companies can fail, but that’s not what we wanna see. It’s so enthralling and it’s so exciting to go to conferences like Velocity and DevOps Enterprise Summit and hear these amazing transformation stories and see what is possible. And to be able to help provide these insights—and really, that’s the genesis behind DORA. You know, we were sitting down—Jez and I, and Gene—we were sitting down and saying, “Every year, someone comes up to us, and dozens of people come up and they say, ‘Help me understand. The report’s come out, and we want to understand how to measure and how to benchmark, but what should I do? Help me understand what I should do next.’”
And it’s just, it’s so difficult to be able to say, “Well, the measures are out there,” but providing companies and organizations and teams with tooling that’s just going to be simple and easy and straightforward to use so that they can measure, they can benchmark, and then so they can prioritize and understand which levers they should be using to really optimize and maximize their investments and their improvements is really exciting, and really kind of fulfilling, honestly.
Alan Shimel: Got it. Absolutely. So, Nicole, let us kind of pivot into talking about this Virtual Summit you guys have coming up. I think it’s next week.
Nicole Forsgren: Yes! I’m so excited. It’s so fun
Alan Shimel: So, go ahead. Tell our audience—I love it when you’re excited. Tell our audience about it.
Nicole Forsgren:[Gasps] I nerd out so hard. I’ve got the best job.
Alan Shimel: Uh huh.
Nicole Forsgren:[Laughter]
Alan Shimel: So this is, I wanna say it’s UVM, or maybe it’s not, I forget which—I know CA is a sponsor of it, but tell our audience about this Virtual Summit.
Nicole Forsgren: So, you know what? I’m super excited about this. I think the Virtual Summit is fun because I know so many people that are like, “Ugh, I wanna go to these conferences, and I just, either I don’t have the budget” or most of the time, now, people just don’t have time, right? Going to conferences is tough because the travel is hard, taking a day off of work is hard. This Virtual Summit is fantastic, because you can log in, you can listen in, and you can hear from so many amazing, fantastic people, and you don’t have to leave, right? Even if you have budget, getting on a plane and leaving—like, it’s hard.
And the Virtual Summit is great, because CA has been able to line up so many fantastic people. Aruna did such a great job. I’m super excited. I’m excited, because I get to meet so many more people and hear all these fantastic transformation journeys.
Alan Shimel: Yeah. I mean—
Nicole Forsgren: I’m so excited.
Alan Shimel: – that’s kinda what makes a lot of what we do, you know, the world go around, when we hear more and more companies—
Nicole Forsgren: Yeah! This is why my job is the best! [Laughter]
Alan Shimel: Yeah, I hear ya. I’m gonna bring up another thing, though, to you, Dr. Nicole, and that is this whole idea of kind of virtual summits, right? We saw it with the All Day DevOps event that a bunch of people put on last year.
Nicole Forsgren: Uh huh.
Alan Shimel: We’re seeing it now in this virtual thing. You know, for a long time—I don’t wanna be negative, but these virtual conferences just didn’t do it for me. But I think the technology’s gotten a lot better. I think the, you know, bandwidth and mobility and cloud and all of the things that help drive DevOps as well have led us to have better experiences in things like a virtual summit where you really can meet people, right? Even though you may not be in the room with them, it’s virtual.
Nicole Forsgren: Right.
Alan Shimel: But it’s really a great experience.
Nicole Forsgren: Well, do you know what? I think it’s a combination of things. I think some of it’s the technology. Do you know what else I think it is? I think it’s also, like, the special magic sauce behind the DevOps community. It didn’t always work for me before, because the other communities just have a different underlying culture. DevOps is just a little bit different, right? I’ve been around tech forever, right? Before my Ph.D. I worked in hardware performance, where I, like, I’ve got a hardware patent, I—I worked in hardcore tech for a really long time. And those communities are, they just have a little bit different DNA.
DevOps is very, very different. And so for anyone who’s gonna tune into this summit—participate, pay attention, reach out. Add me on LinkedIn, right? Everybody say hi, wave—love you all. Add me on LinkedIn, add me on Twitter, follow me, send me a message. The thing I love about the DevOps community is that everyone is in it for the community. Everyone will reach out. So many people legitimately, honestly want to help everyone else get better. And it’s something that I just haven’t seen in very many other communities. It’s just so different.
Alan Shimel: You know, I was—
Nicole Forsgren: And I think that’s what makes so many of these virtual summits work so well.
Alan Shimel: I agree with you. Nicole, I just saw you, I guess it was two weeks ago now or last week—no, it was two weeks ago at the RSA event—
Nicole Forsgren: Oh, yeah, uh huh.
Alan Shimel: – where we did our, you know, we did our Dev SecOps show there. And what was funny is, I got an e-mail from RSA—do you know there were 43,000 people at RSA this week?
Alan Shimel: We—yeah.
Nicole Forsgren: My word.
Alan Shimel: We—that’s crazy.
Nicole Forsgren: And I didn’t get sick—it’s amazing. [Laughter]
Alan Shimel: Yeah! Well, I did, but—you know, we had about 900 people at the Dev SecOps event throughout the day on Monday, and then the cocktail party and all that. But what you said about the DevOps community so rang true for me. Even in this see of 43,000 lives out there, right, the DevOps people kinda gel. They were very into being together and sharing and talking. And it’s contagious to my security community friends, who, you know, they—I can’t tell you how, and I’ve been in security for 15 years—how many came over and they loved the vibe that we have in the DevOps team.
Nicole Forsgren: Oh, yeah!
Alan Shimel: And they wanna be part of it, and that’s very different, Nicole, than three to four years ago when I first started doing that at RSA conference, and it was very much a butting of heads, right?
So yeah, it is a testament. It’s a testament to, you know, we talk about culture all the time and a lot of people think we’re crazy, but it is a testament to that whole culture and vibe around the DevOps community.
So I’ve said my piece. I’ll step back off my soap box.
Nicole Forsgren:[Laughter]
Alan Shimel: So, Nicole, for people listening in on the Virtual Summit next week, I mean, Gene’s talking, obviously, I’m sure it’ll be interesting and he’ll give us some insight into some, maybe, new transformations he’s been studying and analysis. What do you—any previews of what you’re gonna talk about for our audience?
Nicole Forsgren: So I’m gonna be talking about some of the trends that we see, some of the promising things that are coming up. Some of the big—the levers that you can turn that I think are exciting, I’ll be speaking to some of the things that Ops teams, you know, can really contribute that are important.
Alan Shimel: Mm-hmm. Very cool.
Nicole Forsgren: And honestly, I’m also—you know, I’m super excited to hear from some of the other speakers. I’m really excited to hear from Sylvia at United, Kelly at Jewelers Mutual, John at CNN—super excited for that.
Alan Shimel: Yeah.
Nicole Forsgren: Right? They’ve got such a great slate.
Alan Shimel: They do, they have a nice slate and, you know, like you mentioned, Aruna from CA—I mean, CA has a great customer bench there.
Nicole Forsgren: Oh, God. I tell you what, man, Aruna has so many fantastic insights. Anything she comes up with, I just love—super excited to hear what she has to say.
Alan Shimel: She’s actually been a guest on our DevOps Chat regarding her book. So here’s—
Nicole Forsgren: Yeah! So great.
Alan Shimel: – between you and me, I always have a crush on Aruna, because I just, I really—I mean, she’s got it goin’ on, you know? She’s written her book, she’s an engineer, she’s done the business side, she’s full of energy, and I—you know, I just, I tell her all the time, man, if I had a daughter, I’d want her to grow up and be like her. Because she—
Nicole Forsgren: I know. I have a serious intellectual crush on her. She is a mover and a shaker, man. She is doin’ some good stuff.
Alan Shimel: Yeah, that’s good stuff. I’m really, really proud to call her a friend. Anyway—I’m really proud to call you a friend, though, too, because you’ve met my son, and I say, “You need to go talk to her. You wanna do statistics? There’s someone to grow up like.” But anyway.
Nicole Forsgren: That’s right.
Alan Shimel: Dr. Nicole, I’ll be listening, obviously. I hope our audience will be. We are all watching what DORA is doing, and I guess the new report—well, the survey’s closed now, right, so the new report is [Cross talk]?
Nicole Forsgren: Yep. So stay tuned, the report will be coming out early June.
Alan Shimel: Very cool. We’ll be looking forward to that. That’ll probably be around when DevOps Enterprise Summit is in London.
Nicole Forsgren: Close. DOSE—oh yeah, DOSE London is June 5th and 6th. Also, DORA’s coming out with an ROI white paper and then summer, fall, me and the guys, Gene and Jez, are coming out with a book, summarizing all of the research that we’ve done.
Alan Shimel: Ooh! You heard it here first.
Nicole Forsgren: Including—yes—including things that did not fit in the book, surprises, and more on the science [Cross talk].
Alan Shimel: Very cool. I’m looking forward to it. It’ll be an exciting summer.
Nicole Forsgren: Thank you.
Alan Shimel: I think we might be doing some book giveaways here. [Laughter] But until then—hey, check out Gene, Nicole, Aruna Ravichandran from CA and a bunch of other great practitioners at the DevOps Virtual Summit coming up. In the article on this, we’ll probably have links to the registration and so forth.
Dr. Nicole, it was so great speaking with you, and continued success with DORA and all of your endeavors. We wish you lots of success, and we’ll be watching it closely, and maybe we’ll have you back on.
Nicole Forsgren: Thank you so much.
Alan Shimel: Thank you. We’ll have you on to discuss some of this stuff again shortly.
Nicole Forsgren: Sounds good.
Alan Shimel: Alright. Dr. Nicole Forsgren, co-founder of DORA and guest on the DevOps Virtual Summit coming up just next week. Thank you for appearing.
This is Alan Shimel for DevOps.com, and we’ll see you soon on another DevOps Chat.
