Category: Enterprise DevOps

  • Small Teams Focused on Delivering Outcomes Produce the Best Results

    Small Teams Focused on Delivering Outcomes Produce the Best Results

    The days of big team consulting are over. Technology organizations are aligning their budgets to get projects done faster and more agilely. More applications are being built by small teams of cross-functional engineers moving fast, providing higher quality code, designing away waste and cutting out processes that don’t add value.

    It’s becoming apparent that engagements incorporating smaller footprints and shorter iteration cycles are more common. The best outcomes are where an engineering team is small, lean and iterating many times faster than much larger teams. That means that enterprises get their applications to provide value earlier and at lower cost, and they cost less to maintain over time, starting right away. It means applying higher quality and better managed teams using core engineering principles. It means cross-functional technologists, not cross functional teams working in silos in their specialties

    The catch is: How do you incorporate that at enterprise scale? It’s not easy to do because there is a lot of internal resistance and inertia. And many organizations will not achieve this.

    Agility and priorities are key here. Future thinking organizations don’t waste this effort on trying to knock out backlog. They make sure they’re applying themselves in a targeted fashion and they drive velocity higher on things that matter. It’s not about providing spot fixes. Don’t start with building a CI/CD pipeline or an AWS migration. Work with your engineering team on driving down the overhead that’s slowing down the delivery of your platform.

    It’s not about using this tool and that tool, and this programming language and that platform to host it, and that it will integrate a certain way, and you are now able to say that you are delivering in a continuous integrated fashion. You should get continuous delivery set up. However, continuous deployment is one step. What you really want to do is reduce costs by 20% and reapply that to a feature that you’re looking for next quarter, instead of waiting till next year. You can restructure the conversation because from the get-go you front load the delivery strategy with those kind of performance terms. And this is real money. We are applying these processes where customers are paying for software to be delivered.

    What doesn’t work is “Let’s do some DevOps over here, and then maybe a little over there and make incremental progress.” Spend resources doing principled engineering all in the same application—not bits and pieces in various places and hope to bring it together at some point

    One way to know if you are doing DevOps—or any cloud related engineering—better is that you have a better grasp of Amazon’s APIs than Amazon does. Literally, the best companies release updated APIs for Amazon before Amazon releases their new APIs. That’s how good they are.   

    Different organizations move at different rates with an understanding of DevOps. Some people still say “Get me a DevOps engineer.” Others say “Get me a DevOps’ culture change,” which is OK. But, really smart organizations are asking for small engineering teams to build high performance, high priority platforms quickly, at the lowest possible cost, using the best engineering principles available.

    One other way to know that you are doing good engineering and DevOps is when you need DevOps engineers and you talk to a big consulting outfit and they say, for example, that you need 65 SREs with different skill expertise, four project managers, a bunch of data scientists and product people and client partners. They say it’s going to be really expensive and that they’re going to be here forever, and that this is an engineering challenge. Let’s address it from that POV. The engineers are running it, the engineers are closest to the work, they understand the issues and the opportunities, and they just need 10 really good full stack engineers.

    The name of the game now is applying engineers that clearly have a specialty, but are more importantly good engineers with software and know how to write code well. You might have a platform engineer, which is going to be like your infrastructure SRE but he’s going to be knowledgeable and know how to write code at a basic level. That will work. You might have an engineer that isn’t the best software developer, but knows platforms, what clouds are available, how to offer it across those programmatically, etc. That will work, too.

    The bottom line is moving initiatives that are focused on building revenue or saving a lot of money and driving them forward at a higher velocity, with higher quality delivery at their product launch through foundational concepts, such as continuous integration, continuous delivery and continuous deployment. Leading organizations are focused on doing that in a way that is future-forward, focused on stateless and serverless infrastructures.

     

     

    This article was co-authored by Jack Keck. Jack is a seasoned technologist leading a cross-functional engineering organization at a global investment bank. He spearheads transformative initiatives with the goal of raising revenues through the use of future forward technologies.

    — Larry Gordon

  • What the Best Developers Do to Be Great at Their Jobs

    What the Best Developers Do to Be Great at Their Jobs

    Good developers are in demand—great ones, even more so. With software development employment in the U.S. expected to grow by 21% by 2028, the need for engineers is only going to surge further.

    But this doesn’t mean that the labor landscape will become uncompetitive: Now more than ever, individuals with a passion for software are entering the field, and organizations will only be looking for the best. As the amount of people going into software development goes up—with global numbers expected to reach 27.7 million by 2023—it will become increasingly important to demonstrate your skills and dedication to the profession.

    The gap between a good developer and a great one is bigger than you may think. Here’s what the best developers do to excel at their jobs and lead in their field.

    The Best Software Developers Know How Their Skills Fit Into the Big Picture

    While a good coder handles the responsibilities they have within their role, the best coders make sure they see the big picture. They take the time to understand the foundations of how businesses work, including revenue, profit, bottlenecks and the strategies of departments other than their own. 

    This might mean investing extra time into general business self-education or holding meetings with other departments, but great programmers know it’s worth it in the long run. They recognize the importance of grasping how the cogs of an organization come together to keep things running—and exactly how they fit in.

    By having this 360 view, they can hold influence in areas outside their remit and clearly match the business’ mission to their own work. Valuable coders will use their problem-solving mindset to propose solutions to company-wide problems and put themselves in the shoes of other team members.

    Rest and Relaxation Is Vital for Productivity

    Far from the stereotype of frequently pulling caffeine-fueled all-nighters, the best developers know that, like the powerful systems they work with, they also need some downtime to recharge.

    Great coders challenge the stereotype of having a bad diet and inactive lifestyle and prioritize their own health and well-being. They recognize that as soon as you pass the threshold of 50 hours of work a week, you’re no longer productive. In fact, working too much can be counterproductive, as you’re much more likely to make mistakes.

    Great developers know when to shut down their computer and spend time with their families, do some exercise, spend time in nature and eat a healthy meal. This time to recharge is essential in order to go back to work refreshed and ready to tackle any problems with a clear and rested mind.

    Continuous Learning Is Crucial for Progression

    Ideally, all developers should be making efforts to advance their education in the field they’re in. As a developer, if you stop studying, you’re bound to get stuck at one point or another. The best coders make this a special focus in their lives, and make sure they’re always up to date with the latest developments in the technologies they work with.

    It could be a new coding language, new frameworks or a new technique. Great coders are constantly making a concerted effort to advance their own education. Ultimately, they are always asking “why,” and are unafraid to experiment using their newfound skill.

    Developers can easily find online courses to supplement formal education. Taking part in hackathons is another way to boost extra learning, as well as keeping up with technical blogs and podcasts.

    Great developers also follow other passions outside of coding. For example, the logical nature of software development allows coders to excel at learning other skills, such as languages, playing an instrument, martial arts or even flying a plane. It enables them to exercise the same part of their brain while enjoying something fun and non-work related.

    Personal Projects Show Dedication and Independent Thinking

    Coders know that the best way to learn how to build great products is through practice. Even if the project ultimately fails, building a product will always provide important lessons to apply the next time.

    Side projects that a developer might work on include personal tools, open-source libraries, a startup idea or a freelance project. These personal projects allow programmers to be in the driving seat: They are forced to solve problems independently instead of asking for help, equipping them with skills that might otherwise take longer to gain.

    Engaging in independent work also allows developers to explore and utilize the new languages and frameworks that they gain in their personal education efforts. When developers that have spent time working on their own projects try to prove themselves in the job market, they will stand out against those that have remained largely within the confines of their professional role.

    Ultimately, the vast majority of developers are in the game because they have a passion for building exciting and useful products. This makes it easier to find those who go the extra mile to really hone their existing skills and learn new ones. But, like in any profession, there will always be a certain percentage that remains a cut above the rest. Take a holistic approach to your career in software development with this advice and raise your game, and chances of landing that dream job.

    — Phil Alves

  • EP 6: How to Lead a DevOps Transformation

    EP 6: How to Lead a DevOps Transformation

    Is C-level support required to have a successful digital transformation? In the sixth episode of DevOps Unbound streaming broadcast on TechStrong TV and DevOps.com’s sister site Digital Anarchist, Mitchell Ashley of ASG and Alan Shimel are joined by Eveline Oehrlich of the DevOps Institute, Louise McCarthy of Bain & Company and Mark Settle, author of “Truth from the Valley, A Practical Primer on IT Management for the Next Decade,” to discuss DevOps and digital transformation at the C level.

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript

    Alan Shimel: Hi, everyone. Welcome to another edition of DevOps Unbound. DevOps Unbound is brought to us by our friends at Tricentis. So, thanks for their sponsorship. But, you know, they sponsor but we pick the topics. We pick the guests. And we have a lot of fun here talking about relevant topics around DevOps. Today’s topic centers around DevOps and digital transformation at the C level. And let me introduce you to our panel. First of all, we have Louise McCarthy. And, Louise, I’m gonna ask you to kinda introduce yourself with a little background for our audience.

    Louise McCarthy: Yes. Thank you, Alan, and hi, everybody. I’m Louise McCarthy. My background more recently is doing large, complex digital transformations mainly for the financial services. And mainly through the eyes of the business rather than the technology, although I have held roles as CFO and I’ve held roles COO and CIO as well.

    Shimel: Excellent. Next, we have Mark Settle. Mark, welcome.

    Mark Settle: Thank you. So I am a seven-time CIO and I most recently left Okta about a year ago and have been advising startup companies since then. I’m based here in Silicon Valley. And probably my encounters with DevOps are focused on supporting DevOps communities in terms of infrastructure requirements if they have, and also introducing the concept in actually business application support within a conventional kind of an IT group.

    Shimel: Thank you and welcome. And then Eveline Oehrlich, Eveline, please.

    Eveline Oehrlich: Yes, hello there. Eveline Oehrlich. I’m the chief research director at the DevOps Institute. Before that I spent 13 years as vice president at Forrester Research studying a variety of topics, and DevOps of course was one of them, working with many enterprises and C levels and others to explore the challenges and the fun of DevOps.

    Shimel: Very good. I’ve never heard anyone talk about the fun of DevOps but I’m looking forward to discussing that. And then, last but not least, of course is my cohost here on DevOps Unbound, Mitch Ashley of Accelerated Strategies Group. Mitchell? 

    Mitch Ashley: Good to be here. I’m Mitch Ashley, CEO of Accelerated Strategies. We focus on digital transformation, DevOps, cloud native, and cybersecurity. I’m also a former CIO. Not nearly as experienced as Mark is, but have led transformations and also implementation of DevOps as well as been a CTO, CEO of product companies. So I’ve lived on both sides of DevOps. 

    Shimel: Great. What a great panel we have today. I’m really looking forward to it. So we’re talking about DevOps at the C level. As I mentioned to you all before we started today’s show, our friend Gary Gruver wrote a book, “Leading the Transformation.” He wrote several books. One of ’em was “Leading the Transformation.” And Gary is a big proponent of what he calls the top-down approach to DevOps and digital transformation. And in Gary’s view – and I’m not saying right or wrong. In Gary’s view, you cannot have a successful digital transformation or DevOps transformation without sponsorship, support, air cover, if you will, from the C level, including the CIO or CTO, depending on the organization. CEO, et cetera. 

    Let’s start there. Is C-level support absolutely positively required to have a successful transformation? Anybody?

    [Crosstalk] 

    McCarthy: Do you mind if I go first, Alan?

    Shimel: Sure.

    McCarthy: I would actually say, my recent – I’ve done about five or six transformations and tried all sorts of ways to actually get them going. There’s always blockers on the way and it’s usually the blockers that slows the whole problem down. And the blockers either come from the ground up, from the employees or the people that run the processes today, or you get the blocker in terms of the C-level-suite people who tend to wanna own things, and it’s a big of ego that actually blocks things going that way.

    And the most recent transformation was successfully owned by the C-suite. So they drove it. They owned it. They were the stakeholders. But most of the engagement actually came from the employees, from the process owners. So we ran the whole transformation from the process owners actually owning and identifying where the transformation needs to take place. And that really made it hugely successful because you didn’t have people fearing for their own jobs. ‘Cause with digital transformation it always means or often means that there’s gonna be heads that unfortunately are lost in various areas of the – in the transformation. But mainly doing transactional-type work. But if it’s the transactional people that are running the processes, they highlight their pain points and then we work with them to actually change those pain points into something positive. 

    So, I would arguably say both from top-down, but also if you don’t do it with engagement of the people running the processes today, it won’t work.

    Settle: I’ll chime in. I think I have the same answer but I’m gonna state it in a slightly different way. I think the way the question gets asked, it sounds as if the CTO or the CIO’s gonna come down from the mountain with the DevOps tablets and then ____ employees and team members _____.  “I’ve been converted and now it’s your turn. We’re all gonna go do this together.” [Laughs]. Most of the time that doesn’t work. For one person to come in – well, take an example where a startup company here that’s been reasonably successful. Maybe they get up to a product team of, I don’t know, a generic team of 500 people or something like that. They hire a new chief product officer or a chief technology officer who says, “We’re gonna do this completely different. I have a vision.” Et cetera. Well, they’re gonna do it but that’s gonna be a slog to kinda get there.

    What I’ve seen be more successful is almost like – what’s the right term in a battle when you send the beach guy out? The beach master that’s moving people around. So you really want almost like a fifth column movement down at the grassroots level of people who are genuinely interested in exploring these ____ ways of doing business. And then I think what the C-level person adds are the methodological resources to drive scale. And so many DevOps initiatives fail because they sound interesting; they sound contemporary; people think, “Other people do it so it must be beneficial.” 

    But then after like 18 months, everybody scratches their head; “Where are the benefits? We thought this was gonna be transformational.” It’s not. The technical people are having fun. They’ve developed this new terminology, et cetera. And if you scratch below the surface, well, guess what? You’re doing DevOps in like five different ways because the little C-groups have all picked the tool they wanted and decided that they’re gonna have a daily scrum twice a week or whatever. They’ve got their own little process set up. And then people say, “Where’s the benefit? This was supposed to be transformational.” So at some point in the early stage of the journey, it’s needed to drive scale and ____ of endorsement. But I think it can almost backfire if it’s launched from the brain of Mark Settle that, “Yea, and verily, I’m gonna lead you to the promised land.”

    Shimel: Sort of a Charlton Heston beard I’m picturing on you there, Mark. Eveline, how about you?

    Oehrlich: Yeah. I actually would say it should be outcome-driven DevOps. It doesn’t really matter if it is from the C level, from the individual, from the inside-out, from the outside-in, from the top to the bottom, the bottom to the top, from the middle to the whatever. It needs to be outcome-driven. What I mean by that is: Only metrics will make us change. Just like I drive through my town here – this is a little analogy. I drive through my town. They installed one of those things where they measure the speed and if you go over 50 kmh you get a ticket. Well, my husband, even though that thing has been there for two years, my husband manages to get a ticket every three months. But what he does is he intercepts the mail and then he doesn’t tell me that he has gotten that ticket. 

    So now what I have done to stop those 15-euro tickets every two months – because if you add it up you can go for a nice dinner – I am going to go through the mail before he gets home so I see how many tickets he gets. And he gets little lines. And that metric is on the refrigerator – I should’ve brought a picture of it – and that has actually reduced how he is driving through town.

    And so if you go back to the DevOps teams and the digital transformation and you ask the individuals at C level or VP level or practitioner level, “How successful have you been or what’s the challenges around?” It’s easy to point to somebody. But data, if you have data which shows where you are and it maps to where you want to go and you share that amongst all these different individuals, there is no argument. It’s not the guy with the long beard, as Mark would come in, or it is not somebody at the lowest level who could argue with that. We could huddle around it and figure out: What do we need to do?

    So I think it’s not really about who – of course we do need some individuals who pays for it. We need some kind of a fund and budget and all of that. But I really think it’s outcome-driven rather than organizational-level-driven.

    Settle: Can I just chime in on that one, too? So I would agree 100%. And outside of maybe the CTO or CPO or IT organization, bringing evidence of improved sophistication, operational metrics around the process to almost any other C-level executive – they don’t really care. The CFO – if you show that, I don’t know, the year throughput, the story whatever – in one of my books, it’s like reading a Bible to the cat. The cat will look at you and feign interest. But it has absolutely no idea what you’re talking about. And that’s sorta what you get outside the technical functions, everybody – “But that’s your job. You figure out what needs to get done.”

    And, again to your point, it’s more of a business outcome. Like, are we actually getting customer request enhancements through in greater quantity or are we able to respond to a competitor’s new feature set faster than we did at their last result cycle? Or whatever those things are. That’s what people wanna talk about.

    McCarthy: Yeah. And – [Crosstalk]

    At the end of the day, the outcomes is aligned to the strategy of the organization. So you’ve gotta be doing a transformation for a reason. And there’s usually three or four reasons why you’re doing a transformation. It’s around the customer journey so the customer needs to improve the satisfaction. You’ve gotta costs. You’re trying to grow your revenue. And then the one that’s usually an outliner that’s become ____ at the moment is carbon emissions. So for every one of these paint points that I mentioned earlier, there should be a business case that shows measurement of success against each of those four measures. And that’s what you – bottom-up, you report at top level to the C suite to say, “This is why we’re doing it. This is the measure of success of what we’re doing it.” And even breaking some of the pain points and the business cases down to quick wins. So ones that you can be getting on with relatively quickly that free up cash that then can pay for the more strategy-type things.

    Shimel: Yeah. So I think this brings up – all three of you have touched on it. And it brings up what I think is really sort of an entry question, which is – it’s the genesis of digital transformation. Why and how does any organization decide to go through the potential pain, heartache, and change of digitally transforming, of adopting DevOps or agile? Louise, I’m pretty sure we spoke about this – was it at an Emmet Keeffe Insight IGNITE in Iceland that you were at?

    McCarthy: Yes.

    Shimel: I was there with you and I remember talking to you about this. You know, Eveline, in your day, someone like Mark might go to a Forrester conference and come back and say, “Wow, Eveline Oehrlich from Forrester just presented this presentation study on the benefits of digital transformation and DevOps adoption. And, god darn it, she described us to the T. We really need to do that here.” And maybe that worked then, right, seven years ago, ten years ago? I don’t know if that’s the genesis of digital transformation today. I think a lotta times digital transformation today does start with technical champions lower down whispering in Mark’s ear that, “If we were doing DevOps or agile or whatever, we could really make better customer experiences. We can give customers what they want. We could do it better, cheaper, faster.”

    [Crosstalk] 

    Ashley: I think we have a chasm that we have to cross, and that is: DevOps can be sort of a vanity project for the IT or the development organization, right? It can be a purely self-interest: “We’re doing what we do better.” Which is always a good thing, but at the end of the day does it move the needle for the business? You have to be able to translate that into what Eveline said was the why. Louise also said the why. But what is the real benefit we’re trying to get out of it? And when you have a purpose for doing it, now you have metrics you can put around that. Is it revenue? Is it competitiveness? Is it responsiveness to the market? Is it changing customer expectations and behaviors? What is it? That’s truly – then the development or the IT organization knows that they’re making a difference. 

    Otherwise how do you know whether you should be working on more releases per hour, per day, or you should be working on just fixing the backlog? How do you know? You don’t other than just sort of your own internal metrics. And I guess it kinda reminds me of that phrase of the – what was it? Belly-button gazing, right? You’re sorta looking at yourself and what you think’s important. But I think that connection – you have to cross that chasm, whether the business is coming to you or you have an idea of what we can do better and how it might help, and then let’s then make that connection. 

    Shimel: That’s my point. Someone like Mark – Mark, what does it take to push you over the edge and say, “God darn it, we gotta do this; we’re committed”?

    Settle: Yeah. Let me _____. One thing’s funny about this whole line of conversation is of course, in the world that I live in out here, there’s no question about doing it. It’s like: Who is the most sophisticated – [Crosstalk] or more sophisticated? So, in the war for talent, that same 500-person organization I referenced – they might look at each other and say, “Who do we know at Google or Facebook, in their engineering group? We know we should be doing this better. We’ve kind of played around with the concept. We’ve hired some lower-level people who’ve said, “You guys are just behind the times here. So we need some real leadership inserted.” So that’s the other side of the coin. In parts of the country where maybe it’s like an accepted practice, there’s almost a competition to be the most sophisticated or get the most benefit. 

    But the spark plug, the question you’ve asked – I’m thinking about some second-tier markets – and I really hate to call out specific locales, but what the hell? As a consultant, you can talk like that. So if you’re in a place like St. Louis and you’re in an organization that just has done business the same way forever, all the IT management, all the technical management has come up through the ranks kinda doing things the old-fashioned way, quarterly releases, and they don’t keep the same kinda metrics that you would in a DevOps environment, to blow a place like that up – there’d have to be some kind of competitive imperative.

    Or, you know, the white knight does show up from time to time, right? So if you think about it, sometimes when the new CPO or the CTO comes in – and I’ve experienced this in my own career – there’ll be a cause or a initiative that I promoted during my entire tenure – the new guy or gal comes in and they get it done like in a year. And I think, “What the heck happened here?” 

    There was one company where I was trying to get them off of Microsoft Dynamics, get ’em onto Salesforce. There was another company, I was trying to get production systems up into the cloud. And we did every kinda seed activity you can imagine. The cloud one, we sent people up to Redmond for some AWS instruction. We did some prototyping, et cetera. New CIO comes in and says, “We’re going to the cloud,” 12 months later like half the production systems are in the cloud. Like, “How did that happen?”

    So sometimes change at the top – which gets us back to where this kinda started – maybe that is the seminal spark plug. And especially if that person’s been brought in because there’s a broader C-level recognition that, “We’re not keeping up with the times; we need to make some kinda change.” So you have that broader support. When people start to whine, they know that the whining’s not gonna get them out of the change that’s about to come. 

    But for just self-generated change for people that’ve – then I go back to, you know, Mitch, where you said: then you can always have a vanity project which you can point to as evolutionary. Like: “We’ve got a small group. They’re putting their toe in the water. They’re doing some interesting things. So we are keeping up with the times.” But then you just don’t get the scale benefits at the end of the day. 

    Oehrlich: I think there was – just to add to something to what Mark said, as the ecosystem of our organizations has expanded, we have to think of – well, the travel industry doesn’t exist at this point in time, but think of maybe manufacturing, right? In the area where I am in, I’m in Stuttgart, we have large car manufacturers, and they’re connected to Bosch and others, and the ecosystem is rather broad. And these folks at all levels are collaborating, connecting. Not as much now with where we are, but they have exchanged thoughts and ideas, they go to events, and they learn from each other. At all the different levels. And so I think there is that – as Mark said, to some extent there is this – not really a competition, but the exchange of ideas, which has brought additional adoption of DevOps at all levels, right? 

    So if I am the C level of Bosch and I go to a genomic car-driving event and I meet the CF C level of the CF industry down in Friedrichshafen and they are telling me about, “Wow, we really improved our reliability scores and we have really punched up our cadence around the product X, Y, and Z,” I go back and I say to my team, “Hey, I just met this team over there. They’re doing something fantastic. Can we start?” It’s that spark, which to some extent happens to cross over in this ecosystem as we start to be much more connected and dependent. I think that’s one point.

    The other one is: if you look at the individuals who are today in IT, they don’t look like this anymore. I’m a Cobol, Pascal, HP 3000, JCL person. That’s when I grew up. That’s my heritage in operations. The guys and the girls today in these teams – they are completely different. They want Python. They wanna do sexy things. They wanna do fun things. They wanna show what they can do. And they’re looking for inspirations from their leaders as well, and they want to see that they can make a difference. I think that is also, from a cultural perspective, I think – of course it’s important but I think that’s also a driving force. And that’s not just the individual contributors but also the consultants and the C levels who are – I call them the flip-flop-wearing, chino-looking guys who go to the conferences, which we’re seeing them; I’m like, “Oh, I think I’m overdressed with my suit.” That’s the people who actually make DevOps happen, and they wanna have fun. They wanna do things. They don’t wanna do JCL and things like I did in a boring ops group dying of boredom.

    Ashley: I think you’re talking about a couple of things, Eveline. One is the attracting talent, right? People have more selection, economic conditions aside, more selectivity about the kind of work that they wanna do, the companies that they wanna work for. There’s also just: how do you learn to develop software? If DevOps is what you’re coming into, a way of developing software, you’re looking for organizations and you expect that approach. So I think your point is right on. It’s also the audience of the people who are gonna be the adopters or the embracers or the champions of doing this. So that all makes a big difference. The command from on high, as Mark was talking about, the tablets – there’s an expression for that: “This too shall pass,” right? That’s the reaction, usually, of those kinds of initiatives. You have to have an option. And I think you spoke to that really well.

    Shimel: Look, if you’re a Silicon Valley cool kid or you work at one of these organizations where flip-flops and jeans are the rule and you do the hair thing and all of that – I don’t do that so well anymore – that’s cool. But, Louise, you’ve done transformations at what we may think of as large, stodgy financial organizations, right? And they’re not Silicon Valley-based and they’re not a jeans and flip-flop culture. So how does it play there?

    McCarthy: Actually, funny you were talking about flip-flops, high fives and cappuccino. I worked in government actually for a little while, the tax authority, and that was a time when the government was setting up its ____ global digital authority. And everybody in the government, all of a sudden they change from suits to wearing flip-flops and high fives and cappuccinos, and it was like a world away from the starchy government.

    But, anyway, moving on to the financial services, yeah, the financial services – I’ve been lucky enough actually to do transformation in legacy financial services, and the big boys like the HSBCs, for example, which is complex. It’s like turning the Titanic around. And I’ve often written articles around: it’s like moving deck chairs around. The reason why I say that is because automation is the way forward. We talk about automation in many guises. Automation takes out people. It takes out transactional-type activities. But the legacy banks just struggle to do it. So I say it’s like moving the deck chairs around the Titanic because of the legacy system they’ve got. Transformation – they talk about it. They move things around. But nothing real and concrete actually comes out of their transformation, which is really sad. Because there’s great opportunities there in the legacy banks. 

    And on the other side of the fence, I’m working currently with a contender bank, a digital bank, which is new. It’s new. It’s fresh. It’s green. It’s almost as green as your screen there, Alan. It’s a green ____. And it’s great to be able to take the whole end-to-end customer journey and see it digitalized. And how quickly you can change and transform. And developing and changing technology with that suite, the same thing end to end, it’s just awesome to be able to see how quickly it happens. Whereas the legacy banks are still thinking about planning it, talking about it, cogitating over it, when the small guys are out with a new technology. 

    And one of the things that I’m often transforming whenever I do my transformation – it’s not transformation, it’s Christmas; it’s for life in my mindset. So it’s around continuously thinking about the innovation and it’s about encouraging these legacy banks to have innovation departments that are constantly thinking about the new technology that’s out there to take them ahead of their competitors. So like these digital banks, if the legacy banks are doing this, they’re actually seeing a way of making it happen quickly. And I think that’s the problem that we’re constantly battling: the bureaucratic slowness of moving Titanic.

    Shimel: So clearly the kind of organization you’re at determines, or will have a huge impact on this. Guys, we’re coming on the back side of our time limit here. I wanted to talk a little bit about: okay, we’ve made the decision. We are moving forward with our transformation. We are adopting DevOps and agile and these new ways. How at the sea level do we – short of coming down from the mountain with the two tablets in God’s own writing, how do we translate that to the troops? How do we effect the organization? That this isn’t some passing fancy of the CIO and it’s not gonna pass in the next quarter. This is really the way we’re doing it. How do you make that change down the ranks, get people adopting as champions and being successful with it? Mark?

    Settle: You know, it’s not unlike an initiative around security. When you talk about cultural change within an organization, I think leaders don’t always appreciate what a huge impact they have on the people that they lead. You know, people take cues from the way you use your time, the things that you show interest in, things you follow up on. Your time management sends tons of signals to the organization. And some of the operational metrics that I poo-pooed before as being of no interest whatsoever to people outside the technical areas are a great way to kind of keep score. You go back to what Eveline said: If there’s a sense we’re making progress, some of those ____ celebrations can be really important.

    I’ll just leave you with a thought. I worked with a guys several years ago who used to say, “What interests my boss fascinates me.” As long as there’s some sustained interest – and to go back to, Alan, what you said before: I can just come back from the Forrester meeting all jazzed up, and then two months later I go to a Gartner meeting and now I’m on a whole different tangent. Now I think it’s automation; we gotta get RP robots in here. That’s gonna blow the business up. And then people in the trenches are like, “This too shall pass. He’ll go to another meeting. But I’m not gonna get my –

    [Crosstalk] 

    Shimel: Come back with some other toy, yeah. 

    [Crosstalk] 

    Settle: “He’s so crazy.”

    Shimel: Yeah. I used to feel it was whatever was the last book I read. That was the next greatest thing to me, right? ‘Cause I just read that book. But, Eveline, fundamentally – and this is something the DevOps Institute is deeply ingrained in – it also becomes a question of: how do we upskill our workforce? How do we prepare – do we just go out and hire new people? That’s hard. Do we upskill our existing people for this? At the C level, how do we effect that?

    Oehrlich: Yeah. So, from our research at the DevOps Institute, we know that most organizations are actually preferring to leverage their existing resources to train them and to upskill them. Now, upskilling of course has its challenges because you don’t just upskill around processes of learning a new way of doing continuous delivery, or it’s not just about upskilling new technical skills. It’s also about human skills. That’s I think the challenge. A lot of organizations are biting themselves into that – or pushing themselves, or actually locking themselves into the corner. They think, “Yeah, let’s give them AWS training. Let’s let them understand what the latest DevOps platform is and give them some language skills around whatever the latest hotshot technology programming language is.” 

    But it’s also soft skills or human skills which are necessary. People need to understand how to work empathetic, how to collaborate, how to connect, how to cross functional borders, how to understand when a security individual tells me something. So having skills around these different categories is important. And there is investment needed. We see that, unfortunately, 7% of survey participants we had last year said that they have no idea what their upskilling programs are. We know 37% of these individuals – a very large sample size of almost 1,200 – said that, “We don’t have a program in upskilling.” And the rest of them either are planning one or have one. So that’s really unfortunately poor and not enough. So, anybody listening in, get ready to upskill your people. Start thinking about training. Start thinking about exchanging knowledge and allowing them to learn. Give time.

    I, for myself – and we’re doing that at the DevOps Institute a little more, and our CEO does that a little as well. I call Friday my learning day. And it doesn’t matter what happens. Unless the CEO wants me there, Friday is my learning day. I learn French. I learn about design thinking. And I’m learning about metrics. That’s it. And we all have to do that. It’s very important. 

    Shimel: Agreed. Louise, you go in as a consultant. In terms of the skill level of the workforce, do you try to upskill them? Do you supplement it with fresh talent? It’s hard getting fresh talent in. Mark, you know in the Valley they cut each other’s throats for this. How do you do that?

    McCarthy: It’s an interesting thing because you’re trying to fly an aircraft while changing the engine in some cases. And the same people _____ transform are the same people that need to drive the aircraft. And also if you call in too much fresh talent, then you would probably put the noise in the joint of the people that are supposed to be helping you run the process anyway. 

    So the way I’ve done it is I’ve gone in and we’ve identified some highly skilled people that know the processes really well and then supplement them with some more transactional process mappers and people like that. So that those people that run the processes can also see things like: how do they do the process mapping and how do they identify the paint points? How do they convert it into business cases? How do they use it and then convert it into technology?

    So on that journey, the people that are running their processes are also being upskilled at the same time into continuous improvement and transformation. But I’d say that there’s also – it’s not about upskilling necessarily on all sorts of things. Because as technology’s changing, there’s less the need for upskilling because a lot of technology is self-serve-type technology now. A lot of the RPA touring now. People can do it like they learn Excel, like they learn Word and things like that. so the upskilling like it used to be, about spending months and months programming and that type of thing – there’s less the need for that. It’s more about the upskilling in terms of continuous improvement. And it’s a cultural change, I think somebody mentioned earlier. So it’s going in and using the resources that’re there, upskilling those people, and slightly supplementing them and doing hands-on job training.

    Shimel: Excellent. Mark, Mitchell, I’m gonna give you each a chance to kinda close it out here, and then I think we’re about outta time. Mark, final thoughts?

    Settle: Let me just make a final comment along the lines of this last topic. I’ve gone through many of these reskilling things, and seeding that group with new talent I think is essential. I mean, if somebody came into me and said, “We’re gonna go to DevOps and you can have consultant money; the consultants will come in; they’ll be here 90 days, 120 days” – this is an acquired skill. It’s almost an art form at its finest. And the devil is in the details. And the day the consultants leave, everybody’s left to their own devices, and it’d be just like saying, “Let’s just shut down the data center and move everything in the data center up on AWS, and we’ll get some AWS consulting in here, and then they’ll leave after 90 days, and you guys run all the production systems up there in that whole new environment after 90 days of familiarization.” I don’t think too many companies would take that kind of a risk.

    So I think people get so carried away with thinking that reskilling can be effective, that people will come to embrace it. And, again, I hate to say this, but in my personal experience, if you reskill a population of people, 20% will be superstars, 20% won’t get it, and 60% will be good enough. But if one of the 60% left, you’d recruit somebody in with a higher skill level than the person that left. So they make forward progress. I’m not denigrating people’s commitment to new things. But you’ve gotta bring in that fresh talent that’s been there and done that. They know what it’s supposed to look like, right? And they can look around a corner and they can see the big fuzzy hole in the road that we’re all gonna fall into here in the next two months. 

    So that’s my thing.

    Shimel: Mitchell, you wanna bring it home?

    Ashley: Yeah. I’d like to just wrap with the human element, but maybe from a little different perspective. You know, people react to change in different ways. Your advocates – great, they’re all about it. Your resisters – love them ’cause you can actually work on that. You can help. You can help bring them along. It’s that passive unspoken middle ground, the folks that’re waiting to see: “Before I commit myself – I’m not gonna put myself, my job, my career at risk for this next thing. I don’t know if I can do it.” And the idea of bringing people in – I think one of the main purposes is to show the team, A, it can be done, and, B, they can learn it. So it isn’t: bring the consultants in for 30 days, hand over what they did, and now everybody goes, “Now what do I do?” Right? 

    So I think if you can de-risk, address the fear of what this means, that: “You’re gonna be able to do it and we’re gonna stick with you to help you get there, help everybody get there. And if there’s some folks that choose not to go along the journey, that’s okay too. But we’re all in this together.” I think you’ve gotta address the human element from a very personal: how does each person feel about this and whether they’re signed on to do it? And that’s gonna guide your success. That’s been my experience. 

    Shimel: Absolutely. Well, guys, we’re about outta time. Mark, Eveline, Louise, and of course Mitchell, thanks for joining us on this episode of DevOps Unbound. Many thanks to our friends at Tricentis for their sponsorship. Until next time, this is Alan Shimel for DevOps Unbound. Be well, be safe, and we’ll see you soon. Bye-bye.

  • How to Properly Approach Automation

    How to Properly Approach Automation

    Imagine if you backed out of your driveway, tapped the gas and BAM! You are at work. Nothing to worry about along the way, no traffic congestion, no other cars to worry about, etc.

    That is, unfortunately, how many of us approach automation. “Fire and forget” is real, until a problem crops up. I could talk about some complex environments, such as ARA, in a containerized environment with multiple languages, accessing mobility-limited databases and targeting several client platforms, for example.

    But we’ll keep it simple, and I’ll make an admission. Backup was one area where we at Ingrained Tech fell into the fire-and-forget trap. One part of our backup system runs desktops out to a server, the server (and other servers) out to a NAS and the NAS out to the cloud. I don’t think our security peeps will freak out if I mention we use duplicity for this backup and Backblaze for our backup cloud. Backblaze is a dream to use compared to others we’ve looked at, and the pricing is competitive, so it suits our needs well.

    A few weeks ago, we realized that while backups were completing successfully and verification was fine, it was taking nearly the entire week to do our weeklies. That’s bad news. We back up terabytes of data, but this is 2020, and our drive IO/Network can handle it. We started by looking at both of those things, and sure enough, it was drive IO causing our problems.

    Unsurprisingly, reading the disk and creating a collection of encrypted tarballs (a super-simplification of what duplicity does, but gets the point across) on those same disks was causing problems. After investigating, the one build that occurs on that server was suffering from backups stretching out, too. We had seen occasional issues with the build, but nothing jumped out as the source, and it was only one night every month or two. It was on our list. You know, the same prioritized list you have. And this build wasn’t nightly-critical. You get the idea.

    This was a relatively urgent problem, as soon as last week’s backup would not be complete when this week’s tried to start. Not good.

    We took our DevOps mentality to the problem and discovered a simple solution that solves the issues. The backup NAS is RAID 50, so it can handle a lot more abuse than the server that was slowing down backups. So, we took the simple expedient of rsyncing the source server to the NAS, then using duplicity to tar and encrypt before sending it on to the cloud buckets. Internally, we do not believe we need to encrypt backups because if someone is in our network, all of that data is laying around. By putting duplicity and both the tar function and the encryption function onto the backup NAS, we moved it to a far less busy system. Backup times dropped from days to hours instantly, and our unrelated build problems went away.

    We have to rework the automation somewhat to make sure validation is, well, actually validating the data is correct. But we’re well on our way to having a stable, faster system.

    How does this fit with DevOps? We see the same thing happening often in the automation rush of DevOps. Yesterday’s best solution is not necessarily tomorrow’s best solution. “Automate and watch” should be the catch-phrase. Improvement comes from both DevOps culture and paying attention to changes in the tools market.

    As Alan pointed out, someone will eventually win the end-to-end DevOps rush—since I feel the urge to inject my opinion on who stands the best chances, expect a blog on this topic soon. Until then, knowing what your options are and how to best achieve your goals–even on problems that were “solved” last year–is your best path for continued improvement. Yes, it means rework, but just like reworking your code, the point is to streamline things, get closer to your organization’s goals and keep improving after the low-hanging fruits of DevOps have been gathered. The rate of improvement in several areas of DevOps is stunning, so you may find you actually like a new tool better than the one you chose a couple years ago. And remember, knowledge is power.

    — Don Macvittie

  • COVID-19: Don’t Sacrifice DevOps

    COVID-19: Don’t Sacrifice DevOps

    Like everyone else, I’ve been following COVID-19/Coronavirus issues. My focus is more on the IT impact of the disease and its status as a pandemic.

    Let’s start by saying it isn’t the end of the world. Yes, it’s serious. Yes, it will be disruptive. Yes, those impacted by the worst aspects of the disease will leave a hole. But when it is over, and the tide has swept by, the vast majority of people will still be here, still trying to do what they need to do.

    That includes ITOps/DevOps maintaining whatever you put in during the rush to work remotely. Some organizations will be happy to throw something up and then rip it back down, but most will not. Once they’ve worked out how to work remotely, they’ll want to keep that as an option. Some to offer more work/life balance to employees, others to have a backup plan in case of emergency. Ask any company in Katrina’s path, remote work plans born in emergencies tend to hang around.

    It is the nature of DevOps to be nimble and reactive, planning on improving things as time goes on. That’s a good feature of both DevOps and modern IT teams. But some things are less nimble than others, and the rush to remote work is going to show that in a hard way.

    The VPN chosen or expanded to enable remote workers is not likely to be swapped out easily or quickly. While there is a sense of urgency—if you follow /r/sysadmin or to a lesser extent, /r/DevOps on Reddit, you know that urgency may even be too gentle a word. Some management teams seem to be in panic mode—you are going to have to live with some of these choices for a while, don’t make them knee-jerk.

    Normally, an MSP is your friend when moving like this, offering expertise and knowledge your organization doesn’t have internally, and shortening both deployment and the learning curve. But consensus is that MSPs, laptop manufacturers and VPN vendors are going to be booked solid for the near term.

    So, choose software when you can. Not because software is better (truly for some functions it is worse), but because it is generally easier to replace. A software VPN can be expensive, but you’re not buying hardware that is a sunk cost. Consider short-term solutions such as Amazon Workspaces or possibly  Chrome Remote Desktop, old standbys like VNC or RDP (likely with VPN in both cases), file sharing with things such as Sharepoint or OneDrive. Later on, you can look at a Cloud NAS such as Nasuni, or a more solid remote access plan.

    Yes, all the suggestions above have things you need to worry about. No, not everything will be within your control. However, these things are easier to rectify on iteration than “We bought big solution X, and it doesn’t do Y, so we need to replace it.” Start with short-term solutions and improve them as you go–this is truly the DevOps way, but it is easy to lose as you’re looking for high-volume, perfect and urgent.

    IT workloads (particularly operations workloads) are going to increase for any organization that wasn’t already set up for broad remote work. There is no way around this fact. As someone suggested online, “Kiss your family, tell them you’ll see them in a couple months and get to it.”

    You’ve got this. The internet is loaded with information to help, places such as Spiceworks and Reddit have people willing to offer advice all day. Smaller MSPs are likely happy to talk with you as they attempt to grow their business. Determine what you need, how fast you need it and implement the best solution you can in the time available, aiming to improve it on the next iteration. And keep the wheels of business running.

    It’s a flu variant, not the black plague. Businesses that deal with this well will come out well. Right now, “Deal with it well” includes a lot of IT work: from continuity planning, to fast DevOps iterations, to getting people at home productive and supporting them through this.

    — Don Macvittie

  • Scaling DevOps for the Enterprise

    Scaling DevOps for the Enterprise

    Information technology has been an integral piece of the modern enterprise for multiple decades and that dependency is only growing. With digital transformation, there’s the inevitable increase in demand for new applications. As such, software development services will continue to be a vital component of the business amidst the mounting pressure to innovate and deliver. This has resulted in many organizations moving from traditional development practices to more flexible and agile methodologies. Conventional software delivery methods are also being replaced in favor of enterprise DevOps practices such as continuous integration (CI) and continuous delivery (CD).

    While DevOps stimulates innovation and speeds up software delivery, adopting DevOps practices does not guarantee success as there are times when companies have been underwhelmed after implementing DevOps practices across their development teams. Knowing this, it is important to state that most DevOps failures come down to flaws in the execution of the methodology. A best-practice DevOps implementation should increase the value and impact of IT infrastructure by bridging new and old technologies. This allows for companies to better navigate change with clear time-to-value merits and lower risk. In short, it’s about scaling DevOps practices with security and quality embedded every step of the way.

    Look for Small Wins and Build on What Works

    Implementing DevOps best practices can mean a radical shift from the way an organization currently works. It is important that the entire team buys into this new approach. However, different dynamics within different organizations can lead to different results.

    Ideally, you should never have large complex projects. The best approach would be to work in small batches where you and your team can quickly learn from your decisions. By identifying the bottlenecks causing the largest constraints in your delivery process and incrementally changing, transforming and optimizing, your team could demonstrate the advantages of the new DevOps practices and build confidence in the team’s ability to implement the new methodology. Wins like these give a great start to the project before you’ve even introduced the entire IT organization to the changes needed to implement DevOps best practices. 

    Break Down Organizational Silos

    Known as well-intentioned areas of specialization, organizational silos are defined within the business to sharpen expertise and assign roles. Silos may have unintended consequences over time though, as each department potentially starts paying less and less attention to the other departments as they become consumed by their own assignments. As a result, silos inadvertently limit collaboration and have the potential to contribute to delays, bugs and conflicts. This is why C-level executives must initiate the process of removing the barriers to these silos and integrating the different disciplines within the IT department.

    For DevOps to work across the enterprise, executives must reorganize the configuration of software development projects by bringing in representatives from all important IT disciplines.

    Everyone Needs to Be on the Same Page

    Sometimes silos can be deeply rooted in an organization, and even if you do succeed in removing the barriers, there’s always the risk that they could re-emerge. Due to this, it’s important to create motivation for a silo-less enterprise.

    Anyone wanting to change a culture needs to define the actions and behaviors they desire, and design the work processes that are required to reinforce those behaviors. If your ops team is measured only by mitigating risk and your dev team is measured by delivering change, it goes without saying that you have a conflict.

    Through the incorporation of intra- and inter-team collaboration metrics into individual and team performance reviews, you can base a considerable amount of the review on lead times and quality metrics for software delivery. This is important not just for the dev and ops teams, but also QA and security. It’s easier to act your way to a new way of thinking rather than think your way to a new way of acting. 

    Real-Time Visibility Across the Toolchain and Environments

    When a team has poor visibility of their system, it can result in bad decisions and missed opportunities. Teams must replace manual project and release tracking methods that use spreadsheets and emails with an open, web-based solution that your team can leverage to plan, track and govern the entire application release management process.

    As teams and their stakeholders need real-time visibility across the entire DevOps toolchain and production environments, it is important that the chosen solution enables product team members to focus on their critical tasks while providing pertinent real-time information to their managers and stakeholders. 

    Don’t Lose Sight of the End User

    Once deployed, the application or service will interact with existing, and quite often, new systems. As it is developed for use by humans, don’t lose sight of the end user. Their satisfaction determines the success of your application or service.

    Actively involve users every step of the way, not just at the ideation and product requirements level. Teams should take the appropriate time to properly assess and fully understand the issues, challenges, priorities and constraints of their users. This knowledge can be utilized to define key performance indicators (KPIs) and continuous feedback throughout the DevOps toolchain.

    Move Forward on Orchestrating Continuous Delivery

    There’s no question that a lot of work goes into preparing and deploying a release. With the goal of delivering releases that have a high impact, it seems that companies sometimes choose to hold and then bundle together multiple application improvements into a single release. Such a strategy can lead to poor quality and failure—a result of deploying large infrequent releases. As another key driver toward an increased need for orchestration and automation, a lack of skilled employees means that many organizations have trouble releasing software in a reliable manner.

    The orchestration of continuous delivery is a fairly new way of planning and delivering business applications across the enterprise. With a focus on continuous improvement, continuous integration and ensuring you are always ready to deliver what the business needs, release control and deployment automation are vital to ensuring smooth delivery and a basis for governance.

    Conclusion

    The implementation of DevOps is a long term commitment. More a philosophy directing how you work, DevOps is not a rigid set of practices like ITIL, lean or Agile development.

    No organization will reach a state where it can comfortably say it has perfected its DevOps best practices model. Instead, it must continuously experiment with new processes, capabilities and tools identifying those that can yield the highest possible integration across the entire DevOps toolchain to deliver the best value for the business.

    — Mark Levy

  • How Enterprise Architects Can Help IT Operations

    How Enterprise Architects Can Help IT Operations

    Traditionally, the role of IT operations has been to keep apps and systems running well for the business. But in the era of cloud, DevOps and disruptive technology, keeping the lights on isn’t enough. IT operations people need a broader view of business needs. They need to evolve the infrastructure stack in ways that will maintain stability yet also support business goals. This is no easy shift for people who’ve grown their careers by focusing squarely on the technology: procuring and configuring servers, storage devices and network components.

    Enter the enterprise architect. Enterprise architects live and breathe business requirements and are a liaison between business leaders and IT. They’re responsible for building a long-term plan for technology evolution aligned with digital business initiatives. Enterprise architects are stewards of enterprise data and thus have typically established many solid relationships across not only IT but business functions as well. Typically, they are advisers to C-level executives and traditionally report to either the CIO or the CTO. 

    The chief architect can also be a valuable partner to the director of IT operations and his staff. Here’s how:

    1. EAs bridge the gap with business units, helping define business services and their requirements which ITOps need to meet, and gathering end-user feedback which drives planning and staffing.
    2. EAs bring a deep understanding of enterprise data assets, data connectivity and business applications—perspectives that help ITOps define key business service management goals.
    3. Architects also assess the need for new infrastructure technologies and take a strong hand in evaluating vendor platforms. 

    The relationship between IT operations and enterprise architects may not be an easy one, since IT operations have been accustomed to working independently. They may not welcome other people/teams advising them on which monitoring systems to use, for instance, or how to prioritize incidents. Yet, better collaboration will ultimately help IT operations teams by improving their sometimes-contentious role with the business and providing a deeper understanding of the relationship between network and business outcomes. The following use cases illustrate how this can work.

    Driving the Roadmap Between Current and Future State

    One of the activities that enterprise architects conduct is creating a technology reference model: a list of all the technologies used in the business and notations as to which ones are approaching EOL and should be replaced. The EA’s job is to reduce organizational risk from outdated and misaligned tools and systems. On the other hand, IT operations managers have the viewpoint of managing risk; when a key infrastructure technology is suddenly jettisoned, that jeopardizes the health of business applications. EAs and ITOps teams have an opportunity to get on the same page, developing a transition plan together for moving to new technologies in a way that meets business needs and minimizes IT performance risk. 

    Improving the Stack for Business Value

    As part of creating a long-term technology strategy, EAs focus on ROI. When ITOps share performance data on applications and systems, the EAs have critical data points to determine whether an application is too problematic–causing too many issues for too little business value–to maintain. Examples include moving from custom-built CRM to a COTS CRM or moving from on-premises email to Office 365 to increase availability and reliability of email.

    Justifying and Evaluating New Systems

    IT operations regularly want to purchase point monitoring software. They like what they see in a couple of new startups focusing on cloud monitoring and optimization or something specific for monitoring databases. But they may not know that another group, say in the EU region, is using something that could be applied to solve the issue. EAs can supply an holistic picture of the problem and how to solve it. In some cases, it’s not about buying a new system but leveraging an existing one to reduce complexity and minimize spend. EAs can also bring a business/user lens to assess user experience, such as AIOps. What effects could an AI application have on improving service delivery, and showing customers that IT is there to make work easier?

    The EA and ITOps teams need to work together to support the business. Enterprise architects should bring to the table their understanding of business needs, existing technologies and their knowledge of technology investments that are being proposed outside of IT. Meanwhile, ITOps can bring granular information about device failures, common issues and preferred vendors, which helps guide the EA’s recommendations to business stakeholders.  Together, these two critical roles can develop enterprise IT strategies which help the business succeed.

    — Michael Del Castillo

  • Moving from NOC to the SRE Model

    Moving from NOC to the SRE Model

    Is Site Reliability Engineer (SRE) another case of “Do what Google does”? When viewed with a discerning lens, the practices and tooling high-performing engineering teams deliver to the market can be very beneficial. The SRE model is one of them and stands to benefit DevOps teams substantially.

    If I could change the name from Site Reliability to Service Reliability I would. Applications are increasingly more service oriented, architectures are based on services and the SRE model is generally more customer centric than previous application support approaches. SRE has become the production support companion for modern application development.

    While the Network Operations Center (NOC) and SRE primary objectives are to focus on application uptime, the responsibility of supporting applications is much greater. Those who support modern applications are under pressure from the business and development to respond fast and accurately. This has been done previously by a NOC only.

    In its most traditional form, the NOC was a 24x7x365 staffed room where all telemetry from various aspects of infrastructure and the application were presented on screens, usually with one screen dedicated to sports and video games. If you staffed the NOC, your job was to keep an eye on the application, spot any alarms and if there is an alarm, to find the right person—usually based on a spreadsheet—to fix it.

    The NOC has also classically been more tied to the infrastructure than the application. They would get alarms for the application, but they’d be unable to dig in and figure out what specifically in the application was causing the issue. The NOC model services waterfall development and monolithic architectures just fine. But now, with rapid release velocity and new application architectures, a new approach is needed. This is where SRE comes in.

    Customer Obsession

    The first big difference between the SRE model and the traditional NOC model is that the SRE role does not start when something breaks, it starts before. The most high-performing SRE teams I’ve seen always work backwards from the customer. They use the customer to decide what metrics to measure and collect, and how they will interpret the metrics.

    Google talks about RED (Request, Errors, Duration) metrics but these are in no way best practice for everyone, nor the only approach. They are the best practice for Google, perhaps, and how their users interact with the application. Things such as saturation, latency, etc., can be even more important. Also, it is not the individual metrics that matter, it is their relationship to each other and, ultimately, to the customer’s experience. The customer obsession bleeds into everything SREs do—from setting up automation to educating the rest of the technical teams.

    To be customer obsessed, SRE teams need to have good visibility into what is going on.

    Observability

    I’m sure you have heard the new popular term “observability.” You might have thought to yourself, “Isn’t that just monitoring?” Observability’s foundation is monitoring, but it’s purpose is to go from the static point-in-time view of applications that traditional monitoring has been used for. It also addresses new challenges that show up as application architectures become increasingly more fragmented with services, and more complex to monitor holistically.

    Observability in some ways is about removing the human observer. In the NOC, you have staff watching aspects of the application and infrastructure to decide when there is an issue. Platforms that are designed to be observable have the detectors to tell you when there is a problem. It also does not create an assumption that those on-call have existing underlying knowledge of how all layers of the application work in advance because with modern applications, this assumption is not possible.

    Observability is both a tooling approach, but also a strategy for building applications. Applications should be built to be observable, which means the application provides sufficient telemetry to support its observability. Monitoring is basically part of your application now.

    Additionally, application degradation (such as latency) is the new downtime, so the ability to measure depth and breadth of application activity is critical. Observability helps identify reduced performance of the application as well, or impending issues in services that could lead to catastrophic events.

    Automation

    There is one big assumption that comes with the SRE model: automation and the interest to automate. The SRE model enables greater context and visibility across the organization, but it has to do that without increasing human effort. Without automation that is not possible.

    Automation includes:

    • Going from insights from observability to incident response.
    • Gathering context from systems to report, to observability tools and incident response.
    • Mechanisms to support those who are on-call with self-service remediation.
    • Automation to understand similar incidents.
    • Automation to alert/page/call people based on on-call schedules and policies.
    • Automation to recommend responders based on their incident response history.

    Anyone Can Be On-Call

    One of the huge benefits of the SRE model is anyone can be on call. The practices and automation of the SRE model allow anyone to be added to a rotation and on-call schedule. Most companies have this as a dedicated team, and in those who are transitioning from the traditional NOC to modern, that is exactly the first change that takes place. They leave the dedicated physical room and are on-call virtually instead.

    Not just people who support the application are on-call, the SRE model allows for developers and quality engineering to be on-call as well. Arguably, you can’t embrace shift-left without embracing SRE.

    Context

    Automation without data does not create a truly observable and actionable environment. Leveraging data across your delivery chain that can support action is critical. For example, this means the ability to leverage tracing to pinpoint production issues down to a service, or embed more context as part of the payload of alerts to bring more clarity to those on-call.

    Data collection is one thing, but not knowing how you are going to leverage the data makes it useless. The SRE model is obsessed with leveraging data and doing controlled experiments to ensure the data collected supports the uptime and experience for the user.

    Stewardship and Strategy

    Finally, SREs spend more time on stewardship and strategy than the traditional NOC. Some SRE groups run DoJos for the organization to help spread the idea of quality and application support across all technical teams. They will collaborate with developers so they better understand aspects of the application the SRE processes can leverage to support it better. SREs also spend a lot of time creating the strategy of application production support for the organization. Choosing key metrics, deciding how automation will be setup and consumed and creating policies on how on-call should function.

    Before you get upset, SRE does not mean the NOC is dead. The modern NOC can be driven by an SRE model, and I’ve seen this in-place in several companies. The NOC is also necessary to support many on-prem applications.

    What happens is, instead of a staffed room, the NOC becomes a group of engineers who are building automation, setting policies and practices, and stewarding quality across all teams. Similar to DevOps, SRE has elements of culture, principles and practice. Organizations who have a team structure where the NOC is responsible for application support, oftentimes do not change this for SRE, they become the SRE.

    The NOC is changing, and has to change as modern applications become more complex, release velocity increases and users become more demanding. The SRE model is an approach to transform the NOC into a modern version of itself or a whole new organization and strategy. The goal is not only to support applications when something breaks, but to make sure customer’s expectations are met, and systems support current and future development efforts, without increasing the human workload.

    — Chris Riley

  • DevOps Deeper Dive: Mainframe Fallout from BMC’s Compuware Buy

    DevOps Deeper Dive: Mainframe Fallout from BMC’s Compuware Buy

    The acquisition of Compuware this week by rival BMC Software signals that the rate at which mainframe environments are being modernized might not be occurring fast enough to support a vibrant third-party tools provider ecosystem.

    Both BMC and Compuware were sold to private equity firms in 2018 and 2014, respectively. Thoma Bravo, which acquired Compuware, is now selling the company to KKR. BMC was acquired by KKR in 2018 from Bain Capital and Golden Gate Capital for $8.5 billion. BMC was originally taken private by investors in 2013.

    Other rivals in the mainframe category include CA Technologies, which was acquired for $18.9 billion by Broadcom last year, and Micro Focus, which includes software assets originally developed by Hewlett Packard Enterprise.

    BMC executives declined to comment on the merger and terms of the deal for Compuware were not disclosed.

    All these machinations by private equity investors doesn’t do much to inspire confidence among end customers, said Judith Hurwitz, president and CEO of Judith Hurwitz & Associates, a market research and IT consulting firm.

    “Private equity firms always have to eventually look for an exit,” she said. “In this case, one investor wanted out.”

    That leaves IT leaders in the unenviable position of not knowing whether the application development platform on which they have standardized might be around for the long haul, said Hurwitz.

    The challenge faced by third-party providers of application development tools in the mainframe space is largely two-fold. While the number of workloads running on mainframes continues to increase thanks mainly to availability of Linux on the platform, the number of organizations that have a mainframe has declined. The low end of the mainframe market has been eroded by high-end distributed systems. In addition, many applications that previously ran on a mainframe have been shifted into a public cloud.

    The second challenge these providers face is IBM, which is now a much more formidable competitor in the mainframe tooling sector. IBM has launched a series of initiatives to provide tools to modernize mainframe platforms using best DevOps practices as part of a larger digital business transformation initiative. However, regardless of the relative strengths of IBM’s DevOps portfolio, there will always be a significant base of mainframe shops that will prefer tools from IBM over those provided by a third-party rival.

    In a prepared statement shared via email, Barry Baker, vice president of IBM Z Software, said:

    Over the last 18 months, we’ve come a long way to expand the ecosystem for mainframe skills by bringing new tools to IBM Z and LinuxONE. Our goal remains unchanged – to make development on Z a common experience for all developers across the tech stack – not just mainframe developers. We’ve made significant progress toward this goal with the adoption of DevOps and industry-standard tools like Git and Jenkins on the platform, along with our own offerings including IBM Z Open Development and IBM Z Unit Testing. We took another step with Red Hat OpenShift on IBM Z, further enabling cloud developers to deploy z/OS applications using OpenShift without any special Z skills required.

    We continue to invest and over the next few months we’ll be unveiling even more tools across three focus areas: developer experience, automation and operations to truly deliver on our promise for a cloud native experience – and encourage our clients to watch this space.”

    Of course, all of IBM’s tooling rivals in the mainframe space have launched similar DevOps initiatives. The challenge they all face is the rate at which mainframe organizations are embracing DevOps may not be fast enough to sustain a vibrant third-party ecosystem. Not surprisingly, BMC has also been extending its reach to encompass distributed computing platforms. Competition among tooling vendors on those platforms is already fierce.

    There can be no doubt mainframes will continue to play a crucial role in IT. Most large enterprises are still dependent on mainframes to run some of their most mission-critical transaction processing applications. Less clear is to what degree those organizations intend to modernize those applications beyond simply making application programming interfaces (APIs) available for legacy applications. IBM, along with every provider of development tools for the mainframe, has been making a case of deploying new applications running on Linux on a mainframe because the total cost is lower than trying to connect those applications running on a distributed system to a legacy mainframe application.

    A recent global survey conducted by Vanson Bourne on behalf of Micro Focus suggests that argument is starting to resonate. Nearly two-thirds (63%) said some form of modernization was favored over the replacing and retiring of older COBOL-based systems in 2020.

    Regardless of how much economic sense that argument makes, however, the rate at which mainframe applications are being modernized will, at best, continue to be an extended process.

    — Mike Vizard

  • The Role of Soft Skills in Building Strong DevOps Leads

    The Role of Soft Skills in Building Strong DevOps Leads

    As an industry, we spend a lot of time talking about the process and technology that make up a good DevOps practice. We have a strong focus on the development, IT operations, QA” and security roles. We debate tools and automation strategies. We wax philosophical about DevOps culture.

    What is either lacking entirely or simply glossed over in that conversation is what I would consider the backbone of any good DevOps team–the people. 

    You might assume by “people” I mean their skills with the technology or their expertise with a certain process. Those two things are important, but all things being equal, the people aspect of DevOps I am referring to is really more simple–or even more basic–than that. It all starts with the culture that is defined by the lead.

    As with any cross-functional discipline, teamwork is essential to the adoption and success of your DevOps practice. At its core, DevOps is about collaboration among people from a variety of teams and departments. It brings together multiple disciplines, people and processes with the common goal of high quality and high efficiency output. DevOps relies so heavily on teamwork that without strong leadership, your DevOps culture and the success of your practice are destined for a rocky road at best.

    A strong DevOps leader will need to bring together people in roles that have either been traditionally siloed or are now being done by fewer people. This is no easy feat, however building trust, earning respect, communicating effectively and motivating everyone toward the common goal is paramount to the success of the DevOps team.

    These traits–trust and respect, communication and the ability to motivate–are not unique to DevOps leads, rather they are the key characteristics that make up any strong leader. By putting them in the context of DevOps, the importance of each in the process and culture is irrefutable. 

    Trust and Respect

    Paul Zak, the founding director of the Center for Neuroeconomics Studies and a professor of economics, psychology and management at Claremont University, led a study to better understand the neuroscience of trust. His team found that the effect of trust on work performance, engagement and loyalty were huge. Respondents indicated that employees with high trust had 106% more energy, were 76% more engaged and 88% said they would recommend their company as a place to work to family and friends. 

    The numbers might not come as a surprise, but how you achieve high trust might. First and foremost, be vulnerable. Do not be afraid to ask for help. This not only exhibits the trust you have in the people on the team, it also demonstrates the sign that you are secure in yourself as a leader. 

    The trust factor for a DevOps lead is multifaceted. The team must trust the lead enough to also trust his/her vision for the process. By showing vulnerability and leaning of the subject matter experts from each discipline for input, you start to build rapport with the team and the process becomes owned by the different functions involved. By asking for help, you invoke the natural tendency to cooperate. That cooperation leads to mutual trust. And by earning trust, you also earn their respect. 

    Communication

    As with any cross-functional team, communication should be viewed as critical an element as your DevOps tools. A successful DevOps lead will be able to understand and speak the languages unique to each discipline. By understanding the nuance and meanings of the words used by the IT operations teams, QA, security and development, the lead can easily help communicate goals, set expectations and establish rules of engagement. Through daily communication, the team will be able to reduce friction and become far more effective as a group. 

    Motivating Toward a Common Goal

    Former Boston Bruins coach Harry Sinden said it best, “Putting on the same jersey doesn’t make a team. You’re still a collection of individuals until you find a common goal.”

    The DevOps lead who motivates the team toward that common goal will find success. But how do you do that? To find out what motivates the people on your team you need to talk to them and get to know them. Involve them in developing the overall strategy. Seek their input and make sure they feel they have ownership of the outcome. 

    Set milestones and celebrate accomplishments. Reward success. Provide feedback and coaching in the moment–both good and bad. 

    The team that feels like they are part of something are no longer a collection of individuals.

    In the end, when you are assessing your DevOps practice and culture, make sure you take a long hard look at the people. They are truly the backbone of any DevOps practice. The need to technical prowess and experience should not be underscored, however, make sure you look for someone who has those skills plus all of the necessary attributes of a true leader. The success of your DevOps practice may depend on it.

    — Sara Faatz