devopsy.kinsta.cloud

Tag: continuous improvement

  • Not Everything That is Necessary Adds Value

    Not Everything That is Necessary Adds Value

    Since the Lean production method for manufacturing arose in the 1930s, organizations everywhere have been seeking ways to apply it to reduce waste and increase productivity. This model has served thousands of companies well, from manufacturing and product design to software development. Yet, one difficulty that comes up when discussing or applying Lean in software development is the debate about what is and isn’t considered value-added work.

    Some people get defensive when certain activities are labeled non-value-add (NVA). Others argue that if a customer expects an activity and is willing to pay for it, it must be value-add (VA). From my perspective, software engineers, business leaders and others must accept the reality that it’s entirely possible for necessary processes to add no tangible value. We must understand the intent of Lean within these classifications. In doing so, we can get past the defensiveness and disagreements and move on to improvement.

    Let’s explore a few examples of NVA activities that are unavoidable—and therefore necessary—to a successful software development value stream:

    • Resolving important defects
    • Rework to meet client specifications (i.e. after clarifying requirements)
    • Project time tracking and billing (i.e. for time and materials projects)
    • Complying with regulatory mandates
    • Testing

    Some customers expect these activities to be performed. Software professionals generally agree that all of these NVA activities can be necessary. Yet many Lean proponents will continue to insist that they are NVA and should be minimized. So, where does that leave us?

    Distilling ‘Value’ to Its Essence

    Regardless of how important or necessary NVA activities might be, from a Lean perspective, unless an activity is adding value to the finished product, it is technically an NVA activity. Testing is an NVA activity. Defect correction is an NVA activity. Complying with regulations, although important to protecting profit margins for the company—and an activity that might incent customers to select your product over your competitor’s due to their reduced risk—is nevertheless a NVA activity.

    This is where we enter the grey area of “necessary but NVA activities that actually add a lot of value.” One prominent example is resolving defects that cannot be postponed. You’re probably shaking your head and thinking, “No way. Resolving defects increases quality which makes the finished software product more valuable to the end user. Therefore, it must be a value-add activity.”

    The answer is actually, “No, it is not.” Why? While the cost of defect resolution may be incorporated into the final price tag of software, there is no line-item charge for it. Most purchasers will not pay extra for defect resolution. They’ll feel that quality software is essential to their project and defect resolution should be included.

    Furthermore, they believe quality should be built-in— designed into the product and the process used to build it. That makes it an NVA activity. It’s necessary to the delivery of quality software but it doesn’t add value in the Lean sense of the term.

    The truth, whether or not business leaders want to acknowledge it, is that NVA activities are built into almost every product or service. When someone goes to pick up a prescription at the pharmacy they are never asked to pay an extra research fee. When someone test drives a car and decides to purchase it, they aren’t charged for the gas they burned during the test drive. Both of those activities were necessary for the final outcome to be satisfactory to everyone, but they added no value to the effort.

    The same is true for successful software delivery. Lean approaches form the foundation for modern software development, but being Lean doesn’t mean that you have eliminated all NVA activities. In fact, quite the contrary.

    Lean suggests that perfection is never achieved. A Lean organization values continuous improvement and is always looking for ways to reduce NVA activities. For all software projects, some elements will always be NVA. The perpetual existence of both value-add and non-value-add are facts of life.

    Optimizing Necessary Activities

    Despite the fact that these activities might be necessary and, therefore, part of the project cost, they don’t have to be excessive. Responsible software professionals and the companies for whom they work will make an extra effort to reduce waste and optimize outcomes in all NVA activities.

    Reducing or eliminating waste is one of the most important tenets of Lean software engineering. Waste can be found everywhere, from too many people working on a project and ineffective communication to an excessive number of manual processes, backlog mismanagement and rework.

    Breaking the Stigma of “Non-Value Add”

    Companies must develop quality software and sell it at a price that pays for the effort. In my experience, the way to do that is by evaluating every activity. Consider whether it is VA or NVA and whether it is essential or non-essential. Then reduce or eliminate the NVA as much as possible, even if it’s essential, and optimize the VA.

    Software professionals should be ready to defend the importance of the many necessary but NVA activities that go into a quality outcome, while remaining unapologetic about calling them what they are—a process, procedure or other activity that adds no value.

  • Empowering Teams to Make Mission: Impossible Possible

    Empowering Teams to Make Mission: Impossible Possible

    “Your mission, should you choose to accept it….”

    That’s the way the original Mission: Impossible TV show and, later, the movie series drew us in. The bad guy’s operation was explained. The goal was clearly defined; infiltrate and destroy. And although the task at hand was extremely difficult and fraught with unknowns, the team of good guys was briefed and everyone understood what they were doing and why. Cut to the good guys succeeding despite the odds being stacked against them.

    For most of us in product development, I think we’ve all faced—or are facing—our own impossible mission. We’re all working on “urgent” projects against some pretty tough odds. Except, unlike the old TV show, we often lack a clear goal. Sure, we’ve got our best team members on it—but does everyone understand why we’re doing what we’re doing? What, exactly, is our mission?

    No Dossier, No Chance

    In the frantic rush to “digitally transform” by churning out new products and features, the mission (if there ever was one) is often lost. Without context, developers can feel like order takers instead of outcome producers. This dynamic is a ticking time bomb; not just in terms of value delivery and business success, but for developer happiness, too. With a worldwide developer shortage and, more importantly, the fact that 80% of developers are reporting burnout, something needs to change. And fast.

    Improving the software developer experience must be a priority. And a clear mission is a great place to start. When we’re not mission-driven, our chances of being innovative, experimenting with new ways of work and empowering ourselves to be at our best are severely limited. 

    Liberated Mercenaries 

    By adopting a mission-oriented approach, development teams gain purpose. Bringing them in on the business problem that needs solving can motivate them and stimulate their creative thinking. And the cherry on top? We can even help reduce their workloads by removing meaningless blackhole tasks. We begin cultivating a work environment that helps realize the “joy;” part of one of Gene Kim’s Five Ideals. 

    For those overseeing DevOps or other continuous improvement initiatives, we can improve our odds by creating our own Mission: Impossible-like teams. To cite John Doer, venture capitalist, we can build teams of missionaries instead of mercenaries. Folks who are motivated, passionate and outcome-driven. Those galvanized by the excitement of knowing that their efforts will impact the business in a big way will inspire greatness. 

    It is the leader’s role to articulate the mission. When practitioners are empowered to use their creativity for problem-solving, innovation is unleashed and there is little room for misguided, wasteful activities.

    Preparing your Mission Dossier

    You are not alone if your answers to the following are no better now than they were five years ago:

    • Are we prioritizing our customers’ problems first?
    • Are we enabling customers to pull value rather than pushing what we believe is value?
    • Are we providing abundant opportunities for customer feedback and pivoting with learnings from failures?
    • Are we routinely surprising and delighting our customers with exceptional value?

    These are the kinds of questions that indicate how well we understand our mission. If we are aligned with customers, then, by definition, we are aligned with the business since businesses only exist to serve customers.

    So then, how do we articulate the mission so our teams clearly understand their purpose? Here are three steps to get you started.

    Step One—OKRs

    We must emphasize our focus on solving problems for our customers, both internal and external. The objectives must be clearly stated, as they were for Mr. Phelps. And, likewise, we must communicate the sense of urgency that this mission encompasses for the fleeting window of opportunity that exists before customers seek alternative solutions. 

    One highly effective approach is to employ objectives and key results (OKRs) and an OKR tree so that everyone, from the trenches to executive leadership can have line-of-sight to the purpose of their work and their role in the mission. Since DevOps Enterprise Summit 2020 in Europe, OKRs have emerged as a hot topic as leaders seek to translate urgent corporate objectives into meaningful goals for their business and technology teams.

    Step Two—Focus on Flow of Customer Value

    We must set the stage for the successful delivery of customer value. Our processes must be aligned with real customer needs while simultaneously capable of delivery excellence. The best way to do so has been proven out for over 40 years now, i.e.: adopting a continuous improvement approach to the activities which make up the value stream from customer request to fulfillment. 

    That means capturing customer needs via our OKRs and accelerating the flow of work across the value stream against those OKRs. To do that, we must have a means to systematically and continuously remove impediments to flow across all key stages, before and after development

    This approach encourages everyone—top-down, bottom-up—to become single-minded and obsessive about our mission.

    Step Three—Tracking The Mission

    With the mission clear and the teams aligned, we must also be prepared to react and preempt uncertainty. No mission runs perfectly. But we can put literal measures in place to identify danger ahead. Combining team performance metrics, such as DORA, with higher-level value stream management (VSM) metrics that focus on the flow of business value across the whole system can help everyone in the field and in the control room communicate effectively. For instance, if work upstream is holding our developers back from cracking the code, VSM metrics can help direct leadership to send support into the business analysis room.

    With mission alignment and means of communication, accepting a mission becomes enticing for everyone involved and the chances of success improve dramatically. 

  • EP 10: Observability & Continuous Improvement

    EP 10: Observability & Continuous Improvement

    Observability at enterprise scale brings with it additional application life cycle management requirements. Success requires knowing which changes result in improvements and which don’t produce positive results and how that information must make it back to the DevOps team through feedback loops. In this episode of DevOps Unbound, Alan Shimel and Mitch Ashley talk with Kurt Chase, senior director, global release management at Tricentis, Charity Majors, co-founder at Honeycomb and Ravyn Manuel, senior application manager at the Smithsonian as they discuss observability and continuous improvement. The video is below, followed by a transcript of the conversation.

    Alan Shimel: Hey, everyone. I’m Alan Shimel and welcome to another DevOps Unbound. We have a great panel in store for you today here on DevOps Unbound. Our topic is observability at a broad level. We’re going to jump into some of this. Let me first of all introduce you, though.

    First I want to introduce you, of course, to my cohost of DevOps Unbound. He’s the CEO of Accelerated Strategies Group, CTO here at MediaOps, my friend Mitchell Ashley. Hey, Mitch. Welcome.

    Mitch Ashley: Alan, thanks. Great to be here.

    Shimel: All righty. Let me introduce you to our panel for today. We have a great panel. I was so happy that this came together. First, coming at us from the mountains of Virginia by way of the Bronx, Ravyn Manuel. Ravyn, why don’t you introduce yourself and where you work?

    Ravyn Manuel: Absolutely. My name is Ravyn Manuel and I work for the National Museum of African American History and Culture, which is part of the Smithsonian. I am the Senior Applications Developer and DevOps Engineer. And thanks for not putting me in West Virginia. I appreciate that.

    [Laughter]

    Shimel: Okay. I don’t blame you. Next up we have the one and only Charity Majors, CTO and Co-founder at Honeycomb. Charity, a little bit about yourself, maybe introduce to the audience?

    Charity Majors: I am an ops nerd. I have been on call since I was 17 racking hardware. I no longer rack hardware anymore. Hardware doesn’t exist, as far as I’m concerned. Instead, I’m trying to save the world through better observability.

    Shimel: Excellent. And then, last but not least, we have Kurt Chase of Tricentis, Tricentis Releasing. Kurt, why don’t you introduce yourself?

    Kurt Chase: Hi, everyone. My name is Kurt Chase. I’m currently Head of Global Release Management at Tricentis, based in Northern California. Nice to join you all today.

    Shimel: Nice to have you here, Kurt. All right. So, team, panel, our topic this week is observability. And given that is the topic, I feel compelled to start with Charity because, Charity, in many ways you and the Honeycomb team are responsible for all this buzz around observability. Tell us about it.

    Majors: Yeah, a little bit. When we started the company a little over five years ago it was not a term that was really in use – used in tech. Now – which is not to say that we invented it. Far from it. But it has this long and impressive pedigree in mechanical engineering, specifically control theory. And in control theory observability is the mathematical tool of controllability.

    So, we started building this startup, Honeycomb, and we knew that what we were doing wasn’t monitoring. We knew that we weren’t reactively monitoring. We knew that we were – it was about instrumenting the code, getting at it from the inside. But for the first five, six months we couldn’t figure out what to call it. It was really hard to figure out how to talk about what we were doing. And then it was mid-July, midway through the first year when I remember Googling “What is the definition of observability?” And when I read it it’s all about “Can you understand what’s happening on the inside of the system just by looking at it from the outside with no prior knowledge?”

    And a lightbulb just started going off. It was like “Ding, ding, ding. That’s what we’re trying to build. That’s exactly –” if you translate that into software terms, what it means is can you explain any system state that – any state that your system can get into without shipping new code to handle that state because shipping that code to handle that state implies that you had prior knowledge. So, it’s all about the unknown unknowns instead of the known unknowns. So, yeah, that’s my story.

    Shimel: And you’re sticking with it. So –

    Majors: Yeah. I mean, it really encapsulates a lot of the problems that people are increasingly having, that there’s a lot that precedes from that definition of “unknown unknowns,” like you need to have high cardinality, high dimensionality, blah, blah, blah, all this stuff. But those are the things that speak to the problems of microservices and multiple storage systems, et cetera.

    Shimel: I get it. So, Ravyn, this is your life. How does this observability and what Charity has describe, the fact pattern or the problem pattern that she came to call observability, how does that in your life, day to day of what you’re doing, how does that kind of manifest itself?

    Manuel: Thanks, Charity, for actually providing a definition because I was like “Okay, are we supposed to define our own meaning of observability?” So, you’ve given a baseline. And I like that you said it’s not monitoring because I can actually feel the nuance. There’s a nuance between monitoring and observability. And it’s – what’s really interesting is that that’s a problem that I’m trying to solve today at work because we’ve got – and – because we’ve got the black boxes. So, you’ve got observability but when you have software there’s a lot of black boxes. You’re not supposed to be asking what’s going on inside the code because you’re not supposed to know what’s going on inside the code. But when you get a – you put a signal in – I was in the military, so I did electronic repair – so, you put the signal in and you don’t get the right signal out but you’ve got a black box. How do you know what went wrong there if you can’t actually observe what’s going on inside of there?

    It’s one thing to know the state of the software when you are the developer. So, as a developer, yes, I know. But when it comes to being the product owner or the project manager or a stakeholder and you’re trying to figure out what – “Why is this doing this?” or “This is not what I wanted,” actually is how that would come from that end, is “That’s not what I wanted.” So, how to get the stakeholders to get what they want when you can’t actually tell – I can’t tell what the code from my coworker is doing? So, in a way observability is a little bit – it is abstracted from monitoring because monitoring is telling you what it’s doing but it isn’t telling you the state. And so, it’s a problem that I’m trying to solve actually today.

    Chase: I was going to jump in here and say from my experience although monitoring, logging, reporting, alerting, they’re not observability, I think they are fundamental foundational pieces that you have to have in place. You can start observability without any of them, but I’ve found that the more you can get your monitoring and reporting and even get some alerting going, it allows you to then start observing the system. And for me, I know at Splunk and now here again at Tricentis I’m really trying to connect all the pieces because it’s not just about the applications, but I find that some of the DevOps engineers, they really help with observability, especially the ones that are super inquisitive, and they don’t just – when a result appears or something happens they don’t just take it at face value. They’re inquisitive. They dig in. They try and find the root cause.

    And I think that’s key with observability, is taking all the monitoring, reporting, alerting you have, taking the feedback from your customers, all of that comes together, I think, to really help you start to define those unknown unknowns that Charity brought up. I think it becomes very complicated. And then I think I’d love to talk about how AI and machine learning now can start to learn some of these things, because I think we get really good at repeatable incidences, observing those and then accounting for those in our systems, but it’s the unknown unknowns, how do we deal with those? That’s the real interesting part to me nowadays for sure.

    Majors: It is, I think, the interesting part. And I also think that AI ops is mostly bullshit and it will be for the foreseeable future.

    Chase: For a while. I agree, yeah.

    Majors: But I wanted to circle back to what you said about logging and everything, which is – so, there’s a technical definition for observability which has to do with high cardinality, all this stuff, and then there is just understanding your systems. And I absolutely agree: You need some alerting, you need some monitoring. I think there’s a question is – of – it log files? Is it metrics? That’s kind of irrelevant for just data. What’s important is can you understand what’s happening in your system when you need to understand it?

    And I do think it’s important to sort of differentiate the technical definition of observability from just the tendency that we have to, say, observing things because there’s a very rich and mature and well-established set of best practices for monitoring, which are in many cases completely mirror opposite from the best practices for observability.

    For example, with monitoring it’s like for years we’ve been saying, “You should not have to stare at graphs all day. The system should inform you when you need to – when there’s something wrong.” Super canonical best practice. With observability it’s exactly the opposite. It’s – we’re saying that most problems that happen are so subtle that you need to be looking for them. You don’t – you can’t page everyone at the threshold – you would drive them nuts if you were like “Oh, every little thing is going wrong. Bing, bing, bing.” You’re just – you would drive people nuts and they would quit. But when you’re an engineer, when you’re working on this endpoint or when you’re trying to solve this problem you should absolutely be looking through, asking questions, iterating on them, and looking for the trails in your observability tool at that very fine grain level.

    Chase: I agree.

    Shimel: So, I hear what all three of you said and it’s funny how everyone brings their own baggage to this issue.

    Majors: Oh, sorry.

    Shimel: No, no, it’s not – don’t be sorry. I love it. Here’s, I think, fundamentally – for those people out here who are watching this whole – I’ve got a lot of people out here who are DevOps people, who are ops folks, SREs, cyber folks, and their heads are spinning because I think there’s a little bit of what Kurt said to them, which is they don’t do – they do a shitty job monitoring, quite frankly. We don’t do a great job of it. People started throwing this AI ops word around, and there are a lot of companies raising a –

    Majors: Yeah, there are.

    Shimel: – crap ton of money around their AI ops stuff. And I never really understood what was real about it and many times it just sounded like another word for monitoring to me. But whatever. There is no AI in it.

    But – and now, look, over the last year, year and a half, two years we’ve seen observability come up here and they don’t get the difference. Ravyn, I think you do because quite frankly your black box analogy with electronic in the military, that is – you’ve got to figure out what the unknowns are from the unknowns. It’s not just getting all these signals that are known and then recognizing patterns and putting some automation behind it. That’s not observability. I’m sorry. Not in my mind, anyway.

    So, I think number one is we need to clearly define it. And Charity, you guys, you are a major kind of evangelist for this: “What is observability versus AI ops versus ALM,” which is before AI ops, and so forth. And there are SREs who are living this and DevOps teams who are living this every single day out here who haven’t got their heads around it. Ravyn, I always look to you as someone who gets it. Where do you fit on the – where do you kind of see it on the – let’s say there’s a scale of ALM, AI ops, blah, blah, blah, observability. What do you think?

    Manuel: I feel like I would be very – most close to observability because, again, all – anything with ops in it, when you start doing DevSecOps and DevOps, all of those ops just become marketing terms after a while. But observability is – because it’s not quite tangible it’s actually more real, because it’s real – because it speaks to exactly what’s coming, what’s going on when you’re trying to get continuous improvement, because you need metrics to show and prove that you’re doing continuous improvement. But how do you get there? What is it – what are the steps to get you there? How are you alerted that you are not improving or that you are going backwards?

    And so, when – I feel for me as far as observability I’m at the cusp, because I didn’t actually know that the word existed until I found out about the talk. I didn’t know that that had been defined. So, it’s really cool to know that I can now call it observability. It’s because it’s that thing that I can’t – there isn’t a tool to actually help me – actually help me know what it is that I’m looking at all the time. Not all the time. There are some tools that help me some of the time but not all the time. And it’s the times that I don’t know that’s when we are stuck, because it could be impacting the way – the system performance or the team’s performance.

    Shimel: Charity, I think you might have a customer there. It sounds like they need a tool.

    Majors: I feel like one of the key differentiators here between observability and AI ops and stuff is what it tries to center. With observability we are not – AI ops are like “Eh, we’ll tell you what to look at. You don’t need to understand your system. We’ll understand your system for you. We’ll tell you what to look at. We’ll tell you what it means. We’ll tell you what’s important.” With observability it’s the opposite. I think that we need to look to solutions that let computers do what computers are good at and let humans do what humans are good at. Computers are good at crunching lots of numbers but they can attach meaning to something. Only human beings can attach meaning to something. If I see a big old spike of red that could be good, that could be bad. You don’t know. There is no meaning until a human has attached them to it.

    And with AI ops, AI ops only really works if the corpus of data that you’re training it on doesn’t change. So, if you’re actually shipping software every day you’re fucking up your corpus. This – it’s kind of incompatible with modern software development. I have many thoughts and feelings on AI ops. God bless them, but mostly I think that what they’re trying to do is take the humans out of the driver’s seat. And I understand why, because the C-level, CTOs, CIOs, whatever – there’s something that kind of blew my mind when I realized it a couple years ago: They trust vendors over their own people. Their employees come and go; vendors are forever. So, whenever a vendor comes along it’s like you give them tens of millions of dollars; your people don’t have to hold all of the wisdom in their heads. You can replace them. They’re fungible. The tool will tell you; the tool will be the source of truth.

    And I just fundamentally reject – I’ve never seen that work well. I think it’s a shitty way to treat your people. And I would prefer to take the –

    Chase: And I would not agree with that statement, quite honestly. I’ve had leads that are –

    [Crosstalk]

    Shimel: All right. “Mr. Vice President, I’m talking.”

    [Laughter]

    Sorry.

    [Crosstalk]

    Majors: – and then using our tools to make them better, to impress them, to give them a better sense of where they sit in time and space in the history and what’s happened before and what they’re – they should be wearing grooves in the system as they wear – as they use it in the [inaudible].

    So, I think that it should – if you’re centering a person it’s observability and if you’re centering the – it’s AI ops. Sorry. Go ahead, Kurt.

    Shimel: It’s okay. Kurt, your turn.

    Chase: No, I was just – in my experience the leads, I’ve had – I’m a senior level director now; it was the same at Splunk – I did not encounter that where my leads trusted vendors more than myself or the people on my team. I was surprised to hear that, quite honestly, Charity. I have not experienced that.

    [Crosstalk]

    Majors: It’s not universal. But it’s – there – it exists quite a lot.

    Chase: Yeah, that’s too bad to hear that. That is seriously wrong and that’s going to go against everything we need to accomplish internally.

    Majors: Yeah.

    Chase: Yeah, if –

    Ashley: Have fun attracting great people with that mental model.

    Chase: Yeah, yeah, yeah. And the other thing – I’d love to hear your opinion on this, Charity – we have very complicated systems we’re running. The source code management, the build systems, the agents. I’ve found that it always is very helpful to distill it down into discrete parts, try not to blow the ocean at once. Let’s start with the source code management system. How are we observing that? What are we doing here? And would you suggest that’s a great way for people to get going? I mean, when you try and look at everything – you have to look at everything for sure, but to get started it seems like find some of the most important pieces. Source code management is a great one. The crown jewels. What’s your suggestion there for getting started and –?

    Majors: Source code management is important but it’s never – it’s almost – it shouldn’t almost ever be in the critical production path. And when people are rolling something out I always advise them to find something that’s – don’t waste your time instrumenting _____ observability, lights, and backwater to the system. Find the thing that you’re trying to understand and fix now. Find the thing that’s broken. Find the thing that’s causing you pain. Because it should be an order of magnitude better at answering your question, observability should, even than monitoring. I don’t think you can look someone in the eye and ask them to switch up what they’re doing and to use something else unless you can tell them it’s going to be an order of magnitude better than what they’ve got now because the costs of transition are so high.

    But with observability you should be able to go from instead of just the aggregate where it’s like “Well, I see a spike but I can’t tell you what it means or how they’re different from –” with observability you should be able to go exactly those rows, in which ways are those events different from all the baseline events, and very quickly go “Oh, I have a much richer view of my system now.” And if – once you’ve got that you should be able to explain way more about what’s going – so, we see this all the time when people are running out – when they’re rolling out Honeycomb. They start rolling – they roll it out to the service and they’re like “Oh, God, there are problems there. Did you know that this is broken? Oh, there –” and we’re like “Yeah, yeah, yeah, but you’re just picking up the rug. It’s been there – it’s been that way for as long as you know.” And they’ll pick up another service and the problems – I’m like “Yeah, but these problems have existed – you just had no idea because we were dealing with [inaudible].”

    So, I feel like the way to pick up momentum, the way to incentivize people to get excited about this is to look for the biggest pain point in your system and –

    Chase: That’s awesome.

    Majors: – then patch that with observability.

    Chase: That’s great advice. I agree.

    Manuel: Charity, a question. Does – can observability actually take in account process? Because you’re talking about if there’s something wrong in the system itself, but sometimes the thing that’s wrong with the system is the process.

    [Crosstalk]

    Majors: Well, everybody is not about process. And you’re totally right. That’s often – that is not something that you should ever ignore. A lot of the biggest problems are process problems. But observability is not the tool for that.

    Shimel: And that’s good to know too because quite frankly – and Mitchell knows this too – we live in a silver bullet world where people think the latest shiny trinket is the cure-all.

    [Crosstalk]

    Majors: – everything, you should –

    [Crosstalk]

    Shimel: No, observability, yeah, it’s not going to cure Covid either.

    Majors: No.

    Shimel: But –

    Chase: And that was the point I was trying to make initially. I think I said it wrong. It’s not that monitoring and reporting, logging aren’t foundational; they’re complementary. You kind of need them all. If you’re running modern stacks – I said that incorrectly at the start – it’s not foundational. There’s – you need them alongside each other.

    Shimel: They’re adjacent. They’re not –

    Chase: Yeah, absolutely. Absolutely.

    Shimel: They’re not – it’s not a pyramid per se.

    Chase: Right. Exactly.

    Shimel: I think it’s part of your stack horizontally.

    Chase: Yes. Yes. And I’ve found –

    Shimel: Can I – we haven’t’ – go ahead, Kurt, I’m sorry.

    Chase: I was going to say I’ve found with especially new individuals and new people to the DevOps team and the DevOps world, getting them involved in the monitoring and saying, “Hey, this is how we start monitoring,” that’s a great place to start. It’s almost entry level for observability, if you – I’d love your opinion on that, Charity, too.

    Majors: Well, I really think that everyone who writes and ships software should be on call for their code, no matter how –

    [Laughter]

    Chase: Yeah, me too.

    [Crosstalk]

    Majors: Software ownership. And so, yes, that’s very empowering, I think, to show them how to find the trail, the alert that’s paging them, trace it all the way to the system and fix it. That’s something that as engineers, we feed on that. That’s exciting to get to know how to run your ship better. Yes, absolutely.

    Shimel: Well, but – so, you know what? Mitchell and I started a company – or, were two cofounders of a company called StillSecure 20 years ago. And that was one of the things. Mitchell ran the engineering team and that was one of the things he instituted, I remember, was that every developer had to take a turn on the help support desk, level one, answer the phone. “Hello, Mr. Customer.”

    Majors: It builds empathy.

    Shimel: And try to explain it because they had to answer the alert. And even when they weren’t on that level one our escalation procedure was if it went to level two or three it was the developer who did the level two and three support.

    Majors: I will say that, yes, and I do think that every engineer should be on call for their work, but you have to couple that with it’s management’s job to make sure that that doesn’t suck, that it isn’t _____ for them, but it doesn’t just kill you with [inaudible]. It is management’s absolute fucking responsibility to make sure that they’re given the time to – away from project work to fix these problems so that they don’t recur, so that they get enough sleep. I think it’s reasonable to ask any engineer to be woken up two or three times a year for their code. That’s it. Because more than that, you’re burning people out hardcore.

    Manuel: Been there, done that.

    Shimel: You know, Charity, Ravyn –

    [Crosstalk]

    M : I have a question. I have a question I’d love to ask all of you. I’m thinking about observability has been elevated because of, in part, the world that we’re living in. Software architecture is changing. We’re dealing with microservices, all these much smaller things that are interconnected to make software. And at the same time as we’re changing the pace at which we release software, not only is it unknown unknowns, it’s like a river – Riven –

    Shimel: Ravyn.

    Ashley: – we’re in the river of the changing software. So, you’re trying to understand all of these things at the same time. It seems like it’s the right time for observability to be the thing to be working on because ops has to be elevated to the same level of the same kind of architecture and processes that we’re using across the – in the life cycle of software. Do you agree with that or not? It seems to me we’re in the right place, right time.

    Majors: Absolutely. I mean, I feel like the first wave of DevOps transformation was all about “Ops people, learn to write code.” And we’re all like “Eh, okay.” But I feel like the pendulum has swung. I think now it’s about “Okay, software engineers, your turn.”

    Ashley: You’ve got to do ops.

    Majors: I have to learn to write operable services and to learn to maintain them and get called for them and – because what microservices has done, all of that complexity and logic has to be bundled up in the app. The app.

    Manuel: Exactly.

    Majors: And now it’s like we’ve been thrown to the four directions of the map. And what that means is that so much of understanding and operating your application has been pushed into the realm of operations and systems. It used to be that if all else failed you could attach a debugger to your app and just step through, see what was happening. Well, you can’t do that anymore now because the process died and jumped to another service, and another service. And part – the way that we instrument for observability is all about packing up the context of the application and shipping it along with it from hot to hot, one event per request per service, so that you can slice and dice and reason about your systems in a way that you used to be able to.

    Ashley: Doing that really changes the design. I’m thinking about Alan’s story of what I did back at StillSecure. What prompted that was when a customer called in and said, “I’m sitting in – we’re using your project and this dialogue box popped up and it says ‘An unknown error occurred’ and that’s all it says. What do I do?” Well, nobody knows what to do with that. So, we all just said, “Look, everybody does support in that case. Everybody does ops.” And that changed: “Okay, we have to figure this out.” And suddenly that was the problem of “How do we make this easier to support and easier to resolve issues with?” So, it does affect behavior and design or product.

    Majors: Yeah, absolutely.

    Manuel: I think microservices is – will actually do good for observability because if you do microservices correctly then each service is atomic. It does one thing and one thing only. It does that thing good. So, if the unknown is not taken out of it because you know exactly what this thing is supposed to do –

    [Crosstalk]

    Majors: Oh, no, honey, the unknown is never taken out of it. The unknown is never –

    [Crosstalk]

    Manuel: In my ideal world it is.

    [Laughter]

    There are no more unknowns –

    Shimel: In Ravyn’s world, yeah.

    Manuel: – in Ravyn’s world. Exactly.

    Shimel: Yeah. But I’m going to tell you something. The flipside to this, though, too with microservices, and I think we’re starting to see this now, is can you have too many microservices running? Does it – is the complexity load in there such that observability almost becomes meaningless or impossible?

    [Crosstalk]

    Majors: I don’t know that complexity can every really be created – no, it typically is not meaningless. It’s your only hope to understand this shit if that’s what you’ve done to yourself. But complexity can only be moved around. It’s just a question of tradeoffs, like which set of tradeoffs – some folks have gone whole hog on microservices and then they’ve found the edges to those design _____. But –

    [Crosstalk]

    Chase: What I find – I was going to say what I find challenging is both at Splunk – Autodesk was interesting. So, to your point, Alan, we used to make engineers buy bagels. You break things; you have to buy bagels. And so, we had a lot of bagels brought in. But both at Splunk and here at Tricentis we have a world where we’re supporting large enterprise applications and delivery of enterprise apps. We’re also – obviously all the microservices – and trying to apply these techniques across the board is very interesting, very challenging.

    [Crosstalk]

    Majors: Oh, yeah. There’s no one-size-fits-all.

    Chase: There’s not. There’s not. And almost to the point you made, Ravyn, observability almost has to be customized for what you’re observing. It’s – I haven’t had a chance to run Honeycomb yet, but for me my experience has been there’s always those little idiosyncrasies about how we run our ops, how we –

    [Crosstalk]

    Majors: There’s no such thing as a perfect _____. No one else can do it for you. It can be – it can get you started. It can give you a boost. But at the end of the day somebody’s got to fucking understand your software and roll that into your instrumentation. Nobody else can do that for you, let alone _____.

    Chase: And that’s where with AI ops, I won’t say it’s complete shit. I think there is some value there. Some of it is just about automating common occurrences. And maybe that’s not valuable but –

    [Crosstalk]

    Majors: – it’s an AI but they’re just using that to make – to raise lots of _____.

    Chase: Oh, sure. Okay. Yeah.

    Manuel: Didn’t work.

    Majors: Like, “Oh, I have an AI too.”

    Shimel: No, but we – and you’re right, AI is a money term. But ML ops is something that’s caught on a bit, more than, I think, than AI ops. And I think that’s really at the heart of it, which was to recognize repeatable patterns or recognize patterns that we can respond to.

    [Crosstalk]

    Majors: Although, I would still challenge – I think in far too many cases people are reaching for the complicated solution instead of fixing it at the start. They’re like “Ooh, I’ve got 50 bazillion e-mails so I – and all of that disk space so I need AI to solve it” instead of turning off the fucking e-mails and setting up a different kind of alert or some automatic _____. Just – is this really a problem we need to solve? Or could we solve it better by just not having it?

    Shimel: There is that.

    Majors: Sorry. That’s my last bitch.

    Shimel: Do we need to solve every – and I think Ravyn – or, not Ravyn – Charity, you were getting at it before. Do we need to solve every problem?

    Majors: Is it the most important problem? Is this going to be the biggest bang for our buck?

    Shimel: Yep. I mean, we used to have a –

    [Crosstalk]

    Majors: – or the _____. Like, senior engineers write software but very senior engineers don’t need software.

    Shimel: Yeah, no, I mean, look, when Mitchell and I were doing the security company we had a product, “Vulnerability Assessment of Management.” VAMP. Internally we called it the Bad News Generator because that’s what it was.

    [Laughter]

    It was the Bad News Generator. And the problem was we would go to an organization like Ravyn’s and it would pop up a few thousand vulnerabilities, and the security or the ops people would be like “What am I going to do?” And “Don’t worry about 95 percent of them. Let’s just worry about the top five percent.” And it was like – it was employment insurance, was what it was, because there would always be more vulnerabilities for us to find next year.

    But that being said, it really was the case where you’re not going to fix everything. We don’t have the time to fix everything even if we wanted to.

    [Crosstalk]

    Majors: And it’s not really doing a great job of giving you differentiation, letting you know what’s really important and what’s just noise. So much of these tools, they just give you so much noise and it’s almost worse than nothing at all.

    Shimel: Well, that’s why with the Bad News Generator people used to – so, Mitchell got the lucky part of designing it. I had to go sell it. That was much harder.

    Ashley: I remember the –

    Shimel: Much harder.

    Ashley: – guy from the federal government called and said, “This thing is spamming everybody in the Defense Department.”

    Shimel: Yes, “We have all these vulnerabilities.”

    Ashley: “How do I turn it off?”

    Shimel: Because we had this automated workflow before we had terms like AI and ML. But it is hard.

    Hey, guys, we’re coming down the home stretch here. We probably only have five minutes or so left. I wanted to – let’s look forward now. I’m big into looking forward these days. I’m hoping we’re going to do conferences in person again; we’re going to – I don’t know if we’ll go back to our offices but…

    What’s next for observability? How does this continue to evolve?

    Majors: What’s next is I think really wrapping traces in as a first, best tool so that – right now too many people are like “I have one tool for metric, one tool for logs, one tool for servability, another tool for –” they’re spending just ____ data again, every time. And it’s worse than that because you can’t actually seamlessly break down a chain of events, so you’re just copy/pasting this idea into that tool or hoping despite – because it seems to line up timewise, it’s actually bees in the logs – you should be paying to store this data once and you should be able to flip back and forth between loads. Are you slicing and dicing? Are you doing aggregations at read time? Are you zooming in? Are you zooming out? Or are you slipping back so you can view it as a logger file by time?

    I think that observability tools like us and Lightstep – which are the only two observability tools out there by my technical definition – although, I will say I do think that this might be the year that that begins to change. I know that so many of the big players have been working as fast as they can for years on the back end. They’re trying to catch up to where we sit technically faster than we can get catch up to where they sit in terms of the business. So…

    Chase: And I –

    [Crosstalk]

    Manuel: I like that you said that. Sorry, Kurt. That –

    Chase: No, no, go ahead, Ravyn.

    Manuel: – Charity – I just wanted to say to Charity –

    Chase: No, no, Ravyn, go ahead.

    Manuel: – about tracing, because that’s actually the solution that I am actually looking for, that I’m looking for –

    [Crosstalk]

    Majors: It’s got a lot of rough edges.

    Manuel: – for organizations. Yes, to be able to follow –

    [Crosstalk]

    Majors: But there are lots of problems we can’t solve any other way.

    Manuel: Exactly. So, I feel good that I’m doing it inherently, like I know what I’m supposed to be doing.

    [Crosstalk]

    Majors: You’re on the right path.

    Manuel: I think I’ve actually found a way to do that. But yeah, tracing I also think is the way to go. That is the only way you’re going to be able to know, because it’s taking you from point A to point B to point C to point D and you’ll be able to see that all. And there will be no unknowns. [Laughs]

    Majors: Oh, honey. You keep saying that.

    [Crosstalk]

    Manuel: It’s my dream. That’s – let me live my little dream.

    [Laughter]

    Shimel: Kurt, what do you got?

    Chase: Yeah, well, where mind – excuse me – where my mind goes to in the future is what are we doing in the education system to really disseminate this information, how monitoring software is developed, how we monitor it. And then also “observability” is such a loaded term because I know at Splunk they would consider observability something much different, for running a SOC and things like that.

    [Crosstalk]

    Majors: Oh, no.

    Chase: So, I think it – I mean, at its core observability is what it is. But how it’s implemented for different disciplines and how they use it, I think companies would definitely take exceptions with some of the things we’ve said here –

    Majors: For sure.

    Chase: – and what observability means to them. And I would just say keep that in mind, that it is a huge field and I think we’re going to see further improvements. I love the tracing. And being able to have that in your tool set would be fantastic.

    Shimel: Charity, final thoughts before we wrap?

    Majors: Oh, I am just enjoying the fact that some people are so interested in this that they’ll show up and argue about it all the time. So…

    [Laughter]

    Shimel: It’s all good. Hey, you know what? It’s a living. Mitchell, as usual, I’m going to give you the last word.

    Ashley: Well, I’ve learned two things today. Don’t call my tool an observability tool or an AI tool unless it really meets the definition, or I’m going to get crucified.

    [Laughter]

    No. No, what – I mean, I love this conversation. I learned a ton. I always do when we have these – what’s so cool to me is that we’re talking about operations and we’re talking about this part of our world and we’re not talking about better ticketing systems. We’re talking about really meaty, interesting things that help us do our jobs better, do a better job for the company, for the customer. That’s exciting to me. It’s just really cool. I enjoy it very much.

    Shimel: Join us for our next show where we’ll be talking about better ticketing systems.

    [Laughter]

    No.

    Chase: And what argument did I miss? I was –

    [Crosstalk]

    Shimel: No, no, no, no. It’s all good. You know what? I enjoy having really bright, smart people who are willing to express their opinions. And when I knew we had this panel, it was the lineup, I knew there was going to be no shortage of opinions, nor any backing down. So –

    Ashley: We’re going to call the next one “The observability mix-up.”

    Shimel: Right.

    Majors: We’re going to have to call it “Observability AI.”

    Chase: I still want to see that sticker –

     Shimel: “AI Observability.” All right. Hey –

    Chase: We need to see the sticker Charity was holding up. We need to see that.

    Majors: Oh.

    Shimel: Oh, we never got to that. Yes.

    Chase: We have to see that.

    Majors: I’ve got a few of those. I’ve got “Release := deploy.”

    Shimel: You’ve got a lot of them here.

    Majors: I’ve got “Ship dangerously.”

    Chase: Yeah, I love that one.

    Majors: “Deploy on Fridays.”

    Shimel: You know what? You used to be able to get these things on conferences. Now we’re reduced to showing them on Zoom.

    Majors: Yeah, you have to.

    Chase: Yeah. Exactly.

    [Crosstalk]

    Majors: “Perhaps this is a good day to deploy.”

    Chase: Maybe.

    Shimel: Okay, yeah. Perhaps.

    Chase: It’s Friday. It’s not a good day.

    Manuel: No, Friday is not a good day.

    Chase: That’s the one, right there.

    Shimel: And there it is: “I deploy from Cron.”

    Ashley: That’s the one to end on right there.

    Shimel: That’s is. We’re going to wrap it. Hey, you’ve watched devopsy.kinsta.cloud. It’s sponsored by Tricentis. Many thanks to them for sponsoring. Just wanted to quickly mention Kurt is from Tricentis. Charity is from Honeycomb, and it’s Honeycomb.io. Go check that out.

    Majors: We have a very generous ____ here.

    Shimel: A very free _____. Ravyn, you can start playing with it now and see if it’s something that gives you tracing and so forth. Mitchell, thanks for joining us.

    Ashley: You bet.

    Shimel: We are back – actually, we are back with a roundtable next, which will be open to the public. Looking forward to having you on for that. But until then, this is Alan Shimel for DevOps Unbound. Have a great day, everyone. We’ll see you soon.

    [End of Audio]

  • Learning ‘The Third Way’ of DevOps – Continuous Improvement

    Learning ‘The Third Way’ of DevOps – Continuous Improvement

    I refer to the third way of DevOps–continuous experimentation and learning, as simply continuous improvement. In some sense, there is no way to learn the third way of DevOps because it is, by definition, a moving target. It does make sense to discuss what is needed to create a continuous learning organization for DevOps continuous improvement that builds on mastery of the first way and second way of DevOps.

    As indicated in my blog Learning ‘The Second Way’ of DevOps – Continuous Feedback and my prior blog Learning ‘The First Way’ of DevOps – Continuous Flow, it is clear many organizations are struggling to realize well-engineered DevOps, even the first way of DevOps, continuous flow.

    The first way of DevOps, continuous flow, is the foundation for DevOps. Once mastered, an organization will naturally want to master the second way of DevOps to achieve the higher level of benefits that continuous feedback offers, when engineered well. Once the second way of DevOps is achieved, then an organization will naturally want to master the third way of DevOps, to sustain and grow their DevOps capabilities.

    DevOps is an evolving body of knowledge. There is no one-stop shop that covers all roles and the complete range of skills required for DevOps. A DevOps-competent workforce requires an ongoing commitment to DevOps skills development and mastery. Enterprises must approach DevOps training strategically to leverage training resources to their best advantage.

    Learning Continuous Improvement

    There are no training resources dedicated solely to the third way of DevOps—continuous improvement. Many of the same resources as the first way of DevOps and the second way of DevOps do have some relevant information, though. Analysis of DevOps results, third-party research sources, industry sources such as publications, articles, conferences and tool vendors roadmaps for DevOps and related frameworks including site reliability engineering (SRE) and AIOps are good sources of information that stimulate self-directed teams to seek the training information that is most relevant to the organization’s DevOps improvements.

    Why Is Learning Continuous Improvement Important to Engineering DevOps?

    The third way of DevOps—continuous improvement—requires an organizational commitment to continuous learning and mastery at the most advanced level. Typical skills needed include implementation and operation of advanced practices for microservices design and operations, advanced continuous testing, multi-pipelines, value stream management, SRE, observability, DevSecOps and AIOps. Retention and development of highly skilled and hard to find experienced DevOps professionals and SREs makes it important to develop skills in-house.

    How Is Learning Continuous Improvement Accomplished?

    The following are recommended engineering practices for learning continuous improvement:

    • A continuous learning culture encourages individuals to continuously gain knowledge and exchange experiences and ideas for DevOps and SRE.
    • Mentoring is an explicit program.
    • Retrospectives are structured to thoroughly analyze significant, unexpected events and determine solutions that will prevent the events in the future.
    • An active online library of DevOps and SRE training resources is maintained that tracks the latest and most innovative approaches to DevOps and SRE.
    • DevOps and SRE teams evolve their skills and advance their careers by engaging with industry thought leaders to continuously update training content.
    • Partner with trusted education and training partners ensure the highest-quality learning content and tools are made available to DevOps and SRE teams.
    • A portion of DevOps and SRE teams time is allocated for training and education.
    • Cross-team sharing of recommended engineering practices is proactively pursued using the concepts of Yokoten—proactive sharing of practices between teams.

    What Is Needed to Implement Learning Continuous Improvement?

    The following are recommended engineering practices for implementing a learning program for continuous improvement.

    • Assign roles and responsibilities to experts for defining a training program for DevOps continuous feedback (e.g., DevOps sensei, training PM, evangelist).
    • Conduct a continuous feedback DevOps skills gap assessment.
    • Define a training strategy that best address the gaps and fits the needs and priorities of the organization.
    • Define a training budget that includes expenses and time allocations per role and skill.
    • Determine which skills and roles can be covered by inexpensive self-study resources.
    • Determine which skills and roles warrant more costly commercial training.
    • Identify training goals for each role.
    • Ensure recruiting job profiles match skill requirements for DevOps skills that are most relevant to the organization’s strategic goals.
    • Create a continuous learning culture that encourages individuals to continuously gain knowledge and exchange experiences and ideas for DevOps and SRE.
    • Build an active online library of DevOps and SRE training resources that tracks the latest and most innovative approaches to DevOps and SRE.
    • Engage with industry thought leaders to continuously update con tent for DevOps and SRE teams to evolve their skills and advance their careers.
    • Partner with trusted education and training partners to ensure the highest-quality learning content and tools are made available to DevOps and SRE teams.
    • Put in place means to make training resources visible and available.
    • Set training goals for teams and individuals.
    • Put in place incentives and recognition programs for training.
    • Track training progress.

    Some of the best sources of DevOps training resources are listed below:
    1. The DevOps Institute
    2. Partners of The DevOps Institute, such as Learning Tree International and Xellentro.
    3. Udemy.com
    4. Recorded talks and publications of IT Revolution
    5. Books such as my own Engineering DevOps, by Marc Hornbeek.

    What This Means

    The third way of DevOps—continuous improvement—requires an organizational commitment to continuous learning and mastery at the most advanced level. Typical skills needed include implementation and operation of advanced practices for microservices design and operations, advanced continuous testing, multi-pipelines, value stream management, SRE, observability, DevSecOps and AIOps. Analysis of DevOps results, third-party research sources, industry sources such as publications, articles, conferences, and tool vendors roadmaps for DevOps and related frameworks including site reliability engineering (SRE) and AIOps are good sources of information that stimulate self-directed teams to seek the training information that is most relevant to the organization’s DevOps improvements.

    Organizations must approach DevOps training strategically and continuously to leverage training resources to best advantage. To learn more about my blueprint for learning all things DevOps, refer to my book Engineering DevOps.

  • Then and Now: The Evolution of DevOps Culture

    Then and Now: The Evolution of DevOps Culture

    There are many available resources dedicated to delivering better business value faster. Based on this advice, many organizations and teams are adopting common software delivery practices — even outside the usual use case of providing software (take, for example, SpaceX teams using agile, or leadership teams that use scrum practices and have daily stand-up calls). This progression into faster, better, cheaper and easier – no matter what you’re delivering – is what makes incorporating DevOps into business practices critical to delivering unmatched value for end users.

    Then and Now

    A great perspective to consider when approaching the subject of DevOps is how technology has changed just about everything – how we build, connect, share and educate. Up until the 2000s, for example, the standard turnaround time for taking and sharing pictures was one to three weeks, if not longer. This process used to involve film cameras, darkrooms to process the photographs, photo finishing labs and hard copy, physical prints. Today, our mobile devices allow us to snap hundreds of professional-quality images and share them immediately online.

    culture evolution

    These earth-shattering changes don’t apply to photography alone, but to many other industries including financial services, banking and IT. Consistently, technology is finding new ways to disrupt industries, priming them to be more efficient to fit into our fast-moving lifestyles.

    The DevOps Life Cycle

    Now, let’s look at software delivery. In the past, the process of traditional software delivery was linear and sequential; waterfall methodologies offered little flexibility and did not succumb to the nature of change and demand for technology services we see today. What we’ve experienced with software delivery now uses DevOps to leverage new insights through people, processes and technology, giving organizations the ability to deliver continuous business value. Yes, that means businesses are adding value to our lives all the time, whether you see it or not. For technology organizations, this process for delivering business value is modeled through what we call the DevOps life cycle — understood through the graphic below.

    DevOps culture evolution

    Getting to a DevOps Culture

    Often, an organization’s challenges with DevOps do not lie in the technology or processes themselves, but in the people part of the equation – how we work together to deliver value via cross-functional teams. To understand what building a DevOps culture means, we need to look at what culture encompasses and how our people, process and technology work together to enable a successful DevOps life cycle.

    culture evolution layers

    This layered graphic features all the various components related to culture. Each layer encompasses the next.

    ·      Mindset

    ·      Values

    ·      Principles

    ·      Practices

    ·      Tooling

    Getting to a DevOps culture involves articulating change both from the bottom up and top down in our organizations and is, therefore, no easy feat. Before we can introduce or discuss tooling, we need to understand how our practices and principles help us achieve our outcomes. Each inner layer requires the outer layer to be successful before we can truly get to defining and strengthening a DevOps culture.

    Here are my tips for getting to a DevOps culture:

    ·      Learn from and develop skills from mistakes

    ·      Focus on the important priorities

    ·      Capture collaboration

    ·      Perfect your tooling

    The DevOps Principles

    DevOps is the collection and collaboration of people, processes and technology to deliver software value continuously. Here are some core DevOps principles you can use to guide you on your DevOps journey.

    Create With the End in Mind

    High-performing teams work best when leaders and individuals understand their role in the team. Therefore, creating with the end in mind involves reassessing and often asking big picture questions, like “Who is this delivery for?” and “Why are we doing it this way?” Keep the vision in mind and make adjustments along the way.

    Even the most famous photographs in history involved many adjustments and numerous attempts to get it right. Creating with the end in mind means knowing we may not have everything right the first time around. That’s okay! Similarly, life isn’t without its own distractions, conflicts and changes. Keeping the target outcomes in mind helps us align to initiatives and actions, and to question when they don’t align.

    Cross-Functional Teams

    Many products and deliveries involve multiple people. For example, some of the best Marvel cinematic universe (MCU) movies were made thanks to the help of multiple production teams and partner companies coming together to build the powerhouse we know today. Not to mention, in my opinion, some of the best superhero movies of our time. We succeed with software delivery in the same way: by having fully responsible cross-functional delivery teams.

    The best engineering teams I’ve been on are made up of independent and autonomous individuals who know how they can contribute to the work in progress. They’re also willing and unafraid to collaborate with others to get this work done.

    Commitment to the Journey

    “I am not here to be right. I am here to get it right.” – Brené Brown, from the book Dare to Lead.

    Even with cross-functional delivery teams and creating with the end in mind, our teams still may not get everything right the first time around. Committing to DevOps is about accepting any failures that come during the journey. Committing to the journey doesn’t mean you can’t define what it involves, even if it’s out of the ordinary. Experimentation is also a part of how we, as developers, adapt to change and adopt novel approaches to delivering faster, cheaper and better software.

    The DevOps Principles as Guides

    Use these principles as the foundation for how to work together, but also use the right tools, and define and improve processes along the way. In photography, there are three components that serve as building blocks to the final images we capture: aperture, shutter speed and ISO. In DevOps, it’s our people, processes and technologies. Achieving high performance with DevOps involves all of these components working together and adjusting to our environment, situation or team.

    The key to getting to the results you desire? Not getting blindsided by any single component of getting to that result.

    The DevOps Practices and Tools

    I like to consider this section the call to action; the one says to our teams and individuals, “Gear up!” No reasonable photographer would leave for a photoshoot without their camera and camera gear, right? Similarly, DevOps practitioners and teams need the right tools to get the desired results.

    Source Code Management

    Source code management is about allowing teams to collaborate, have version control, correct and revert software code. This is fundamental for many teams to increase not only visibility and collaboration, but also start automation workflows.

    If you are an individual just starting out with writing code, I recommend learning the basics of Git, such as how to add, commit, push, merge, branch and clone code. If you’re a team, I recommend discussing the practices and standards you want to use for collaborating. Always fit your Git structure and workflow to your codebase and team dynamics.

    Continuous Integration

    What continuous integration (CI) does is take code away from local machines to package it into a deployable artifact in a CI server. What this is designed to do is automate the integration of newly committed code changes. This process is often triggered by code changes pushed into your Git repository. Using checks and other tools, teams can version and release artifacts that have already been vetted for quality and security.

    Continuous Delivery

    Continuous delivery, on the other hand, takes artifact code to production environments. CD aims to deliver code changes of all kinds to users in a fast, repeatable and safe manner. CI/CD consists of both CI and CD pipelines working together to deliver code to users.

    Infrastructure-as-Code (IaC)

    Infrastructure-as-code allows users to define hardware through code. We can improve how we provision, configure and maintain hardware systems. Infrastructure provisioning tools like HashiCorp’s Terraform and AWS Cloud Formation, configuration automation tools like Ansible, Puppet and Chef and package managers like Helm allow teams to accomplish much more when software scales and demand increases.

    Secrets and Identity Management

    Security is everyone’s responsibility. Secrets management and identity management tools make accomplishing DevSecOps easier by securing and managing sensitive information. Teams should consider how they are handling secrets and users within their architecture. HashiCorp Vault, KeyCloak, KMS and other similar tools can help you achieve your goals around security.

    There are many other practices and tools DevOps teams and organizations can use to achieve their desired outcomes. What works for one team may not work for another — hence the need to innovate and experiment in a controlled and well-led manner. Other practices that can increase your team’s ability to deliver value include value stream mapping, progressive delivery, GitOps and chaos engineering.

    People, processes, and technology belong together in a DevOps culture. While practice makes perfect, what’s more important than perfection is gathering the right answers along the way. Everything doesn’t need to be right the first time around. Just remember that no leader, individual or team in DevOps is alone. We started by defining what it means to build a DevOps culture, change mindsets and set principles and practices, so now I ask you: What does DevOps mean to you?

  • Data and Tools Don’t Solve Problems, People Do

    Data and Tools Don’t Solve Problems, People Do

    When developing software, many managers believe that data and tools will solve all their problems. If they just have enough data or the right data analysis tools, development and delivery become automated and decisions are made without human input. But in reality, humans are a necessary part of the process. There are a few pieces of data you may want to start gathering, but you often don’t need a lot of data to make the right decisions. And sometimes, you don’t need data at all. You can use the data and tools you already have in place, but your most valuable asset is your brain. To illustrate this point, I’ll provide a few examples.

    If it takes too long for your team to fix a bug, you can dig into data from your bug tracker or ALM tool, but reasoning about the process is also a necessary step. It’s not sufficient to simply look at some reports from your tooling. In fact, reasoning about your process may be all you need. You may already know where your bottlenecks are and where the waste is. Nevertheless, when you need to dig into the data, the point solution tooling you use could be the best place to do so.

    If your epics are moving too slowly, you can use a report provided by whatever tool you are using to understand lead time, as well as the cycle time between states. You can drill down from there to look at stories as they flow from inception to completion and use whatever tool you are using to look for bottlenecks and wait states. Likewise, for build through deployment to production, you can look at data out of your tooling to understand how long it takes to go through each phase.

    To illustrate this further, relatively few companies today are deploying more than once per sprint, and many, less often. Until you get your lead time down to two weeks, measuring how much undeployed code inventory has built up is useful only for its shock value. And then, you don’t need to update it every month. Measuring lead time is sufficient, and easier.

    Whatever tool you are using, however, can only model what you have in the underlying systems. If the underlying systems don’t model the wait states, your analytics aren’t going to give you any insight into your delays. Include wait state columns in your scrum board or kanban.

    A great approach to continuous improvement is the Toyota kata, and there is a great book by that name. You should read it and then implement it in your organization. But know that this can’t be tooled-up or made automatic. You do need to be able to measure for the intended improvements, but it still takes humans to do the thinking. Toyota was successful because its approach ingrained in every employee the notion of improving their processes. Toyota never minded people seeing what it was currently doing because Toyota knew that its reason for success was its ability to change and to improve faster than the competition. This requires humans. Brains.

    Example: Maybe you are looking at metrics about builds from a tool. You might look and see that you have multiple builds per hour. But if you don’t really understand the metric or what it is you’re looking at, you might not realize that some individual builds may only be built once per day, or worse. Or you might not realize that your dev team’s builds only go to QA every other week, slowing down your overall flow of value.

    Gary Gruver, in his book, “Engineering the Digital Transformation,” says, “Most organizations don’t really think closely about how value flows through their organization,” (page 99). I agree. Moreover, they too easily buy into the promise of not having to think closely about how value flows.

    Gruver continues: “The biggest change for the industry is to create a culture of continuous improvement.” He doesn’t say “tooling for VSM” or tooling for metrics. He says culture change. The stuff of people and brains.

    Gruver also says, “Software flow is very hard to measure. … That is hard with software because the size of each feature is different, which makes it very difficult to have a quantifiable measure of flow.” Another way to say this is that you shouldn’t have unrealistic expectations that your tooling will make the decisions for you. You should expect to have to think. And then maybe dig into the data already provided by various tools in your IT or software dev value stream, and even modify some of those to capture additional data. You should expect to have to analyze each component value stream, each type of work, the mix of work types and the variability of each.

  • Review, Review, Review

    Review, Review, Review

    Ossification is an issue in IT, and always has been. It is borne largely out of staff and management knowing what to expect from existing software/infrastructure, and the deep product knowledge that allows teams to make existing systems do what is needed. But this becomes a series of rocks that the ship of progress smacks into.

    Contrary to popular belief, DevOps and other new methodologies are not immune to this problem. Quite the contrary, we’ve spoken with people that say “We’re a Jenkins shop,” which sounds exactly like “We’re an Oracle shop” of a decade ago. The wave just ushered in new tools, it didn’t change tendencies. OK, it did more than usher in new tools, but it didn’t change tendencies.

    So, it’s time for a friendly reminder: Review what you are doing, review cloud usage, review tools in the toolchain, review methodologies, review security posture (which has suffered in this environment for far too many shops) and review customer satisfaction.

    Many organizations used the push of Agile and DevOps to update stagnant parts of the infrastructure and development toolsets. Don’t let them become stagnant again. Just as you don’t want change for the sake of change, you don’t want to stay with one tool or methodology just because it is comfortable.

    This isn’t some easy task that I ⁠— or anyone else, no matter how much they try to tell you differently ⁠— can dictate to an organization. There isn’t a handy list, because no two organizations are exactly the same. Not in toolsets, not in culture and not in appetite for risk. The only thing we can offer is guidance to look closely at where you are, what’s available in a given area ⁠— Application Release Orchestration, for example ⁠— and evaluate the available tools just like you were making a fresh choice and did not have a tool in place. Then do the same for processes. Is this the best process for what we’re trying to achieve? Are there tweaks we can make to it? Should we completely change the process to take in changes in environment that have happened since we adopted this process?

    Finally, if you don’t have someone responsible for monitoring cloud costs, appoint someone. Get them the tools, help them understand where the money is going and what cloud-based server usage is. Could these servers be spun down outside of business hours? Could we follow the sun better by staggering cloud data centers in which cloud servers are active, rather than hosting them always in one location or leaving them on 24/7 in multiple locations? Is one team using a cloud feature no one else is? If so, how does it apply, and are the other teams not using it because they don’t need it, or is there some other solution they’ve adopted?

    Basically, question everything. We’re launching a new project that will be a simple dynamic app. Make a few selections from a limited list, get a summary of what you should do and why. It is all really quite simple. Should we host in a web page or make it a standalone mobile app? If mobile app, what tools for development and what targets? Integration of Git — should it be local or on one of the big providers? Do we want any team member to trigger builds on commit or do we want someone managing branch merging before-hand?

    These are all things we already do on other projects, but we’re digging in and seeing what could be better. What could we improve, and what would the rewards for that improvement be? It takes a bit of time and some discussion. We’re confident we’ll burn through it in the next day or so and come up with a plan for development and architecture that best suits the app, but is informed by where we’re at now. It’s been a fun exercise, to be honest. If nothing else, it gave us a strong reminder that any given cloud provider’s “simple” pricing is not that simple, and we need to pay more attention to app characteristics versus fees.

    You’ve rocked it this far, even kept the lights on during the great shift to working from home. Don’t stop improving the environment though, find innovative ways to make it better and keep clipping along.

  • Increasing Software Velocity While Maintaining Quality and Efficiency

    Increasing Software Velocity While Maintaining Quality and Efficiency

    At Compuware, we have been exploring the need for continuous improvement across three key dimensions of software development: velocity, quality and efficiency. Given today’s current market demands and the importance of digital services, this discussion couldn’t be timelier.

    On my previous piece, I focused on quality — how to maintain and improve it as (supposedly) competing demands for velocity and efficiency increase simultaneously. This article will focus on increasing velocity and how it is possible to do so without compromising quality or efficiency. As I discussed in my earlier piece on quality — automation, integration and measurement are once again the key.

    Compuware recently conducted a survey finding the overwhelming majority (97%) of application development managers said IT is under more pressure than ever to deliver software innovation faster. Higher demands for innovation are being imposed universally across platforms, from the cloud all the way back to the mainframe.

    Mainframe software delivery is a particular area of focus, with 95% of respondents noting their IT teams are under pressure to adopt modern development practices on the mainframe in order to accelerate overall software delivery and innovation. Adding to this pressure, organizations reported they expect to increase the frequency of new mainframe application feature deployments by an average of 41% over the next year, compared to 38% for non-mainframe applications. This shouldn’t come as a shock, given the proliferation of modern web-based applications placing unprecedented demands on back-end mainframes.

    Let’s take a closer look at how automation, integration and measurement are equally applicable to the goal of advancing velocity.

    Automation

    Throughout the software lifecycle there are many opportunities for automation — from software design, to development, to build, test, deploy and, ultimately, the production state. The more of these steps that can be automated, the faster developers can work.

    Perhaps the biggest area for potential time savings is testing, because of all the phases of the software lifecycle, testing assumes the most manual labor. The Vanson Bourne survey found test automation to be the single most important factor in accelerating innovation, according to 90% of IT leaders.

    On the mainframe specifically, development teams spend more than half their time on testing during the release of a new mainframe application, feature or functionality. If you can automate more testing, you can roll out applications faster while maintaining (or even enhancing) quality combined with an extremely high level of developer efficiency.

    Integration

    Integration helps software development technologies interoperate with each other. An integrated pipeline leverages existing investments and enables developers to build, test and deploy faster while providing the additional benefit of reducing errors resulting from human intervention (furthering quality). Integration also decreases the manual labor needed to execute and manage workflow within software delivery processes. When these technologies integrate out of the box, setup time is saved and productivity gains can be realized significantly faster. 

    For example, the integration of Compuware Strobe application performance solution with Jenkins, makes it easy for developers to take performance measurements early and automatically as part of a Jenkins CI/CD pipeline.

    Ongoing Measurement

    Measuring velocity is critical. Without measuring, it’s impossible to know how changes are impacting the software development process and what areas need to be improved. Measuring is the key to understanding if there are any activities that can be sped up across the phases of build, test and deploy. This enables development teams to find and address bottlenecks with a great level of granularity. Some common metrics for measurement include mean-time-to-build (MTTB), mean-time-to-delivery (MTTD) and mean-time-to-repair (MTTR).

    Conclusion

    Velocity does not need to exist at the expense of quality or efficiency — on the cloud, the mainframe or any other development platform. Like quality, the key to achieving and improving velocity lies in the disciplines of automation, integration and ongoing measurement. The next piece of this series will explore the keys to maximizing developer efficiency amidst near constant demands for exceptionally fast, high-performing rollouts.

  • Enterprise Kanban Implementation: Where to Start and How to Scale Across an Organization

    Enterprise Kanban Implementation: Where to Start and How to Scale Across an Organization

    The Evolution of Kanban 

    At its core, Kanban is a management method designed to improve customer satisfaction by creating fast and reliable workflows capable of delivering products and services of superior quality.

    Kanban originated in the automotive industry of Japan and was originally developed as a method for improving manufacturing efficiency at Toyota. However, at the beginning of the 21st Century, key players within the software industry realized that Kanban could be used to positively change the ways in which products and services were delivered.

    The result is astonishing. In less than a decade, Kanban has become the most popular tool for software development and IT teams, surpassing the Scrum framework, which has been the de-facto standard until recently. According to the State of Scrum survey, 60% of Scrum teams utilize the Kanban method, which is a testament to the benefit it brings.

    Kanban left the realm of the automotive industry and conquered the IT world, but now, with an increased focus on flow and by harnessing advances in computing technology, it is about to expand its reach to the entire business. Leaders all over the world have realized that Kanban has the most flexible architecture that allows it to scale across the enterprise in a natural and human-centric way. More and more companies are embracing the Kanban “start with what you do now” principle and are on their way to true business agility. But, how do you go about that exactly?

    A Bit of Expectations Management

    Managing an enterprise Kanban implementation starts with expectation management. Senior leaders that sign off on multi-million dollar budgets for agile transformations should be informed that business agility is a strategic choice and not a one-off project. No company in the world has ever managed to become agile overnight. There’s no shortcut to that, nor is there a recipe to follow.

    Even if your company is doing well, unless you push hard to improve, sooner or later your competitors will catch up. So, in terms of expectations, people should know that there is no “we’re done” state. The constant, never-ending process called continuous improvement is what makes you resilient in the long run.

    If there’s one area where Kanban really shines, it is continuous process improvement, which will almost always lead to qualitative and quantitative overall improvements. Applying Kanban to a process will typically drive up to 2000% improvement—and that’s not a typo. Scale that on a company level and let the numbers speak for themselves.

    Start with What You Do Now

    Once an organization has made the decision to implement Kanban, it can be tempting to jump quickly into building the system itself without conducting a thorough analysis of enterprise operations and processes. As a first step, it is vital that organizations assess their current situation to understand where concrete improvements can and need to be made. This includes aspects such as:

    • Processes: Is production stable and predictable? Answering no to this question doesn’t mean you can’t properly create and scale Kanban in your organization, but it does mean that you need to put some extra buffer and flexibility into your planning. 
    • Involvement and Investment: This is where the notion of scale is important. A company-wide Kanban would be immensely more effective than local implementations, but it will require involvement from senior leadership and a significant budget. It is often a good idea to start small, learn fast and then go full speed.

    Kanban’s flexibility allows it to be overlaid onto existing workflows, systems and processes without disrupting what is already successfully being done. It will, naturally, highlight issues that need to be addressed and help to assess and plan changes so their implementation is as non-disruptive as possible.

    Kanban’s versatility allows it to be introduced incrementally, and sympathetically, to all types of organizations without fear of over-commitment or culture shock. This makes Kanban easy to implement in any type of organization as there is no need to make sweeping changes right from the start.

    If you are starting on a smaller scale with a goal of growth, once the Kanban mentality is put in practice and successfully functioning, it is important to set goals for future scale—six months, one year and three years down the road. If you conduct a pilot where Kanban is embraced and productive, don’t lose time and momentum waiting to deploy it across the rest of your business or enterprise. 

    Training

    Training is an aspect of Kanban implementation that is often overlooked. It is important for the team designing and implementing the system to consider every level of user across the organization. What seems logical, obvious and understandable in the context of a planning meeting may not prove to be so on the shop floor.

    While a well-designed Kanban system may appear visual and obvious, end users who don’t understand the theory behind it, will not always trust it. This is especially true for mature employees who are accustomed to doing things a certain way based on learned habits. When introducing Kanban, put yourself in the shoes of someone who does not know anything about it and may not initially embrace the idea of change. As straightforward as Kanban should be, if the end-user is not on board, it won’t work.

    Training resources such as Kanban University offer certified Kanban courses and a global network of resources for organizations to help onboard employees and users at every level, build awareness and assure quality in Kanban implementations. Upfront and ongoing investment in such training teaches organizations how to fully maximize the value of Kanban, and train them to understand and visualize systems of work to continually improve and deliver effective results. 

    Past the Team Kanban Phase

    If you do Kanban right, many tough questions will have to be answered, including, “How do you connect the high-level strategy with the goals on a team level?” and “How do you bridge the gap between planning and execution?” The good news is that Kanban has an answer for these questions, too.

    Kanban integrates naturally with strategy deployment frameworks such as Hoshin Kanri or OKR. All you have to do is use Kanban across every level of the organization. Practice shows that companies would typically use three flight levels. The lowest level is the team level, where work gets done. The second is the coordination level, and the third is the strategic level. All levels are connected to one-another, so that strategic initiatives (level 3) are broken down into projects (level 2), which are then broken into work items for the teams to work on (level 1).

    When you “Kanbanize” all three levels and connect them into a single whole, then you get an extremely powerful management tool with the following benefits:

    • All work is visible, starting from the very top to the very bottom.
    • Impediments across all levels are explicit.
    • There is bidirectional feedback loop between all flight levels.
    • The senior leaders can quickly change the course of the whole company.
    • Much more accurate and reliable status reports that are based on real data.
    • Through the usage of proper Kanban tools, the data gathered on the team level can be used for forecasting on the project level. This helps leaders take informed decisions.

    Driving Enterprise Agility with Kanban

    The demands of businesses, customers and markets will always change. And in response, businesses must be able to quickly pivot and adapt to meet shifts in expectations and requirements. Successfully implementing Kanban across an enterprise requires a deep understanding of the theory that stands behind it and upfront investment to determine the best approach to make it work within the context of unique organizational challenges and objectives. When implemented and designed appropriately–with buy in and support at every level–Kanban enables enterprises to be agile amidst change, continually improve vital processes and the ability to scale to meet the demands that tomorrow can bring at a moment’s notice.

  • How to Properly Approach Automation

    How to Properly Approach Automation

    Imagine if you backed out of your driveway, tapped the gas and BAM! You are at work. Nothing to worry about along the way, no traffic congestion, no other cars to worry about, etc.

    That is, unfortunately, how many of us approach automation. “Fire and forget” is real, until a problem crops up. I could talk about some complex environments, such as ARA, in a containerized environment with multiple languages, accessing mobility-limited databases and targeting several client platforms, for example.

    But we’ll keep it simple, and I’ll make an admission. Backup was one area where we at Ingrained Tech fell into the fire-and-forget trap. One part of our backup system runs desktops out to a server, the server (and other servers) out to a NAS and the NAS out to the cloud. I don’t think our security peeps will freak out if I mention we use duplicity for this backup and Backblaze for our backup cloud. Backblaze is a dream to use compared to others we’ve looked at, and the pricing is competitive, so it suits our needs well.

    A few weeks ago, we realized that while backups were completing successfully and verification was fine, it was taking nearly the entire week to do our weeklies. That’s bad news. We back up terabytes of data, but this is 2020, and our drive IO/Network can handle it. We started by looking at both of those things, and sure enough, it was drive IO causing our problems.

    Unsurprisingly, reading the disk and creating a collection of encrypted tarballs (a super-simplification of what duplicity does, but gets the point across) on those same disks was causing problems. After investigating, the one build that occurs on that server was suffering from backups stretching out, too. We had seen occasional issues with the build, but nothing jumped out as the source, and it was only one night every month or two. It was on our list. You know, the same prioritized list you have. And this build wasn’t nightly-critical. You get the idea.

    This was a relatively urgent problem, as soon as last week’s backup would not be complete when this week’s tried to start. Not good.

    We took our DevOps mentality to the problem and discovered a simple solution that solves the issues. The backup NAS is RAID 50, so it can handle a lot more abuse than the server that was slowing down backups. So, we took the simple expedient of rsyncing the source server to the NAS, then using duplicity to tar and encrypt before sending it on to the cloud buckets. Internally, we do not believe we need to encrypt backups because if someone is in our network, all of that data is laying around. By putting duplicity and both the tar function and the encryption function onto the backup NAS, we moved it to a far less busy system. Backup times dropped from days to hours instantly, and our unrelated build problems went away.

    We have to rework the automation somewhat to make sure validation is, well, actually validating the data is correct. But we’re well on our way to having a stable, faster system.

    How does this fit with DevOps? We see the same thing happening often in the automation rush of DevOps. Yesterday’s best solution is not necessarily tomorrow’s best solution. “Automate and watch” should be the catch-phrase. Improvement comes from both DevOps culture and paying attention to changes in the tools market.

    As Alan pointed out, someone will eventually win the end-to-end DevOps rush—since I feel the urge to inject my opinion on who stands the best chances, expect a blog on this topic soon. Until then, knowing what your options are and how to best achieve your goals–even on problems that were “solved” last year–is your best path for continued improvement. Yes, it means rework, but just like reworking your code, the point is to streamline things, get closer to your organization’s goals and keep improving after the low-hanging fruits of DevOps have been gathered. The rate of improvement in several areas of DevOps is stunning, so you may find you actually like a new tool better than the one you chose a couple years ago. And remember, knowledge is power.

    — Don Macvittie