devopsy.kinsta.cloud

Tag: automation

  • A Full Pipeline: CI/CD for Network Automation and Orchestration

    A Full Pipeline: CI/CD for Network Automation and Orchestration

    The intersection of CI/CD and network automation has grown substantially over the past few years. Network teams have seemingly reached the point in their automation and orchestration journey where it has become critical to implement a strategy to manage the testing, version control and deployment of both network changes and automation assets. 

    While CI/CD has grown in popularity, there are—at least on the surface—many differences in the types of pipeline implementations to which network teams want to apply the technology: Different tooling, different scenarios and different problems being solved. Despite the seeming disparities, there are some key themes that are consistent throughout most pipeline implementations. For instance:

    Network teams either want to develop pipelines to better manage specific changes to the network or they want to use pipelines to better manage the testing and deployment of automation and orchestration assets like workflows or scripts. Where initially those teams may have turned to network infrastructure-as-code (IaC), a significant (and growing) number are investing in automation asset pipelines. 

    The trust and resources being dedicated to these pipelines is just one of a growing number of indicators that automation and orchestration solutions are maturing and moving beyond initial implementations to large-scale, business-as-usual technologies.

    How Pipelines and Network Automation Complement Each Other

    A close examination of recent pipeline developments provided more clarity about the respective strengths of CI/CD tooling and network automation tools for network change management and life cycle—GitLab, GitHub, Jenkins and CircleCI to name just a few. Most network teams would prefer to stick with a “right tool for the right job” philosophy, wherein the best tool for any given task can easily be integrated into an overall solution. 

    In this context, the “right tool” philosophy looks at pipelines and at network automation/orchestration tools to identify what they do best. Then, a strategy is developed that maximizes the benefit of both technologies. Using this approach, teams can avoid the pitfalls of trying to solve everything with one or the other, leading to a suboptimal process.

    Some good current rules of thumb are based on two simple points:

    • Pipelines focus on creating consistency and repeatability by managing activities related to code versioning, testing coordination, code merging, code validation and approval. The primary benefit of pipelines is the ability to automate the change process.
    • Network automation focuses on generating the network payload and applying that payload to the network elements. This may also include automation of engineering and business logic steps that typically must be performed in a specific order. The best uses of network automation within the change process require attention to business logic, order of operations and network engineering rules.

    By integrating both technologies into a common pipeline, network teams can get the best of both toolsets. 

    The example below illustrates an integrated model where network automation tools provide the engineering logic for the generation, testing and deployment of configuration changes, while the CI/CD tools (represented by the orange arrows) carry the activities through a repeated, automated process, ensuring that the changes go through a consistent path with proper version management, approvals and testing. Multiple teams have successfully implemented this model and they are beginning to iterate their solution to incorporate their unique experiences and improvements.

    The widespread expectation from the industry is that, in addition to the integration of CI/CD and automation into a common pipeline, we will begin to see other indicators of maturation of orchestration and automation. Some of these are areas that are already taking shape, such as the shift in skillsets and competencies outside of traditional CLI-centric engineering toward DevOps and CI/CD skills, or the growing importance of quantifying the business impact of automation and orchestration. 

  • The Human Condition

    The Human Condition

  • DevOps Unbound EP 18 – Standardizing DevOps – TechStrong TV

    DevOps Unbound EP 18 – Standardizing DevOps – TechStrong TV

    Alan Shimel: Hey, everyone, welcome to another DevOps Unbound. If this is your first time catching a DevOps Unbound, welcome. DevOps Unbound is a biweekly video series where we talk about relevant topics in DevOps. In addition to the biweekly show, we do, it’s every four to six weeks, a live roundtable show where we invite the audience to participate and drive the discussion with online questions and comments and so forth, so we open it up to a live audience. Unfortunately, this show is not one of our live roundtables, it is prerecorded, but we have a great panel and I think you’re gonna enjoy it.

    I’m gonna get into today’s topic in a second, but before we do, let me introduce you to today’s panel. First of all, I wanna introduce a business partner of mine and my friend, Jayne Groll. Jayne, why don’t you introduce yourself to the audience, if you don’t mind?

    Jayne Groll: Thanks, Alan. Hi, everyone. I’m Jayne Groll, I’m CEO of The DevOps Institute, we’re a global member association and a certification body around topics related to DevOps. I’m really, really excited to be here and I’m hope everyone listening joins us at DevOps Institute and becomes a member.

    Alan Shimel: Fantastic. Welcome, Jayne. Joining Jayne, I’m happy to have Paul Bruce. Hey, Paul, welcome, and if you wouldn’t mind, you know, introducing yourself?

    Paul Bruce: Yeah, hey, Alan. Good to be back. My name is Paul. In my full time job, I work with Tricentis in a number of different areas. I’m historically a performance and reliability nerd. In my off time, I organize a number of events, help to co-organize volunteer things like DevOps Days Boston, just did o11yfest this year about observability, we had that conversation, I think, with Christine Yen a couple months ago.

    Alan Shimel: Good stuff. Mm-hmm.

    Paul Bruce: And generally, I’m a people advocate. I had to pull DevOps out of my LinkedIn title because there were too many recruiters that I don’t need, because I’ve got a pretty good job. But also, because people advocate really describes what I care the most about, right? People are some of the hardest part of the DevOps people, process, and technology, really, software. So, I really focus my time on that.

    Alan Shimel: That’s great to have you back on, too, Paul—good stuff. Next, we have a first time guest on DevOps Unbound today, a new member of the band, Sarah Baker. Sarah, welcome to DevOps Unbound, if you wouldn’t mind introducing yourself?

    Sarah Baker: Thank you. Wonderful. Welcome. My name is Sarah Baker, my pronouns are she and her. I’ve been working in Operations my entire career and I’ve moved into DevOps as something I always—I needed a name for it. I always wanted the people part of Ops and technology and DevOps was that. And I’ve been doing that as soon as I learned the term. And I’ve been moved and started working with Paul Bruce in the IEEE work group 2675, who is trying to create a standard for DevOps.

    Alan Shimel: Fantastic.

    Sarah Baker: I work for—and today, I work as the Manager of Biztech in Airbnb.

    Alan Shimel: Thank you, and thanks for joining us. That sounds like a great place for you to do this. And then the last member of our panel today is my co-host and another business partner, a long-time friend as well, Mitch Ashley. Mitch, if you wouldn’t mind introducing yourself?

    Mitch Ashley: You bet. Always good to have a good panel on with you, Alan. Mitch Ashley, I’m CEO of Accelerated Strategies, an analyst firm that’s focused on DevOps, and also on cloud native, cyber security, and digital transformation. I come to DevOps both as a practitioner and a vendor leveraging it, so very interested in hearing the perspective of folks on our topic today. So, thanks for joining us, everybody, too.

    And by the way, our DevOps Unbound is sponsored by Tricentis. I think you may have mentioned that, right, Alan?

    Alan Shimel: No, I think I didn’t, Mitch, but thanks for picking it up for me. [Laughter]

    Jayne Groll: [Laughter]

    Mitch Ashley: I was trying to give you credit where credit may not be due.

    Alan Shimel: Yeah, no, I don’t wanna take credit when I messed up. Yeah, DevOps Unbound is sponsored by Tricentis, and many thanks to them for their sponsorship. It makes all of this happen.

    So, let’s turn to the topic at hand today, guys. You know, when I first started devopsy.kinsta.cloud eight years ago, I met Jayne maybe a year, within a year of that or a year and a half of that. One of the interesting things about DevOps is, there was no manifesto. There was no hard definition of what is DevOps and what is not DevOps, you know, contrary.

    And ever since then, it’s been a double-edged sword. In some ways, it’s helped DevOps grow, it’s helped DevOps stay current over the last 8 to 10 years as the world has changed, DevOps has changed along with it. On the other hand, though, we’re still dealing with the, “So, what is DevOps?” Right? “What exactly is DevOps and what exactly is not DevOps?” which is sometimes harder than what is DevOps.

    And as we sit here now, August, 2021, there really still is not a quote-unquote standard for DevOps. There is no standard DevOps. It is what you want it to be, right? As I think Andrew Clay Shafer said, “The DevOps you get is the DevOps you deserve,” right? But have we outgrown that stage? You know, that seems like a terrible twos stage of DevOps to some; to others, it’s what kept it cool. Has DevOps matured? Do we need standardization? Will it be something that kills the goose that lays the golden eggs?

    We’re gonna discuss that today, and I’m interested in all of your views. Jayne, as the CEO of DevOps Institute, how many people have taken certified DevOps classes and certifications with DevOps Institute? It’s gotta be tens of thousands.

    Jayne Groll: Yeah, about 25,000 so far. So, it’s a pretty large community.

    Alan Shimel: Mm-hmm.

    Jayne Groll: And if you don’t mind, I wanna kind of just lay the stage, because I’m really interested in the standardization initiative. So, you know, I had a privilege of seeing, going to DevOps Days in 2012 in California. So, if you can imagine, it was in an old, dusty data warehouse, right, and it was at the very, very early stages. And so, one of the reasons DevOps Institute emerged was because there was a lot of really cool things here culturally, from an automation perspective, right, doing things faster, collaboration.

    And so, we started up DevOps Institute because we wanted to make sure the humans were represented. And I love that Paul and Sarah in particular both represented people. The problem with that is, at that point, it was very, very open, it was very kind of organic. You know, it was really a group of passionate people that were trying to just change the way IT operates, right, from a people, process, and technology perspective.

    What happened after that, though, is that there’s so much noise about DevOps that if I were an enterprise CIO right now, my head would explode, right? Now, is there one right way to do DevOps? No. Is it only gonna get you there with automation? No. You know, is there a very, very strong human element—absolutely.

    So, it’s a double-edged sword, Alan, as you said, is that we wanna be able to kind of scale down, right, some of the common language, some of the common definitions. I always say I’m married to a CPA, they have generally accepted accounting principles; maybe someday, we have generally accepted IT principles. But how do we distill the noise, right, and everyone having a different interpretation of how to do DevOps and what it is compared to, let’s get past the, “what is it?” We’ve spent, what, 10 years trying to figure out what it is, to, “how do we do this?” Right?

    Paul Bruce: Mm-hmm.

    Jayne Groll: And it isn’t just one roadway that takes you there. So, I think it’s a challenge. I mean, we introduced foundational certifications around DevSecOps and SRE and DevOps Foundation and DevOps Engineering, really, to create kind of a common vocabulary more than the, “here’s the single way to introduce CI/CD.”

    So, I don’t know if that helps kinda set the stage, but I think it’s a really big challenge right now that, if we standardize it, we run into the risk of what happened with ITIL where it couldn’t keep pace, right, and it became religious. I come from the ITIL space, right, so it’s like, open chapter 1, verse 3 and sing a hymn before you introduce change management to how do we then, on the other end of the spectrum, it’s so open and so undefined that it’s really complicated for legacy, I call them heritage organizations, to be able to move from this step to this step.

    Paul Bruce: Mm-hmm.

    Alan Shimel: The church of ITIL. Paul, what’s your feeling on that?

    Paul Bruce: Well, everybody’s on a journey, right? Your personal journey, your professional career, whatever, maybe you go do a walkabout on a particular topic to learn more about that. And so, we started this venture, there was an early round of, “Who wants to be involved in something like this?” in 2016. I answered a call from Bob Aiello and Lynn Carter, who were kind of forming this idea of, there’s a bunch of people who really care deeply who have practitioner practical experience in the field. And the early part of this process was just learning, kinda level setting on what we have done, right?

    Jayne mentioned that it’s hard, maybe, to have accepted IT standards, even though there are standards, right, that not every organization applies the same way. I don’t think that invalidates some of the general principles that we know to be true, which is, you know, you ship broken software, you hurt people, you hurt yourself, right? But those are high level. How do we break this apart and start to make sense of the world, as you said, the heritage, Jayne, you said the heritage organizations—what is so wrong, what’s going on that we have these kind of security problems, these performance problems, this inability to be competitive in terms of acceleration and stuff.

    And so, I think the purpose of a standard is not necessarily to force people to do something a certain way. I think it’s simply just kind of like, if you’ve ever had the Christmas ornaments or just ornaments box that has the insert that allows for things to be put in careful places and also work together as a unit, right, to provide a normative, not a prescriptive standard, but a normative standard to say, “Look, these processes that you already know? Well, you already followed these from, let’s say, 12207, right?” These processes around supply acquisition, knowledge management, human resources? Right, human resourcing in DevOps is a big problem, at least in the Boston area. Try to hire an SRE—there’s only about 15,000 to 20,000 in the United States, legitimately. There’s a reason for that demand.

    So, how do we think about the processes we already know and start to say, how are these things different because there are some accepted principles and practices around DevOps? Now, the practices are not things that you would ever wanna dictate in a standard. This is not the same granularity level as some other standards that dictate how to encode time formats. That’s very important at that level. But the level that we were working on was how to level set across the things that, especially organizations in high compliance and high risk industries, right, risk-averse industries—how do they take the practices and processes that they know from a process standard and start to go, “I understand how to plug, now, what the DevOps principles are into the standard to encourage both of those practices in a more predictable manner?”

    So, when we did all this work—it’s taken years, right, of a lot of learning collaboration between a whole host of people I don’t have even time to list out, but a lot of people from a lot of various different aspects of software have a lot of experience in those aspects. You know, it took synthesis and refactoring, and I was up nights and weekends just rewriting some of the statements, you know? And each of these, it’s a line to 12207, so there’s multiple processes, process declarations like quality management, quality assurance, verification and validation.

    Like I said, these project enabling processes and then these technical processes, and each of those process sections have outcomes, activities, and tasks. Things that are likely to occur in this area, right, as well as, regardless of how you do this, right, not dictating what the DevOps looks like, it’s outcomes of these things that are auditable, that not compliance but conformance. That you can confirm at various levels to these outcomes to say, if it’s important to get this right, when you hit the piñata, what comes out should be candy, not garbage, right?

    So, I’ll pause there, because I wanna give Sarah a chance to kinda provide perspective, because she’s also been working a lot with us, for all those years. [Laughter]

    Sarah Baker: I think I counted 27 people on the final author list for the IEEE.

    Paul Bruce: Yeah.

    Sarah Baker: And it was many, many weeks of arguing, reading together the exact language. And I agree very much with some parts of what Paul says, and I wanna step back and talk a little bit about how I see, you know, what, to me, was the big shifts. And so, the way we wrote the doc was to kinda look at the 12207 and say, “How does DevOps change it? How has it changed the game? What are the perspectives that you need to have in mind in order to get the outcomes that Paul’s talking about, there?”

    And so, when we were writing it, we were writing it to that. How is this different, and to what extent do you wanna consider these areas? It’s a framework of thinking. At its largest level, it’s a framework of, for me, the value chain of delivery of software. And it’s not—and it uses many of the concepts that come out of learnings of industrial engineering, way back in the ‘40s, ‘50s, ‘60s. But it had never really come into a lot of the thinking of software, because software is different. And the more I was working and kind of digging into what are the concepts underneath DevOps, I came quite aligned with a lot of industrial engineering concepts about how things flow and how things move back and what the peaceful aspects are in that.

    And so, that—actually, if you read through the standard, you’ll see a lot of those flow pieces and all of the pieces that you then say, “Oh, it’s the feedback loops, yeah, lots of feedback. Collaboration, it’s all there.” And it’s just applying a lot of those learnings into software engineering, to me. And it was stuff I was like, “Oh, it’s so wonderful to have this.”

    And the other thing, one of the big learnings that I think I brought into the conversation is how—and Paul will back me up on this—how security for a lot of us changed slightly as we started walking our way through the 2675 work. The terms of risk became a very interesting concept that flowed through the document more than I would have expected.

    Paul Bruce: Mm-hmm.

    Sarah Baker: And so, there was some learning within the totality of our team as we wrote this. It was quite fascinating.

    Mitch Ashley: I’m really fascinated to hear from both of you that have been working on this, because it’s not something I’ve been involved in. I sorta have a mixed history with standards, because I’ve worked in areas like networking and things like that where standards are very beneficial or really fundamental, but I’ve also learned standards are kind of what you’re gonna do, not how you’re gonna do it. Because we ran a test lab that, even though it says we’re gonna do least cost routing in a network device this way, people implement it differently and they don’t always work together. So, it’s not a nirvana, but it’s also not a pariah, either.

    Paul Bruce: Mm-hmm.

    Mitch Ashley: So, what I wrestle with here is that, I think some of the thinking—and Jayne can speak to this, but innovation happens so quickly now. So many things are created without standards; maybe they evolve into standards, maybe they don’t. And it doesn’t bother me that there’s lots of noise around DevOps, because we’ve all seen the cycles, the hype cycles, right? If it’s worthwhile, everyone and their brother, sister, cousin, uncle will all jump on the bandwidth and call what they’re doing fill in the blank—it’s AI, it’s DevOps, it’s whatever. That’s just part of our industry. And it’ll either fizzle—you know, value stream management, is it a real thing? We’ll see. It’ll fizzle or it’ll happen, right, if it’s a real thing. And DevOps has proven itself not only is it a real thing, but other parts of the organization now see an opportunity to play, to participate, to become part of the software creation process.

    Paul Bruce: Mm-hmm.

    Mitch Ashley: And I think that’s what’s so powerful about DevOps—it’s not about a technology, it’s about how we create software. And if you were gonna start out, you know, Jayne, when you went to that meeting in 2012 when you said, “We’re gonna go create a standard for completely doing software differently because we have cloud and we have the ability to automate and we were gonna change the software architecture,” you couldn’t have known how to do that. But being the industry would’ve, you know, the antibodies would’ve crawled all over all of us and stopped it. So, I think there’s a lot of power in the innovative, organic nature of it.

    And I don’t want DevOps to turn into ITIL, [Laughter] because that’s sort of taken on a life of its own, and now it’s trying to catch up and it’s done some good things, too. So, I wrestle with trying to constrain it when I think it’s real power is the innovation behind what we’re doing.

    Paul Bruce: Yeah, so, before I—

    Sarah Baker: If I could take a moment—

    Paul Bruce: – yeah, go ahead.

    Sarah Baker: – I’d like to take a moment and call out something. Part of the problem with ITIL is like, it’s ITIL. And, you know, a gazillion managers said, “ITIL, let us get a hammer and hit you over the head with it.” And DevOps, fortunately doesn’t have that problem. It has, you know, a little fuzzier in its definition, but the great thing about DevOps is that it has that sort of fuzzy edges. You can’t, it’s really hard to beat someone over the head with it. You can try. I’ve seen it happen, you know? Everybody’s a DevOps engineer now, but it’s not necessary.

    And what people talk about is not that, but the work, you know? It’s not, “We’re conforming with ITIL,” it’s like, “We’re doing DevOps.” The doing part is the important part, not the compliance part. So, when you say, when you hear people talk about it, they talk about it differently, and that’s the good thing.

    I use ITIL, I just don’t tell anybody the word.

    Jayne Groll: Which is a shame. It’s such a shame because, you know, I come from the ITIL space, and in the early days, this was not what the intent was to be. It was very much like DevOps in terms of, “Let’s help IT grow up,” right, in terms of some things.

    And, you know, to what you’re all saying, you’re absolutely right. A framework, whether it’s a standard, is a guideline, right? It helps you with the journey, as Paul said. And so, if I wanna kinda check, check, check—you know, I’ve worked with some of the ISO standards, right? So, in the ITSM space, there was ISO 20,000, 27,000 for security, right? And it basically, like, in the one or service management says, “You should record all your changes.” It doesn’t say how to record it, it doesn’t—it’s a really great practice, you should do this.

    The problem, sometimes, with standards—and I’m sure you know this and it’s a risk—the pure intention is pure. The ability to weaponize it, right, from the auditors and from others is a real risk, right? Because we’ve seen that with other standards as well, where it becomes more about the compliance than it is about how having it being a guiding light. So, I think there’s always a risk there.

    And then the other risk is, you know, think about observability, right? So, observability has been trending, what, for a year, maybe two, right? Four—okay, but really mainstreaming, like, talking about observability as kind of a mainstream practice, right? And I will share with you, Mitch, to your point, when we launched DevOps Institute in 2015, we were not welcomed with open arms, right? We were vilified for trying to put some boundaries around this.

    Now, you know, move forward seven years and I think we’ve overcome a lot of those and we have some that were detractors or really great supporters. But I think they were concerned that it would become weaponized or commoditized or whatever.

    So, I’m just gonna say one last thing because I think it’s important. Commoditization of knowledge, which, this is kind of in support of the standard, right? Commoditization of knowledge where we make sure it’s available to everyone, some people see that as a negative—it’s not. What would happen if the whole world, right—every IT organization, every IT professional decided to take this standard and use it as a guiding light, according to the needs of their organization? They weren’t worried about the audit, they got trained, they shared knowledge—wouldn’t that be amazing? But that’s commoditization.

    So, I think the risk of good versus evil on initiatives like this is real, but I also think that distilling it to everybody, kind of like accountants have done, right, is important because IT needs to grow up.

    Paul Bruce: Yeah, well—

    Mitch Ashley: Well, and if you look at the accounting standards, they don’t tell you how to do it, they tell you what to do.

    Jayne Groll: Right.

    Mitch Ashley: It isn’t prescriptive in the how very much, so it’s very much like an IEEE standard or whatever.

    Paul Bruce: Yep. Well, first off, I forgot to mention when we first started—I speak as an individual contributor. If it isn’t abundantly clear, [Laughter] I don’t speak for the IEEE, and I don’t speak for the other people who have spent years contributing to this meaningfully.

    So, when I say certification processes inherit all the exact same things you just mentioned as problems—weaponizing, actually low ROI maybe in some cases—we see these deficiencies and these outcomes that aren’t so great. And if you’re somebody who’s just gonna sit back and say, “I’m not willing to hold myself accountable” to stand up and say, “What does this thing actually mean like a real engineer should, you know, to start to break apart the problem and see what are the components, how does this thing work and how do I want it to work? How does it work best for the people around me as well?” If you’re not willing to do that, then I would call you the nay-sayer, you know what I mean? Like, not any one in particular, but that would be a nay-saying, like, let’s avoid accountability, let’s avoid doing what we know needs to be improved?

    Continuous improvement is a huge element that is all throughout the entire standard. It’s one of the things that’s identified as some of the core principles in an ISO study that was done before we started in on this standard to see what are the common themes of people, when they’re really feeling like there is this DevOps mindset, this culture, these practices—what do they mean by it?

    And a number of things came out. One was a high degree of automation, right? Okay, well, that’s very tactical. No matter how you do it, right—high degree of automation. Duh, right? How about effective communication and collaboration? How about shifting things and continuous everything? The notion of, that these things are not this one time and then three years later, you get to come back. No, no, no. If it’s a good idea now, maybe don’t bite off a three-year apple. Maybe take a bite, but know that you started in, there’s a continuous improvement process.

    I can say, we mentioned earlier about a holy war that might exist when you start getting into what is DevOps and what is not DevOps. Like, I started—the best way I can describe to people, because like I say, half of my day is spent on and off of Slack with people who are actually doing, working this out in a highly statistical, large distribution across the country, you know, and sometimes outside the U.S., too, just time zone supply.

    But I can say, for sure, after seeing the complete garbage Dumpster fire where nobody is willing to stand up and say, “Hey, aren’t these things common and standard?” And oh, by the way, not vendors that have something to push, right? I mean, that’s, I think, Jayne, where you were talking about maybe observability is now a thing. You know, the only reason why it’s a thing for the most part is because it’s in our face now because all the vendors are hash tagging the crap out of it.

    Same thing happened with DevOps five, six years ago—everybody had to get on the bandwagon. I remember CEOs way back then going, “What is this DevOps thing?” Oh, man. You know, just talk to some of your engineers.

    And what I can say, right, I have developed at least an internal personal perspective on what is not DevOps. So, if I can borrow the last 30 seconds of the mic for a second, here are some things I wrote as notes as I’m going back over the standard, re-reading it, going what I violently agree with from both others and myself, we did not have a chance to see this before it went live many times, right?

    So, what I got from one of the higher order sections in Section 5 was this feeling of, like, what is not DevOps. And those things are burnout. DevOps is not burnout. You know, you’ve probably been burnt out if you’ve worked in DevOps before, so where is this coming from? Why is this the case? It’s not burnout, it’s not no help. You don’t get any help, go do this thing, you know? It’s not heroics. It’s not superheroes in capes, right? Single points of failure, as I call them. It’s not cliques—which, by cliques, I mean power silos. Huh-uh, there is no such thing as The DevOps Department, people. There’s no rogue decisions. There’s traceability in decisions. There’s accountability in decisions.

    Leadership thrash—not easy to, if you’re really adopting this DevOps mindset and way of thinking about how to do things better, leadership thrash doesn’t really fit in. Toil and debt, right? One of my biggest things is, you know, I seek to eradicate toil, which is not easy. [Laughter] Just reducing it a little is nice, but it’s, and it’s probably pragmatic to say so, but really eradicating toil from other people’s plates and from your own, that’s a challenge, right? That’s kind of a life goal. And if we do this stuff together, right, if we say these are not the things we want, the question is how do we move towards something that isn’t those things, right, and that is closer to where we all align and agree, like, this is a better way of delivering software and doing it faster, right?

    So, anyway, I’ll pause there.

    Mitch Ashley: You know, one of the things I’d love to hear from both Sarah and Paul is, there’s always sort of that, “So what?” answer—question, right? “If we did this, what is the result? What’s the impact of it?” And I mean that in a positive way, I’m not being critical.

    Paul Bruce: Yep.

    Mitch Ashley: If this standard was widely adopted, how would it improve? How would it—what things are we wanting to improve and change from where we are today? Can you describe that in any way to kinda give myself and others maybe an idea about that?

    Paul Bruce: I just did. People already implemented it.

    Sarah Baker: Well, at the base, there’s the vocabulary, yeah.

    Paul Bruce: Sorry, people are—

    Sarah Baker: At the base, that is part of it. Sorry. Paul and I always get [Cross talk].

    Alan Shimel: Sarah, go ahead. Paul spoke—you go.

    Paul Bruce: Yeah, yeah.

    Sarah Baker: Vocabulary is that first piece, making sure we’re all talking about the same framework. And so that, when we’re exchanging ideas, how did you do it, how did you do it—you know, not any one way to do it is gonna it, but knowing what your options are and talking about it with others and trying something out and figuring out if it works or not in your environment is one way to get everybody on the same playing field. But if you’re not talking about the same thing, you can’t do that.

    So, it’s having the ability for a large corpus of options to apply to your situation. To me, that’s one of the biggest things that it provides, and it also gives you ways of interacting with the rest of the business that is not IT, and kind of, “Oh, this isn’t—it’s not just IT. You’re a stakeholder, you’re a customer.” And it includes all of those pieces for the collaboration and really helps support it.

    So, to me, that’s what the standard brings. It widens the organization’s perspective a lot, and allows that cooperation to seep into the organization across all of these different pieces. Because to me, that’s how it started as DevOps, DevSecOps, DevOps Observation Ops. I mean, it grows because that’s part of its DNA is to include.

    Paul Bruce: Yeah.

    Sarah Baker: And so, it’s very natural that it has done that and it includes other parts of the organization into that basic thought processes. I totally could see DevOps marketing security—I mean, you see all that. That’s what’s, to me, the real advantage of thinking this way is that it includes the entire org.

    Jayne Groll: Well, and first of all, utmost respect to Paul, Sarah, and the other 25 people that are working on this, because this is a gargantuan task, right? Trying to encapsulate something that creates a common framework, that has the purest intention and also, you know, if you think about the heart of why DevOps emerged in the first place, it emerged because we were not collaborating together, right?

    So, you know, in the early days of IT—I always say I’ve been in IT longer than I really want to admit to—you know, there were 25 people that sat on the same floor, in the same office building, and we all knew each other and we went to happy hour. And then, you know, the tree grew and there were lots of branches and other things. So, you know, giving a central source of truth that everybody can kind of internalize, process, speak the same language—you know, IT, we have so many different languages, we’re like countries that are butt up right next to each other.

    And I said, I have so much respect, because this is a gargantuan task. And Sarah, I agree with what you said, the purpose is really to take this need for collaboration and optimizing automation, right, so making automation essentially members of your team, right, and then taking that out into the business because every business is a technology organization, it’s—I’m sure you’re aware of the risks. You know, we’re not telling you anything that you’re not aware of, but as I said, I just have so much respect. Because it’s not gonna be an easy road, right? There are so many different opinions from vendors and from everyone who’s involved in this space that are going to have an opinion or whatever. And I think, you know, that’s something that everybody’s gonna have to deal with.

    Again, the only thing I would suggest is build it in a way that it’s nimble—dare I use the word agile, so that—

    Paul Bruce: So, this is already done, right?

    Jayne Groll: Yeah.

    Paul Bruce: Like, this is not a draft, this is not an initiative. This has already been done for the past four years and is now an official standard. So, the good news is, to your point, I think, about suggestions about how to do this, we are working on the second version of this draft already.

    As soon as it was out, even before it was out, we knew there were things that we just couldn’t—we couldn’t spend the time properly addressing. Like I said before, there was a lot of refactoring, but there was a lot of careful avoidance of not already repeating what was already in a good security standard over here, or in a good risk management thing, or one of the folks that worked with us, Tafline Ramos, works at the ISO, right? And what is it, 29119, right, verification and validation, the processes—that was already stuff that’s already been, and great thinking and not slow, you know? Can fully be put into a modern, unicorn sneeze rainbows and fart sprinkles type approach as well as the large Fortune 100.

    So, just to say, this is already—we’ve already put a ton of, it’s already out there, and the easiest way is, I mean, some people might be wondering, “Oh, yeah, but it’s a standard, so now I gotta pay for it?” Some standards, you do; some standards, you don’t. This one, IEEE, they cover their costs with the standards. The easiest way, if you wanted to get a free copy, become part of the new, the next version of this, right? See what we’ve done, poke holes in it, have collaboration, let’s work on it together, because that’s the point of this is to bring a framework of how we are working together in a more effective manner.

    Jayne, I’m really sorry I cut you off.

    Jayne Groll: No, no, you didn’t cut me off. Just a cautionary tale. One of the reasons ITIL kind of lost a lot of its momentum—and I’ll be very frank, I owned an ITIL training company for 15 years and I was a chief evangelist, but it took 8 years between version 3 and version 4. And so, it wasn’t built, it was a monolith.

    Paul Bruce: Mm-hmm.

    Jayne Groll: And I think that was—you know, in the meantime, lots of other things happened, and I think there’s a cautionary tale. I’m not suggesting you’re looking at eight years, please don’t misunderstand, but this market is so—

    Paul Bruce: No, it’s probably about three, right?

    Jayne Groll: Yeah, this market’s pretty fast. Sorry.

    Paul Bruce: Our target is about three, between where we officially started and now it’s out and the next revision, at most, would be another three, right, that’s our target. It could be earlier or later than that.

    Think about, we’re all citizens of the United States here, right? Or anybody in Canada?

    Alan Shimel: Is that a HIPAA violation to answer?

    Paul Bruce: Yeah, right.

    Alan Shimel: Just kidding, just kidding, just kidding.

    Jayne Groll: [Laughter]

    Mitch Ashley: I’m not showing you my vaccine card.

    Paul Bruce: I ask because, I ask because—

    Alan Shimel: I didn’t wanna go there, but go ahead.

    Paul Bruce: – we all know what the U.S. Constitution is and where it came from.

    Alan Shimel: Mm-hmm.

    Paul Bruce: Do we think that it was perfect and never has to be revised? Absolutely not. That’s what the Amendments are for. And arguably, I people wanna go back to the original version, there are some serious problems with that original version. Same thing with ITIL, same thing—remember with, I think you might have heard about The Agile Manifesto. You know, that was a drunken party with a bunch of white dudes, okay?

    Jayne Groll: I know. [Laughter]

    Paul Bruce: So, yeah, there were some good ideas that were encoded in there, but do they not have to be revised and worked out? Even Dave Thomas says agile is dead, long live agile. Like, what people have taken it to is not the point.

    So, the goal of this is to really get a full perspective. To what Sarah was saying, it’s not just—I’ve talked with a lot of the DevOps Days, various different people from DevOps Days, from a number of different, like, The DevOps Enterprise Summit and stuff, and there are some people out there that are like, “No, DevOps is just about developers and operations and it’s about just these things, and if you start getting biz in there—no, no.” And it’s like, what a narrow perspective, because you know what, you had a good idea and then you screwed it up by not including other people, by not realizing there are meta learnings that come out of what you locally did—maybe, arguably, you did well—that apply broadly. And when they do apply broadly, they bring us back to the same premise, which is, we’re all in this together, right? We all win when we all win. As Tim O’Reilly might say, provide more value than you consume, right?

    So, that’s where we intend on taking this. It’s not to turn into a bunch of agile consultants again or ITIL consultants, right?

    Mitch Ashley: [Laughter]

    Jayne Groll: Framework growers, that’s what we call it.

    Alan Shimel: Yep. Guys, I gotta pull the plug on ya, I’m sorry, guys, and ladies. We are out of time. I think a great way to end this, though, is—Sarah, Paul, where can people get more information on the standard from the IEEE?

    Paul Bruce: Sure. Google it, Google IEEE 2675, you’ll get to that page if you wanna buy a copy.

    Alan Shimel: 2675.

    Paul Bruce: I think without being a member, it’s like 99 bucks—so, look, you know, if you wanna buy it for yourself—

    Alan Shimel: It’s not breaking the bank.

    Paul Bruce: You probably also have, in your organization already somebody’s a member, your Risk and Compliance are probably already members to IEEE.

    Sarah Baker: Yeah.

    Paul Bruce: And again, like I said, the best way to get a free copy is to come work with us on what does the next standard mean. It’s not a huge effort and commitment like Sarah and I have put in. [Laughter]

    Sarah Baker: [Laughter]

    Paul Bruce: You can dip your toes in, see what makes sense, what works for you, and we love to—we include people in that effort.

    Alan Shimel: Great. Well, Paul, Sarah, Jayne—thanks for joining us on our panel today. Mitchell, as usual, great job. This is another edition of DevOps Unbound in the books, check it out. Go check out the IEEE standard. We’ll be back in two weeks with a fresh show and a fresh topic. Until then, this is Alan Shimel, be strong, be well—Tech Strong. Take care, everyone. Bye bye.

    [End of Audio]

  • DevOps Unbound EP 19 – How AI and ML are being used in DevOps Today – TechStrong TV

    DevOps Unbound EP 19 – How AI and ML are being used in DevOps Today – TechStrong TV

    Alan Shimel: Hey, everyone. I’m Alan Shimel, CEO of MediaOps, devopsy.kinsta.cloud, Container Journal, Security Boulevard, and you’re watching DevOps Unbound. DevOps Unbound is sponsored by our friends at Tricentis. So many thanks to them. And DevOps Unbound is a biweekly show where we cover topics of interest to the DevOps audience. I am the host. My co-host is our CTO and CEO of accelerated strategies group, my friend Mitchell Ashley. Mitchell, welcome.

    Mitchell Ashley: Thank you. Good to be here as always, and with this illustrious panel.

    Alan Shimel: Absolutely. And an illustrious panel, it is. We have two panel members joining Mitchell and I today, let me introduce you to both of them. They’re both great folks in their own and I’m gonna let them introduce themselves. Let’s start with our friend Judith Horowitz. Judith, welcome to DevOps Unbound.

    Judith Hurwitz: Thank you so much, Alan. It’s a pleasure and honor to be here.

    Alan Shimel: It’sour pleasure.

    Judith Hurwitz: So I’m Judith Hurwitz. I’m the co-author of 10 books. Have been in the industry for 30-plus years. Focus on everything from DevOps, security, manageability in cloud, hybrid cloud, and really looking at how you take technology to transform organizations. It’s a complex topic, not easy, but it’s what we’re in the middle of.

    Alan Shimel: Absolutely. Then last, but certainly not least, my friend Brian Dawson. Hey Brian, welcome.

    Brian Dawson: Hey, Alan. Thank you. Good to be on with you again. To tell the audience a bit about myself. As I’ve told you before, Alan, I’ve been in software development and delivery for about 30 years. I consider myself a technologist and even prior to focusing on DevOps, I’ve had a focus on optimizing software development and delivery.

    Excited to talk about AI. I kind of dabbled and dipped my toes in the space and my time that I spent at the company that is now PlayStation. Things have come a long way since, and during that time, I’ve spent the past 10 years of my career focused on identifying, applying and spreading DevOps practices. So I’m excited to discuss the two together here with this group.

    Alan Shimel: Absolutely.

    Participant: Brian, I’m not sure if you mentioned your present position with the Linux Foundation.

    Brian Dawson: I did not actually, because also this organization I’m at today has a big footprint in the AI and ML space. So today I’m with the Linux Foundation, I oversee our developer relations in ecosystem development. And of note within that is the LFAI & Data Foundation which houses a number of impactful products in this space projects. Excuse me.

    Participant: Thank you. I just thought the folks at LF would welcome that.

    [Crosstalk].

    Brian Dawson: Well, appreciate it. Thank you.

    Participant: Yep

    Alan Shimel: All right. So the topic for today’s DevOps Unbound is artificial intelligence and machine learning, how can they help improve performance of DevOps teams? In other words, what’s their role in the DevOps world? And let me preface our conversation by giving you sort of my view of it.

    DevOps for many people was all about automation. What can we automate? Let’s automate everything we can, automate, automate, automate. And obviously in discussions around automation, topics such as artificial intelligence and machine learning, you would tend to think help with automation. And therefore anything that helps with automation is good for DevOps.

    And I should also say that we lump AI and ML together, like their siamese twins joined at the hip, and they’re not necessarily conjoined. You can have AI without ML and you can have ML without AI, at least I think so. I’d be interested in your thoughts.

    But has it lived up to the hype? Will it ever live up to the hype? Was the hype unfounded? Was it unrealistic? What role has AI and/or ML to this point played on DevOps and what role will it play going forward? I think that’s our topic today. I mean, any one of you, if you want to kick it off with your own feelings or respond, go right ahead.

    Brian Dawson: Well, I’d like to jump in as a troublemaker early on and say –

    Alan Shimel: Okay.

    Brian Dawson: – I challenge Alan that DevOps is really about automation is kind of the basis for the tie. If we look at it, DevOps is really a set of cultural practices and tenants that align DevOps and other software delivery stakeholders around the shared objective of delivering quality software reliably, rapidly, and repeatedly.

    Now the reliably, rapidly and repeatedly component, absolutely lends itself to some of the benefits that we could mine from AI. But I actually think it’s a great opportunity if we don’t say DevOps equals automation equals AI, but rather we say DevOps equals a culture aligned around a shared objective and how can AI and ML help support those shared objectives.

    Judith Hurwitz: So I think you make a great point. Brian, I think one of the most important issues, and you mentioned culture. We have been, for what, 50 years through this development and operations perspective on how we get things done. And the promise of AI is, “Okay, can push a button and it will take care of everything.” And, and this promise has been there for, what, 20 years.

    The reality is, it’s just not that simple. There are definitely things that we are doing today, and we are seeing evolve that is definitely helping the developers and the operational professionals. For example, if you have repeatable functions that happen all the time, that for example, if you press a button and a certain task should happen, you can probably use a model and collecting massive amounts of data to do that automatically.

    nd that’s a very good use of AI, and will work today. They call it ML Ops or AI Ops, and there’s a good reason why that’s what we’re focused on today, is because you’re look for predictable patterns and predictable anomalies so you can avoid making stupid mistakes that, “Oh my God, why did I do that? I knew what the right answer was.”

    But AI is not a panacea. And I think if we look back even two, three years ago, you had companies that all were saying, “Okay, we have automated, we’ve put AI into DevOps and all you have to do is press button and all of your problems are over.” That’s just not reality.

    Alan Shimel: Yeah.

    Mitchell Ashley: Yeah, there’s the marketing AI and ML, and then there’s real AI and ML. Marketing, meaning everybody glams onto the term and uses it, meaning it’s a case statement or an if statement in people’s logic in their software. I started doing some work in AI in the 80s and 90s programming and Lisp prologue, and doing some corporate education in that area, some expert systems with triage, things like that.

    So dabbled in a little bit there. And then I was exposed to more recently, about five, six years ago, worked with a gentleman named Dr. Bernardo Huberman, who’s one of the experts in the field. And I asked him a me understand why is machine learning taken off as this sort of, he described it as a subset of AI. So you can know that was his model anyway.

    What he told me really made sense, and it kind of helps me understand where can we apply machine learning? And that being kind of the most popular part of AI. And he said, “Machine learning takes massive amounts of data.” And the fact that we have the cloud and all these applications, creating data, a lot of data exhaust. Now, people are going back and mining that data.

    So we have this whole, Judith, in your writing, in your books about data analysis and leading up to AI, machine learning is great for that, because you can have supervised or unsupervised algorithm.Supervised, meaning that’s a cat, that’s a cat, that’s not a cat, or unsupervised, which is gonna pour through the data and start to look for those patterns and trends and anomalies that you were talking about Judith.

    So when people talk about using AI or ML in our industry, I always think about, well, where’s there lots of data, and can we leverage it in some impactful way? So as I look at products or technologies that claim they’re doing that, that’s at least one sort of criteria to fair it out. Is it real there? Is there something real there, or is it more spin and fluff to help us? We all have case statements in our spot, right?

    Alan Shimel: Absolutely. Actually, Brian, it looked like you were gonna say something, I didn’t wanna jump on you.

    Brian Dawson: No, Alan. I mean, of course I have a lot to say, but nothing in particular, I just, since I’m off mute, I’ll say I absolutely agree with Mitch and Judith and underscore the points that they’ve made.

    Alan Shimel: Yep. You know what, I wanna take a moment though, and explore, Brian, you disagree DevOps isn’t about automation. And I get the whole cultural aspects of DevOps and all of that, but certainly automating as a way to be more efficient, to get more done faster, I think is part and parcel of the DevOps mindset.

    Brian Dawson: Yeah. And if I may and without waiting to see where you wanted to go with that, I’d say, no, it, it is absolutely important, but there’s a line Jez Humble would use in talking about CD. And frankly, I think the book that Dave Farley, Jez Humble and team, Road Around Continuous Delivery is undercelebrated and under referenced.

    And in talking about that, one of the things he said is continuous delivery. And I’m gonna say by extension, some aspects of DevOps don’t require any tools. I can do continuous delivery with a bash script.Catch is it’s not the most efficient in pursuit of effective collaboration of delivery velocity so you can iterate. And as we’ve talked about in this past, establish that control feedback loop that requires things like automation to achieve those goals.

    I just think sometimes when we talk about what we see going on with AI and ML, and in those terms, we overload a principle or practice with expectations that are core to it. And I worry a bit about that, so I wanna call it out. And sorry, Alan, I don’t know if that’s where you wanted to go with the commentary.

    Alan Shimel: No, no, no. I wanna go wherever you wanna go, Brian. It’s a collective. Judith, what about – I’m sorry, go ahead. Oh

    Judith Hurwitz: Oh, yeah. So,Alan, I think one of the key issues is how the whole area of continuous delivery. In previous generations, you would build an application and it would live pretty much as it was written, could be for 10 years. Today, applications are constantly having to be revised because customers change, partners change, the sources of data change.

    So, unless you have the ability to constantly update and change and modify, you lose out. One of the values of using automation and using AI and machine learning models with lots of data, is to support the ability to do this. You don’t have enough hands and enough brains to where problems may occur, because you’ve changed things.

    How many times have we seen problems occur because somebody has added a new service to their environment and somebody forgot to change a configuration file? Now that’s not something that you need a massive brain to be able to do, but people get busy and they forget to do simple things.

    Mitchell Ashley: I think, Judith, that’s really a good point, ’cause I wanted to ask you about this in the software creation process. Given things are so dynamic now and constantly changing, understanding just the environment, the infrastructure’s code all the way up through the application and how much all of that is changing, it seems like the stepping in for humans in certain conditions in that environment is a great application of AI.

    For someone to synthesize all the factors that might go into where a problem exists or what might be causing a problem, seems to be where some of those algorithms might be helpful. Do you agree with that? Has that been your experience of what people are thinking about for AI?

    Judith Hurwitz: Yeah, definitely. I think what we’re seeing is this is where sort of the human factor comes in. You set it up so that if the printers turned off, don’t send me an alarm fast emergency. There’s a problem with the printer. We all have gotten used to that. But when there are problems that you you’ve never seen before that you don’t have data on and the model doesn’t take into account, then the AI Ops or ML Ops then gives you a message, “There’s something strange going on. I don’t know how to fix it. What do you want do here?”

    And so over time, when that appears again, well now you had data, well, this occurred once before, maybe it was anomaly twice. But then when you have enough data and enough experience over time, then you build that into the model so the next time that occurs, you make a fix.

    On the other hand, you don’t want the system to say, “Oh, I know what that is, make a fix,” when it turns out, no, no, no, you don’t understand the context of, of what’s going on here. And just because there’s a correlation, doesn’t mean that there’s a cause for that.

    Mitchell Ashley: It’s good to know, ’cause it’s not always about automating a response to it. I know in the security world, we’re very cautious, we’re skeptical about those things happening, blocking, legitimate traffic and financial world. That’s a huge issue that we run into.

    So you pointed out a good scenario, whereas we see repetitive patterns over time and that can help machine learning algorithms now understand, “Okay, that’s what that pattern looks like,” so you can identify what it is next time instead of it’s just an anomaly.

    Brian Dawson: Yeah. It’s interesting Mitch that you say that, and going back to the original, kind of the starting topic about AI and ML, not really achieving, its its promise yet. And as you called out, getting data is an issue, it’s not necessarily applicable to every space. You need a level of determinism and predictable patterns that you can learn from and build on.

    And I have for a long time been excited that when we look at CI and CD, when we look at, Alan, the automated workflow of software development and delivery, or even if it’s not automated kind of the standard, there’s a couple of things that you do have. You’re doing builds, sometimes thousands of times a day, builds in deployment, getting that out into pro. You could be generating a ton of data.

    It’s also in its nature, you’re striving to achieve a level of consistency and repeatability in how you deliver software. And I’ll say I’m not an expert to the level. Some of the people on this episode are, but I get at a very sort of top level get really excited about the opportunity for AI and ML to help better CI and CD, and in line with the principles of DevOps.

    What I see it really helping with is reducing cognitive loads so developers can focus on coding, innovating and solving problems while helping ensure that quality and stability that’s difficult to maintain while you’re moving fast. So just thought I’d called out.

    And I am curious to see when vendors and others really start to do like what our friend Koski has done with Launchable, and really apply AI and ML to the left hand side or dev side of the process to reduce the load, insure quality and stability

    Alan Shimel: Guys, we’re all lumping AI and ML, AI and ML, AI, and ML. Does it have to be AI and ML? Isn’t it, in my experience, a lot of what we call AI and ML is a lot more on the ML than it is on the AI? So, is it fair to call it truly AI or is it really ML? And maybe ML is for today and AI is it tomorrow. I don’t know.

    Judith Hurwitz: So I think there’s definitely a problem of nomenclature here. What we’re really dealing with primarily now is models and modeling data and creating models from data. That’s the reality of today. I think that there is a lot of misuse of the term AI and there have been some absolutely wild predictions. I can’t remember the name of the computer scientist who predicted that he would be able to replicate the human brain with AI.

    Mitchell Ashley: Marvin Minsky treat.

    Judith Hurwitz: No, this was past Minsky.

    Mitchell Ashley: Oh, was it after? Okay.

    Judith Hurwitz: Yeah, no, I mean, this was in the last five years.

    Mitchell Ashley: Oh, okay.

    Judith Hurwitz: So, over time, you’re always gonna be dealing with models. You’re never gonna get rid of the models. AI is sort of the next – Models are a subset of what you eventually achieve with AI, but AI is really a concept that will probably take decades to evolve. And I think one of the problems we have right now is where you have vendors, and I’ve talked to hundreds of them over the years where they say, “We have an AI application.”

    It’s because it’s a hot buzzword and you can look back and in the history over the last 30 to 40 years and see whatever the hot topic is, all of the vendors say we do that. So I think that that’s one of the problems that businesses are facing right now. What is the difference? What does it really do? How does it make your company better? How do you use this technology to be prepared for change?

    Mitchell Ashley: I think that’s a good way to break it down too, because early days of AI were about emulating human thinking. We’re still long from – I’m not sure – All my intelligence is artificial, by the way, I acquired all of it. I don’t think I was born with any of it. But that was kinda where it started. And then AI became about expert systems largely, and then machine algorithms.

    Machine learning algorithms was really where I think most of the activity is today because of that prevalence of data. It seems like most organizations are faced with sort of one of three strategies, what do we look from our vendors and how they may use AI in a meaningful way, or machine learning in a meaningful way that’s gonna help my business.

    Do I build models myself from the data like you were talking about, Judith, to apply to like a financial analysis kind of situation, more of an expert system, or do I use machine learning algorithms in my own software to do interesting, valuable things that my software can do.

    And maybe you play in all three of those or a subset of those, but that seems to be the question I think most individuals or organizations are at. We’re not at a place where we can go build models, but we’re looking for these capabilities either in our own code or in third party products.

    Judith Hurwitz: So I heard an interesting story from one of my clients a few years ago, maybe five years ago. He had a client that was very gungho about AI. So he went out and he hired five data scientists, paid them a million dollars each, gave them their own space and left them alone. These are the smartest people on the planet. Came back in six months. Didn’t wanna bother them. They’re so smart.

    “All right. What have you found out? Have you written this application?” And they said, “Well, we’ve been discussing this for six months and we have determined what algorithm we’re going to use. The point is, that they worked in isolation. They thought they were the smartest people on the planet. They did not talk to people on the business side about what business problems they had.

    They didn’t talk to the people who understand corporate data. They didn’t ask them, “What data do you actually have? What data do you need?” They didn’t talk to people who knew the business strategy or the business processes that were in place or needed to change. So it really is a team sport.

    And I like Brian’s discussion in the beginning about culture, because it really is about sort of hybrid organizations where you have people, you have leaders that know a little bit about all of these areas, and then a team that’s brought together that can work across these areas.

    Mitchell Ashley: Sure.

    Brian Dawson: Judah, can I ask, do you have any recommendations for DevOps teams on how do we evaluate or investigate where ML is a solution? And I ask this based on observation, we’ll see if we agree, that oftentimes people are looking for a problem to apply the solution to, as opposed to looking for ML as a solution to a problem they’ve identified. I’m just curious if you [crosstalk].

    Judith Hurwitz: I violently agree with you.

    Brian Dawson: Okay.

    Judith Hurwitz: So I think, and a lot of times people get so enamored with a new technology that they look at it as a way to solve all problems.And for a DevOps team, I think we are finally getting to the stage where there’s really reality in DevOps. It’s not the developers who are saying, “That’s not my problems. The operations people have to make this work.”

    There’s really beginning to be this collaboration between development and operations. And this shift left is definitely becoming real. So that’s definitely true, but for these teams to be successful, they have to have a holistic view of, where is your business going? What do they actually need? Why have they come to you and say, “We gotta do something?”

    Is it just because it would be cool to do something and spend money,or is there a real rationale behind this that they need from? What’s the pain that’s out there that they can solve? So, they have to start with that. They have to start not only collaborating between developers and the operations team, but with the business leaders, with the people who understand all of the data, people who understand security. So, all of this comes together in very much a holistic pattern.

    Alan Shimel: Fair, fair. Hey guys, we’re way past halfway through here and I wanted to kind of turn our conversation to the future. We’ve had a discussion on sort of the history of AI and ML and lessons learned, et cetera. But when we look forward, Brian, you mentioned the Linux Foundation has a, I dunno if it’s a daughter foundation or a subgroup, dedicated to AI and technologies.

    What do we see when we look sort of near term future? Forget, yes, one day, we’ll mimic human brain patterns, who knows if we’ll be alive by that, but near term, what do you see? What is Linux Foundation planning for?

    Brian Dawson: Well, looking forward, I – And it’s funny, ’cause you brought me in and I started to think about the impossible dream of what’s gonna happen in the future. I will call out and will mention again, I said at the start, the Linux Foundation is a parent foundation of what we call a Linux Foundation open source project that hosts other project.

    So LFAI & Data, you can call it a sub-foundation of Linux Foundation because they host, I believe at this point, 12 graduated projects with about 20 to 30 projects in incubation. So we’re talking 30 to 50 projects under the LFAI & Data umbrella that are all working on various aspects of shared efforts, multiple, large commercial organizations, as well as standards bodies coming together to drive the future of of AI and ML.

    Brian Dawson: What I do see coming out of LF AI & Data Foundation in the short-term is standards and foundational implementations, i.e. moving sort of beyond the sort of rudimentary discovery around AI, building sort of packaged or gray box implementations that everybody agrees and collaborates on, which I think will help unlock the less initiated, less expert vendors to begin to deliver truly AI based differentiating capabilities.

    So, to put that in short, because there was a lot of words is, I think it’s about establishing a foundation for us to start to build on and accelerate our progress within the AI and data space applied to DevOps.

    Judith Hurwitz: And so, Brian, I think that you are 100 percent right. And that’s when commercialization really happens, is when we have those standards and when everybody agrees to use those foundational services. I think the challenges that we face are, how to get the commercial vendors who don’t necessarily want to –

    They want to give lip service to these standards, but they really want you to only use their version of the “standard” so that customers never leave. So, it’s that stickiness factor that they are looking for. So, I think it’s it’s a hard journey.

    Brian Dawson: Yeah. Well, I think, Judith, you actually nailed the reason the Linux Foundation exists, frankly. And if we look at Kubernetes as a model, it was hard. Press Google could have easily said, “We’re not going to hand this over to the Linux Foundation. We’re gonna dominate this space. We are going to be the modern cloud OS.”

    But they understood for it to gain traction, for it to grow and truly offer benefit industrywide, they had to hand it over to the Linux foundation, a grow manage. They had to bring in Microsoft. They had to bring in Amazon to play in that space. And I see LFAI & Data serving that same role, what I tend to call in some language unlocking innovation, or as our tagline is, decentralized innovation.

    And I would beg to say that that is, we didn’t call that out directly, but one of the challenges that we’re seeing in the AI and data space. If there’s not short-term monetization, we’re not gonna do it. And if there is, then we wanna own it. So can we create a impartial playing field for everybody to come in and innovate together, and then build commercial solutions off of that?

    Judith Hurwitz: Yeah, it’s the challenge. It really is. And I think Kubernetes is a great example. I think data in some ways is more complicated because data’s really the crunch –

    Brian Dawson: Yeah. Well, I know everybody wants to own it. Everybody wants to own the data.

    Alan Shimel: No doubt about it.

    Mitchell Ashley: There’s discussion about AI needs to become an engineering discipline as opposed to this sort of edge specialty. Doesn’t mean that necessarily is gonna be applying everywhere, but I think how we apply DevOps or, I’m sorry, AI to certain kinds of problems, it seems to me the two most ripe areas are that highly complex environments.

    So it’s more and more infrastructure, it’s automated, more things are gone to digital. How do you manage the infrastructure or manage the triage problem-solving? And then of course, I think that the other areas, people will continue to find niche areas where AI can be applied to gain competitive advantage in a certain domain or space.

    And it seems like that’s the trajectory we’re on for quite a while. I don’t think it’s ever gonna be AI takes over everything, and it’s the thing that replaces DevOps or whatever. But it seems to me kinda a tool, but – Sorry, go ahead, Judith.

    Judith Hurwitz: No, what I was gonna say is very interesting. If you look at healthcare, for example, if you look at the spectrum firm, I can automate certain DevOps functions that are repeatable, where I can identify patterns. And as I collect more data, I can automate more things.

    You have something like healthcare, which in terms of orders of magnitude of complexity are just huge. And what I’m seeing is a lot of the vendors who think that I’m going to tackle healthcare with AI and we’re gonna own the industry, a lot of them are getting out of the business because right now, it’s just too hard and it will be too hard for quite a while.

    Mitchell Ashley: It’s true of the medical industry in many ways. It’s a hard nut to crack.

    Brian Dawson: Yeah. If I may start to dream a little and chime in based on something you said Mitch, I do see that in the near term, we are at a point where we talk about modern software development and delivery, the rapid pace which we’re delivering change, we’re building on inventions and progress made over decades. Library reuse is really heavy.

    And we are getting to a point where as we build more complex systems, we have to figure out how can we outsource sort of maintenance and management for one of a better word to use it kind of grossly to solutions like ML? How can ML help us continue to improve, grow and build on what we’ve done, but manage the scale and complexity?

    And I think we’ll move from some standardization, foundational blocks will apply more ML and both the operations and development space to manage that complexity, to maintain stability. But then I also eventually see the next stage being now, how do we apply ML and surface it to a developer at the time of a commit?

    Here’s what the expected outcome of this changes to help provide guardrails to the developer, the end state, or I wouldn’t even say the end state. I think we’d agree existentially there’s never an end state here. That we’re at a point where we don’t even have explicit CICD pipelines. We can commit code to a repository, the language can be inferred.

    We can give queues or signals to infer where we want it deployed, how we want it deployed. And I actually see ML and to an extent, layers of AI, just figuring out what to do with code at rest. So if you flash forward in 10 years, and we’re truly applying these to technologies at scale in the cloud, I just change a line of code.

    That change automatically is delivered to production. And AI/ML is helping us do that. So that’s sort of my dream. I don’t hit compile, I don’t hit build, I don’t build out stages and workflows, I just change code, and that’s running in a system somewhere.

    Judith Hurwitz: So, don’t you also want to have the ability to say before you say, you look at what’s happening, and don’t you wanna sort of at least at this stage say, “Are you sure you wanna do that? Are you sure you wanna delete that file?”

    So until we get to the point, because you have to be able to trust that the system is smart enough to understand if I make this change, what cause will that unleash? Because we’re not dealing with the perfect world in DevOps by a long shot.

    Brian Dawson: Yeah, I do want that. I want protection. I want it to know where the vulnerabilities are and warn me. And I do think, Judith, that would be a prerequisite into this deploying code at rest, or truly applying dev AI.

    Alan Shimel: Fair enough. Hey guys, we are just about out of time. I think this was a great discussion. If there’s one thing I could take out of it though, is that the crystal ball remains cloudy in terms of how this is all gonna play out and we’re gonna have to wait for it to kind of come into focus as we – But it certainly will be an important part of software development, of operations going forward. And it’s gonna be an increasingly important part.

    Brian Dawson: [crosstalk]

    Mitchell Ashley: I think the crystal ball is in Magic 8-Ball, ask again later.

    Alan Shimel: That’s a good one, Mitch. But for now, we’re gonna call an end to this episode of DevOps Unbound. Again, thanks very much to Tricentis for their sponsorship. Thank you so much, Judith. Thank you so much, Brian, for appearing here and hope to see you on a future episode. Mitch, as always, great job, riding shotgun with me.

    Mitchell Ashley: Good to partner with you. Absolutely.

    Alan Shimel: Absolutely. But this is it for this episode of DevOps Unbound. This is Alan Shimel, have a great day, and we hope to see you soon on another DevOps Unbound, as well as don’t forget, every month we do a live round table open to you, our audience, with questions. So stay tuned for that as well. Take care, everyone. Bye-bye.

    [End of Audio]

  • Applying Automation to DevOps

    Applying Automation to DevOps

    In the book, “Life and the Art of Engineering,” author Haresh Sippy said, “Automation is cost-cutting by tightening the corners, not cutting them.” Today, businesses and organizations are constantly on the lookout for ways to improve productivity while reducing inefficiencies across their operations. Automation has emerged as the natural answer as it addresses these issues while enabling organizations to scale their operations.

    And this is a natural fit with DevOps, which aims to automate and streamline and accelerate the development process from code generation to application performance monitoring. Automation promises better results, greater efficiency and reduction or elimination of errors, but many organizations remain hesitant to deploy it in critical operational areas.

    But there are many benefits to DevOps automation that organizations shouldn’t ignore:

    • Consistency: The most crucial benefit of DevOps automation is consistency. The absence of human effort can remove errors because automation tools repeat processes in the same way, ensuring the development environment remains of consistently high quality.
    • Scalability: DevOps automation not only allows new code to be released quickly but in larger quantities, as well. Automation also allows teams to deliver multiple projects in multiple environments simultaneously, reducing manual processes.
    • Speed: One of the first visible benefits DevOps automation will bring to the table is the sheer speed of the product delivery process. DevOps automation streamlines the configuration process, the development environment and other aspects of deploying new software, making the process significantly faster. With DevOps automation, software delivery becomes less dependent on people. 
    • Flexibility: Flexibility can often be an overlooked aspect of automation. However, with DevOps automation, organizations can change and modify critical code based on new requirements and deploy these changes in real-time, rather than having to train the team to manage the operations based on the new changes. 

    Critical DevOps Automation Processes

    Once an organization is aware of the benefits, the obvious next step is to figure out which processes to automate. 

    While a core tenet of DevOps is to automate everything logically possible, that is a long-term goal. On a more immediate basis, organizations would be well-advised to focus on automating a few critical standard processes. 

    This will allow for a smoother transition as well as more efficient results. But the question of which exact processes to automate is a subjective decision that will depend on each organization’s unique needs and technological requirements. However, some fundamental areas that may benefit from DevOps automation in any organization include the following:

    • Software Testing: Any organization with a digital service or product to offer spends considerable time and human resources testing the new offering or updates before rolling them out. Naturally, automating the testing process would bring greater efficiency to the test results while significantly reducing any errors, especially human errors. Additional tests that may be automated include unit tests, smoke tests and UI tests. 
    • Monitoring: Most organizations understand how important it is to adopt a proactive approach to monitoring. A key aspect of DevOps is consistently reading, recording and maintaining crucial data from regular monitoring. By automating this process, organizations can further increase the extraction of valuable B2B data and insights while ensuring greater security against possible breaches and data theft. 
    • Infrastructure: Infrastructure management is a critical area that serves a vital role in ensuring things run smoothly. This includes networks or servers that are consistently in need of maintenance, configuration modifications and initial setups. By deploying an automated infrastructure, organizations can increase efficiency while freeing up valuable human resources to be deployed elsewhere. 
    • Continuous Integration/Continuous Delivery (CI/CD): CI/CD’s entire purpose is fast application development and delivery. By its very nature, this task is ripe for automation as it requires testing and monitoring every little change or modification. Similarly, automating the deployment phase of CI/CD would ensure the distribution of successful updates is carried out proactively the very second it is ready. As a result, organizations will see updates being rolled out more efficiently with minimal errors or bugs in the final updates.
  • Quality Is a Top Challenge for Data-Driven Projects

    Quality Is a Top Challenge for Data-Driven Projects

    Software systems continue to produce more and more data. And making use of it has proven benefits — so much so that many analysts have, over the years, referred to data as the new oil. As a result, the majority of organizations are expending effort into refining their data — in fact, a recent study found that 84% of organizations have either already deployed or have data-driven projects on their roadmaps.

    Corporations like Facebook and Google are the poster children for business models that harvest and monetize end-user data. However, this is only one aspect; valuable data is being produced by internal systems as well, which, if leveraged correctly, can provide insight into software observability and increase process automation for DevOps teams and developers. For example, time-stamped application logs are necessary for informing SLOs to maintain reliability standards. There is an ongoing parallel investment into AI/ML deployed with cloud-native tools to act upon production data to drive further business growth. As such, 63% of IT decision-makers say new revenue opportunities have emerged due to data and analytics.

    The 2022 Data & Analytics Study, conducted by Foundry (formerly IDG Communications), explored how data-driven initiatives continue to be an important investment area for executive leadership. Below, I’ll review the key results from the survey to consider how organizations can continue to make intelligent decisions now and into the near future.

    The State of Data-Driven Investments

    We undoubtedly live in a data-driven world, and investment in related projects continues to rise. More than half (55%) of IT decision-makers plan to increase their data-focused investments — the report found the average spend to be $12.3 million in the coming year. This figure rises to an average of $23 million for financial services, which makes sense given the nature of modern banking.

    Naturally, the use of analytics platforms is increasing in tandem with the amount of data generated. In terms of specific analytics tools, 50% of respondents said they used business intelligence platforms while 47% used relational databases and 19% planned to invest in them in the next one to two years.

    We’re also noticing an increasing use of cloud-based solutions. For example, 27% of an organization’s data analytics workloads now run in the cloud. The report also found an uptick in cloud-based enterprise-scale data warehouse technologies, an area that shows signs of increasing in the coming year.

    Common Objectives

    So, what are the most common driving factors behind these types of projects? Well, automating internal business processes is the top goal for data-driven projects—half of IT leaders described this as a primary objective. This is closely followed by other ambitions such as improving customer insights (46%), aiding customer support (43%) and automating IT operations (43%).

    In terms of type, transactional data tends to be the most useful—54% of companies are using transactional data in their data-driven projects. This includes consumer purchasing behaviors such as sales, returns and credits. The next-most-common type is machine-generated data, which includes information from logs, sensors, telemetry, networks, security systems and/or IoT devices. The third-most-common type is customer profile information.

    Data has a powerful impact in a business context as a means to refine existing digital products. Other respondents added that a data-driven approach to DevOps helps increase visibility and drives continuous improvement.

    Data Quality: Highest Ranked Challenge

    Undoubtedly, particular challenges remain. The greatest hurdle is retaining data quality—41% of organizations reported dealing with this issue. Quality may be poor because data is unstructured or incompatible with other sources. Other widespread challenges include governance issues and integrating data from multiple sources.

    For those undertaking data-driven projects, 44% said they lacked appropriate skillsets, such as analytics training, data management, security, business intelligence and integration expertise. Companies also often faced funding and talent-related quandaries when beginning data-driven projects. For example, 26% of small-to-medium-sized businesses lacked the necessary funding to take on data-driven projects, according to the research.

    Another challenging area (that I’ve covered previously) is optimizing how data is collected and stored. As cloud storage fees rise, organizations will likely want to refine retention and resolution to avoid creating unnecessarily large, expensive data lakes.

    Rising Deployment of AI/ML

    AI/ML is already well-established as a way to advance the use of data. More than half (54%) of companies used predictive analytics or planned to incorporate it into their systems in the next 12 months. Just under one-third (31%) also either used or planned to use anomaly detection in the future. These areas along with other functionalities (such as natural language processing and predictive analytics) can be used to aid applications such as process automation, decision support, customer analysis, virtual agents and others.

    The industry is at an exciting moment for leveraging data in DevOps. Data is becoming more and more accessible to enable DevOps with a full-stack picture of the application life cycle. To direct future engineering investment, data-driven decisions will rely on things like performances and usage habits. Thus, it’s an interesting time to consider how you might leverage data for greater process automation and software development fluidity.

    The 2022 Data & Analytics Report surveyed 872 IT decision-makers (ITDMs) from around the globe working in various industries. The respondents’ average company size was about 12,000 employees. To view the report for more insights, you can pick it up here.

  • Why is Security Still in the Way? A Look at DevSecOps Right Now

    Why is Security Still in the Way? A Look at DevSecOps Right Now

    According to industry trend reports for 2022, DevSecOps is now considered to be one of the most effective approaches to building software quickly and securely. This effort, of course, means development, security and operations teams commit to addressing security as early as possible in the software development life cycle (SDLC). The goal of the shift into DevSecOps revolves around the need for more automation, streamlined CI/CD pipelines, less friction among teams and plenty of time for productive innovation. Incorporating DevSecOps practices into an organization unites security, compliance and development issues into a collaborative concern, one that allows DevOps to move fast without jeopardizing code quality. 

    As we approach the middle of 2022, it’s worth looking back and considering why security is still standing in the way of smooth software development. Security vulnerabilities have always been present in the process, and yet these pesky bugs continue to be the biggest threat to software performance. The question is, why?

    The Truth About Security

    Software has always been deployed and shipped with some existing bugs. But as software and services continue to dominate our everyday lives, the reality of this statement continues to plague both businesses and the end user. On some level, software development has yet to evolve into the next phase of security. Code is still rolled out regularly, only to discover security vulnerabilities in production. Some say the way software is developed today actually lends itself to defects, mostly because the pace of work required supersedes best practices around AppSec. 

    While the larger cybersecurity industry—and every DevSecOps team in the world—is striving to make software more secure, many organizations still prioritize speed and productivity over digital risk. This means the state of security (and, to some extent, the DevSecOps world) has become somewhat lopsided, with some enterprises making great efforts while others embrace a more lackadaisical approach. To make matters worse, many professionals today believe there has been a general under-investment in cybersecurity over the past several decades. 

    The Next Phase of DevSecOps

    The shift to remote and hybrid work models has led to mass adoption of cloud technologies and digital transformation, across all sectors and industries. As a result of this evolution, organizations today need to bear in mind that the more services they use, the more digital threats and overall risk they face. As this exposure to vulnerabilities increases, so does the need for new approaches to security. 

    The majority of modern enterprises do, in fact, have a security process in place, as well as other risk management solutions. But with the cybersecurity reckoning of 2021 and the stakes of cloud computing being higher than ever, businesses right now are feeling the pressure to think long and hard about why security still isn’t a guarantee. The trend of slowing down software development to address security more thoroughly, also known as the secret sauce of DevSecOps, is not free from roadblocks—but it is also far from optional. For many organizations, the implementation of DevSecOps practices feels less like security and more like speed bumps that just slow down the shipping of software. 

    Even so, the effort to improve security is happening now. Organizations do not need several solution platforms or tools to get started. They can begin right away; evolving their security tools and practices as they grow the business. With this growth comes one of security’s biggest challenges—scalability. Because toolchains across the cloud have become more and more complicated, setting policies and/or workflows in one tool is challenging. While these issues can be largely addressed by the adoption of an end-to-end security solution, organizations must continue to focus on strengthening supply chains, improving security oversight, and bolstering their approach to DevSecOps. Some ways to make this happen include:

    • Making sure CI/CD pipelines are integrated with DAST and SAST testing in development
    • Ensuring developers and DevOps teams complete vulnerability scanning before commit or merge, as they write code
    • Minimize manual security as much as possible with automated AI/ML tools to support more effective scanning and monitoring
    • Invest in management solutions that improve security in a multi-cloud environment

    Addressing the Supply Chain

    According to an industry survey, 60% of companies have decided securing the software supply chain should be a top priority for 2022. When we think about software supply chain attacks, we need to think about better controls to validate its integrity through all stages of the SDLC. Obviously, DevOps and automation have provided considerable help in this area; however, they cannot pull all the weight of security without proper governance and guardrails. 

    With a secure supply chain comes better security, as well as compliance, privacy and transparency. When a supply chain is secure, it can automatically enforce policy requirements for things like instrumentation, testing and architecture. As a result, software development and deployment become a repeatable process that is also reliable. Teams across the organization can rest assured that the products and services they are building comply with security expectations, such as validating code before production with static analysis and scanning tools. 

  • Technical Debt: When It Works and When It Doesn’t

    Technical Debt: When It Works and When It Doesn’t

    Every software team has probably faced technical debt—the accumulation of little development deficiencies that will require rework down the line. Just like financial debt, they accrue interest as they build up, in the form of difficult software modifications or challenges in adding new features. Time is the usual culprit; your development team has one timeframe for creation but your leadership team wants to bring the software to market faster. And when it comes to choosing between building a clean and polished piece of software or hitting that delivery date, you usually need to bow to pressures from on high. It’s at that point that your collaboration might go off track, with little pieces of incomplete code jumping into the mix.

    Without careful monitoring, technical debt will eventually slow down your development and deployment processes, degrade your product quality and limit your ability to innovate in a changing market. But is technical debt unilaterally bad? Not always. Sometimes it’s in service to a greater strategy. Let’s look at the differences.

    When is Debt Part of a Good Strategy?

    Pankaj Gupta, senior director, product marketing at Citrix, said when technical debt impedes an organization’s ability to function, that’s a clear sign they need to wipe out the debt and start with a clean slate.

    “But there are a few scenarios under which it makes sense to let debt accumulate,” he said. “For example, if the cost of addressing a technical debt is significantly higher now than in the future; if the debt is not impacting your immediate and short-term business needs or if you have an emergency release like a major security vulnerability fix.”

    Daniel Viveiros, chief technology officer at CI&T, recommended that teams keep the big picture in mind when making the right tradeoffs. “For me, it’s about putting the business first and understanding that constantly activating value is imperative for success,” he said. “Well-managed technical debt can be an effective tool to shorten the lead time. Deferring refactorings, modernization efforts and other common debts in order to prioritize deployments is a constant tradeoff the engineering teams deal with on a daily basis.”

    What are the Signs of Too Much Technical Debt?

    Gupta provided some warning signs for technical debt overload.  “You know you’re accumulating too much technical debt when the cost and/or time to fix the technical debt is higher than starting fresh,” he said. “And the time taken for each release and deployment is consistently increasing.”

    A less obvious result: a DevOps exodus. The development team will be the first to feel the impact of too much technical debt, such as having a harder time completing tasks and working long hours to fix frequent breakages. “Employees are leaving the company out of frustration from working on legacy systems, spending most of their time fixing ad-hoc issues, routinely missing and being on-call all the time,” Gupta said. What debt reaches that level, resignations begin to mount.

    Viveiros pointed out that compiled debt will impact system resilience: “Bugs are originated more often and the change failure rate goes up. Speed is perceived as risky instead of something beneficial to the digital delivery process.”

    The ultimate sign of too much debt, he said, is poor business performance and lost revenue. “If downtime equals lost opportunities, the technical debt will become a financial debt,” Viveiros said. “Too much technical debt may impact the perceived quality of the product. As a result, the product and brand’s image will suffer.”

    What’s the Best Way to Manage Debt?

    Failure to deal with technical debt means kicking it down the road to another developer to tackle—and no one wants to be that person. So what’s the best way to take control and pay it off? To start, approach it as a collaborative effort and implement these four practices.

    1. Identify the types of debt. Not all technical debt is created equal. Distinguish between the debt you accept in the moment as something to fix later on and the inadvertent debt you discover—and categorize it into code debt, design debt, build debt and other forms to identify origin patterns.
    2. Analyze and automate. Analyze the origins of your debt and look for ways to tighten workflows or automate certain tests and processes. By reducing common errors and hidden bugs, you can stop them from snowballing into technical debt.
    3. Develop new policies and standards. These should clarify when debt makes sense and when it causes unacceptable damage. For instance, an immediate release of a security patch would be deemed acceptable while errors that will eventually cause considerable downtime would not.
    4. Communicate the cost to IT and business leaders. When another hyper-speed deadline comes your way, make sure your leadership understands the risks to product quality and developer retention. With the price tag made clear, they’re more likely to change delivery dates or provide funding for additional developers.

    Like most out-of-control spending, harmful technical debt begins with poor decisions and a certain level of denial. By approaching decisions with greater clarity and foresight on both the Dev and Ops sides of the house, teams can give themselves more breathing room and the space to produce exceptional software.

  • Survey Surfaces Challenges Ahead on National DevOps Day

    Survey Surfaces Challenges Ahead on National DevOps Day

    A survey published today for National DevOps Day found nearly two-thirds (63%) have seen an increase in the frequency of service incidents that have affected their customers over the course of the last 14 months.

    The survey polled 1,046 engineering, IT operations, DevOps and site reliability engineering (SRE) professionals at organizations with more than 300 employees. The research was conducted by Transposit, a provider of a platform for connecting DevOps workflows, and found well over half (58%) of respondents reported the cost of downtime for their organization could be, on average, up to $499,999 per hour, with 40% of respondents noting that the cost of downtime has increased in the last year.

    Respondents said the top contributing factors to the increase in service incidents were digital transformation (61%), new product rollouts or product updates (55%) and methods and tools for collaboration that do not adequately support their team working remotely (49%). A full 90% of respondents reported there has been an increased focus on digital transformation during the last year with 73% having expanded the technology stack they support. More than half (55%) have increased the size of their DevOps teams.

    Overall, 46% of organizations reported they have experienced between six and 19 major incidents over the course of the last 12 months. More than half (53%) also reported an increase in the amount of time it takes to resolve incidents over the course of the last year, even though three-quarters of respondents (75%) said their organization increased their adoption of SRE practices in the last 12 months. Additionally, 65.1% of those organizations planned to hire site reliability engineers in the next 12 months. However, only just over a third of those respondents said their organization planned to expand their SRE efforts in 2022.

    A full 83% said up to 50% of their engineering operations processes are automated, but over half of the respondents that use SRE practices noted they still manually enter data into an IT service management (ITSM) application or other system-of-record to keep track of actions that were taken by humans during the resolution of an incident. Nearly two-thirds said their approach to automation was incremental, in which they begin by codifying processes and worked up to more advanced, fully automated scenarios.

    The top three tasks respondents would like to automate were service requests (53%), change requests (43%) and user provisioning (40%).The top barriers to implementing automation were cited as inadequate documentation of institutional knowledge and existing processes (56%), lack of clarity about what to automate (55%) and an inability to share knowledge (52%).

    Investments to improve their incident management processes over the next 12 months are focused on automation tools or applications (48%), communications/collaboration tools or applications (42%) and integration tools or applications (41%). Only a quarter (25%) said their tools are integrated through one tool or platform.

    Lack of unified communication with teammates (45%), processes that have changed or are harder to follow while working remotely (42%) and lack of visibility into dependencies and what teams or people are responsible for the code or infrastructure (38%) are all cited as reasons why serviced requests are now more complex.

    Transposit CEO Divanny Lamas said despite all the challenges facing DevOps teams, it’s fortunate that advances in low-code automation are making it simpler to more widely implement DevOps best practices. The challenge is finding a way to implement those best practices faster at a time when IT environments are becoming more complex.

  • Creating Automated GitHub Bots in Go

    Creating Automated GitHub Bots in Go

    Events management requires quick responses, which can be challenging since there is typically an incredibly large volume of events that take place every day. Therefore, relying on humans to respond to these events quickly is error-prone, messy and often frustrating. These events have easily reproducible steps that can be codified and are perfect reasons for a bot to exist.

    Creating a bot reduces the daily noise of events so that important humans can focus on important human activities—such as programming, innovating or, most importantly, sleeping.  There are already many bots that exist and that can solve numerous problems. For example, Dependabot helps keep your GitHub dependencies updated with automatic pull requests. Google Calendar has a Slack bot that reminds you when a meeting is about to start. Of course, there are also situations where there isn’t a pre-made bot yet, which provides the perfect opportunity for developers to create their own!

    My preferred programming language is Go, and I’ve used it to create multiple bots. A favorite bot I created helps to manage a large open source project called Telegraf, which I discuss in more detail at the end of this article. Bots are not limited to open source projects; another bot I created proved valuable when I worked for a larger enterprise company. It was a Slack bot that could run commands on Amazon EC2 Windows instances and help teams quickly perform actions for customers, such as resetting passwords in seconds as opposed to days.

    Creating Your Own Bot for GitHub

    Let‘s look at how you can get started creating a bot written in Go that can interact with GitHub. The package google/go-github lets you interact with the GitHub API and makes creating a bot for GitHub easy. Tools like these are your first step in developing your own bot. Using this package, you could create something similar to what Dependabot does. In order to begin receiving GitHub events for you to react to (such as someone creating a pull request), you need to create a GitHub App.

    GitHub App 1

    The GitHub documentation does a great job describing how you can do this, but the important piece to be aware of is a callback URL. The callback URL will be a publicly available address associated with your Go program so that GitHub can route events to it. I use AWS Lambda functions to host the bots I’ve created and managed by using Serverless Framework, but you could use anything you want as long as you get a public URL. If you are interested in using a Lambda function as well, your “main.go” can be as simple as just calling “lambda.Start” and passing in the function that will hold the business logic.

    GitHub App 2

    Because you are using a public URL, security is important. Once you have created the GitHub App, you will be given a secret token that you can use to help validate incoming requests. The google/go-github package makes this easy; all you have to do is call github.ValidatePayload and pass it your secret token and incoming HTTP request to ensure the payload is valid. If successfully validated, the function will return the JSON payload that defines the event you just received. You can parse the payload by using the function github.ParseWebhook, which will return to you an empty interface and can hold any value. Therefore, you will need to use type assertion to access the underlying type to see what event you have received.

    GitHub app 3

    Depending on the event type, you might want to take different actions. Perhaps if someone made a new issue you would receive the type github.IssuesEvent, which would be good to perform initial triage such as adding the appropriate labels. It is also possible to receive the event type github.CommitcommentEvent, which would indicate someone had just made a comment that you could check for special commands such as automatically restarting the build system. You could, for example, also create a welcome comment whenever someone creates a new issue, as seen in the code snippet below.

    Picture 4

    Creating a Slack Bot

    These examples have focused on interacting with GitHub, but can be replicated to create a Slack bot similar to the Google Calendar bot. The package slack-go/slack allows you to interact with the Slack API with which you can post messages or create “modals” that allow users to interact with custom buttons and lists.

    Creating a Slack bot is a similar process to creating a GitHub App. Slack provides a user interface to register a bot, and similar to GitHub, the main thing it needs from you is a public URL so it can send your bot events. You can then also verify incoming events by using the function slack.NewSecretsVerifier using the secret token provided by Slack. As an example, once these steps are completed, your bot can then begin sending messages to channels to help you answer questions.

    SlackBot

    Bringing it to Life

    An example of a bot can be found in the Telegraf project, which is InfluxData’s open source data collection agent. We have a bot written in Go helping us maintain the repository named “telegraf-tiger.” One of the first interactions users have with the bot is when they submit a new pull request. When the request is made, the bot will chime in with a friendly reminder to sign our contributor’s license agreement (CLA).

    reminder

    Once signed, the bot can be asked to verify if the CLA has been signed by adding a comment “!signed-cla” and give the pull request a shiny green checkmark to let the maintainers know everything is ready.

    Another area our Tiger bot supports is sharing the artifacts that our continuous integration (CI) system builds for each pull request with a helpful comment that links to each type. This helps spread awareness of the artifacts so that anyone can easily help test the new feature or bug fix without having to set up a local developer environment themselves.

    CI pull request

    The Big Takeaway

    Bots are an exciting developer tool that can automate mundane assignments and allow engineers to focus on bigger, more engaging tasks. Resources such as GitHub also make it easy to develop bots using Go in various ways. Bots are commonly found across interfaces, with some simply reminding us of upcoming meetings, and others ensuring that important license agreements are signed.

    The use cases for bots are endless, and with the right tools, bots can help engineers focus on more significant tasks at hand. Developers can rest easy knowing their bots are automating tasks and responding to the endless alerts they don’t want to deal with. Instead, they’re able to focus on important human activities, like sleep, so they can dream up the next big thing.