Category: DevOps Toolbox

  • Splunk: Creating a Resilient and Dynamic Organization

    Splunk: Creating a Resilient and Dynamic Organization

    Mitch Ashley: I have the pleasure of being joined by Ryan Kovar, Ryan is a distinguished tech security technologist and leader of SURGe with Splunk, and Cory Minton who’s field CTO for The Americas with Splunk. Welcome, guys.

    Ryan Kovar: Thank you.

    Mitch Ashley: Good to be chatting with you both. A topic that’s very top of mind for CISOs or IT leaders, of course, is how do we create a resilient and dynamic organization that can keep pace with the business and the change that we see in the technology landscape, whether it’s our own infrastructure, own application portfolio, cloud, et cetera, all of those things, but also, of course, the tax services that are evolving and changing as what the bad guys, the threat actors, are doing. We’ve also not only got to have a great and resilient technology stack, but organizationally, process-wise, all of those things have to fit together into a cohesive strategy.

    We’re here to talk about that, thinking about it as an IT leader, whether you’re in security or IT or combination of both, and discussing your all expertise experience, but also you talk to a lot of customers as leaders in Splunk and the kind of things that you do. Cory, it’d be great to have you kick things off. Maybe we should start with if you want to start with the people domain or maybe you want to set it up a little bit differently. I’d love to hear your initial thoughts on this.

    Cory Minton: Yeah. No people’s perfect. I think the people process technology lens on tackling any sort of problem for leaders today is an appropriate framework. I’m happy to talk about the people portion of building great cybersecurity and IT organizations that, like you said, deliver resilience.

    Ryan Kovar: For me, at least when we think about this, you said IT leaders, and I think one of the big changes I’ve seen across for cyber for CISOs is it’s no longer IT or security. It is business leaders. When we started talking about people, that to me starts resonating because that is a cognitive change in how CISOs think of themselves and they think of their value, which we often get stuck in the technology part of people process and technology because a lot of us are technologists at heart, but I think the big change that Cory and I have seen when we start talking to CISOs and CIOs is this convergence of skills and the need to support the business differently.

    Cory Minton: Yeah, and it’s an interesting sort of people market, too. I think leaders have to think about the fact that, yeah, in technology, there has been some turmoil in some of the big tech companies, but there’s still a lot of really great talent out there. I think that there’s choice. With unemployment so low, and there’s, what, like 16 million jobs unfilled currently in the US and a lot of those being in the tech sector, people have choice and they have a place where they can go.

    I think, leaders, if they’re going to build a talent pipeline and build a great organization, they have to find folks where they are and bring them in and connect them to a value, a mission and a vision that they get excited about. Certainly, securing digital services and protecting against bad actors is exciting in and of itself for many folks, but actually connecting those people to how it affects the outcomes that matter to the business and that whole value creation process, I think, is a real critical skill for folks to understand today as they, again, are out communicating and interacting in the network, building networks of talent pools. I think that clear vision of why it matters and why the work that somebody would do with you matters is critically important.

    Ryan Kovar: That reminds me of the cliche, of course, that people don’t quit jobs, they quit leaders. When you flip that around, one of my mentors, Susan St. Ledger, told me, you can evaluate the success of a leader by how many people follow them to another company. I know right now, in this market, what I’m finding is people are staying longer because they like the people they work for or they like their leadership team, they like the culture. The money, even if they’re not making as much as they were last year because of inflation, the people I know who stuck around longest are the people happy with the people that they work for and work with.

    Mitch Ashley: It’s interesting when we talk about people. I’ve worked in part of my career where the sort of higher the purple unicorn that doesn’t exist, that got eight skills, that needs 20 years of experience for things that have only been around for three. We set our subs up to hire these phenomenal people of which there may only be a few in the world that are like that, but it’s about growing our people, but also growing our organizations, developing them. It’s not just hiring people. It’s building a team. It’s building an organization that’s got the right talent at the right time at the right place to match with the business needs, and that changes. That means you can’t always hire for it. You’re partnering for it. You’re working with companies like Splunk to bring in skills and expertise maybe you need for the moment, you need for a project, you need for strategy, all of those things. Love to hear your perspective about broaden what we think about people and how we incorporate that part of it into our strategy.

    Cory Minton: Yeah. I’ll say, top to bottom, start at the executive level down to practitioners. Everybody has a partner. Everybody has a consulting partner typically that they bring in to, like you said, fill those gaps where it’s a new skill, it’s a new technology, it’s a new capability that they’re trying to deploy for the reasons that they’ve chosen to do so. They’re valuable, but you may not have those skills internally. Training takes time and, oftentimes, just finding that partner that can help you solve that particular problem is incredibly important. I think, as the security landscape and, frankly, the tools being used to deliver and develop the latest digital services continues to get more complex, expecting to hire that skillset completely, probably not realistic especially as digital transformation objectives have an ebb and a flow of momentum and the amount of work being done on particular projects.

    Yeah, I see it as one of the top areas where, when I’m talking to CIOs and CTOs and CISOs, when they think about, “Hey, we want to deploy this new capability within Splunk,” or, “We’re, frankly, trying to figure out how to integrate better with our observability and IT teams to derive more value from the tools we’ve already purchased,” it’s oftentimes that people conversation of, “How do I actually leverage your talent to show us what good looks like and bring the experience from other companies similar to us that have already been down this path to, call it the cheat code, how do I do the thing that you guys already know how to do? How do I buy that capability?” Oftentimes, it’s through people.

    Ryan Kovar: My experience, I’ll go the other side of the halo and horns there down to just the nitty-gritty hiring. Recently building out my team, I intentionally carved out three slots for entry level, early career folks into our cybersecurity team, which is pretty rare for Splunk and pretty rare for a lot of security research teams, but the way I did that was really rewriting the job description to be embracive of second-career people. I actually really dislike the term “there’s a problem with the pipeline for hiring”. I think that’s completely false. The problem is not in the pipeline. The problem is we have a valve on incredible talent and gate-keeping who actually gets into the pipeline. We do see a problem with the pipeline when I start looking for folks who are underrepresented in cyber especially 15, 20 years down.

    What I don’t see is a problem of, basically, the reservoir. The reservoir is healthy. The reservoir is huge, but people either don’t feel like they’re welcome or they self-select out especially of cybersecurity. For me, a lot of the people problems that we have, if we can start working on the problems of today, future us, as a security leader, will be very happy if present me can help unlock that reservoir by turning open the valve by creating job descriptions and roles that are less around years of experience and understanding technologies that I can Google or generative AI my way through today.

    I think there’s just a lot of flexibility especially for folks who have critical thinking and communication skills. I can teach you TCP/IP. I can’t teach you how to communicate. I can’t teach you how to synthesize information. I need you to walk in the door with that. Something for me around people and these hiring gaps that we see is really about being more embracive of nontraditional cybersecurity and IT roles and then also facilitating them in ways they can succeed.

    Mitch Ashley: Well, and it’s also about hiring for what we need today, but also where the ball is going to be downfield when the ball lands. Part of that is hiring people who have demonstrated their ability to learn, adapt. Yesterday, they were TCP/IP expert. Today, They’re the security threat landscape expert. How did they do that? How did they get there? They learned it. It may have been on the job, but there’s also a lot of self-motivation and just skill in that learning, and that repetitive learning skill is I think something you can harness or leverage to accelerate their career as well as what you need.

    Cory Minton: Absolutely. It’s one of those things that, always, some folks get turned off. As Ryan said, sometimes there’s a valve, and one of those valves is like, “You must have a college education.” While I don’t necessarily believe that a college education is always the right thing, I think what it proves if you have one is that, as you said Mitch, you’re able to go through a structured learning process in an organization that’s institutional, understand dynamics and things and achieve an objective which was set before you that had some measurable outcomes that you had to deliver, which I think is something true for all of us in a corporate responsibility job as we have to operate in an institutional environment. We have to consistently learn new things. We have to interact with people around us. I think college is a good measure, but, as Ryan said, oftentimes, a second career is maybe even a more powerful measure of somebody who’s done that successfully.

    Like Ryan said, the technical skills are learnable and, frankly, the fun part is some of the technical skills are actually getting obfuscated by advancements in technology. As we think about things like, Ryan, you mentioned, generative AI, we can make the joke, but, candidly, wouldn’t you rather hire somebody who’s got incredible cybersecurity skills and understands the landscape more so than somebody who’s just really good at crafting queries because then, if you have somebody that has that domain expertise as technologies like generative AI and other sort of assistive technologies continue to evolve, then the domain expertise becomes the most important because then you start interacting on a natural language way and you don’t have to have those same technical skills to get there, which I think is probably one of the most interesting uses for generative AI is actually bringing out the barriers for technical ability to execute in a job like security or IT.

    Ryan Kovar: Prompt engineering will be one of the most significant requirements for entry-level and mid-level jobs by next year, in my opinion, categorically.

    Mitch Ashley: We’ve already been through one generation. It’s called search engines. Now, we’re doing it with generative AI.

    Ryan Kovar: Yeah. Yeah. No. We’re talking about this, but the reality is I’ve been doing cybersecurity and IT since 1999. Cory, there’s enough gray there. I’m sure you’re about the same generation. When I started, there was no Google and you had to read the Microsoft TechNet documentation, and then there was Google, and I put 65 CD binder in the trash and said, “Never again.”

    The fun thing that I always tell people that I’m mentoring or advocating for in cybersecurity is I’ve been doing this for 24 years and, of the 24 years, I have about four years of knowledge that’s relevant. I have 24 years of wisdom, but four years of knowledge that’s relevant. Everything else, no one cares that I can fix Exchange 5.5 driven pub EDB and I know how to defrag a Windows NT 4.0 server. That doesn’t matter. That’s one of the great things about cybersecurity and IT in the general is that you can become a subject matter expert in something very quickly and not have the bias of age.

    Mitch Ashley: Well, let’s turn our lens to the process part of it, and we’ve talked a lot about people and we could talk a lot more about it. There’s some great conversations that we’ve had already about that. We’ve got to have processes that are well-oiled to highly tuned, but can be responsive of incident management when things happen. The organization now has to operate cross-functionally, the stove-pipes, the things that we’ve lived with for so long. Now, we have to operate cross-functionally. We’re trying to tear those down, but our processes have to work across those.

    I’m interested in your all thoughts about how are organizations, what do they best do to adapt to what we need today so we can be speedier, respond more quickly and more reliably to threats or incidents or needs of the business?

    Cory Minton: Yeah. Do you want to to hit it, Ryan?

    Ryan Kovar: No. Please, Cory, go ahead.

    Cory Minton: Okay. I was actually going to say it’s actually not even an option anymore. We talk about that they need to do it. It’s not an option. If you look at some of the SEC’s recent rulings on the disclosure of material and incidents that happen for publicly traded companies in the US, you have to report that. You have to have that cross-functional view of how if a cybersecurity incident happens or an IT sort of incident outage or breach, any of these categorical things that happen, if they have a material impact on operations that would affect shareholder value, then you must disclose those things, and so now the impetus is on every publicly traded company to get this figured out really well. It’s got executive buy-in now that we’re going to tear down the walls between security and IT operations and our engineering teams developing our next digital capabilities because, anywhere across that spectrum that we have a process breakdown, whether it’s externally caused or internally caused, if it’s material, we better report it.

    These are no longer like, hey, it’s a good idea. This is like “test that theory and report the results” kind of stuff that the SEC will send somebody to jail over. We’ve already seen some convictions on previously publicly traded companies on misreporting things. It’s no longer a game. We’re serious about this. I think, from a process perspective, one of the things I’m seeing is reaching across the aisle. CISOs are talking to CIOs and CTOs more so now than I’ve seen in the last couple of years because they have to understand those impacts and they’re looking to organizations that actually already do some of them within the company that are already being trusted by different pockets of the organization. They’re looking externally for that guidance and help, whether it’s from consulting partners that are, hey, advising on security operations or advising on software development capabilities or technology partners.

    I mean, even the CISA, the Cloud Infrastructure Security Agency, their recent strategy update talked about one of the key pillars of their annual strategy for resilience included technology partners that were going to help them achieve that resilience. I think we have the impetus. The measurement is required now. There’s no longer a game, and I think executives are starting to understand it and they’re looking external to say, “Who can help me deal with this?” No matter where we land on what is material and some of the legal questions on how it’s implemented, we still have to respond and we still have to report some of those capabilities. Now is the time to start reaching across the aisle and starting to ask questions of your partnerships that you have in the org on how can you help us solve this problem.

    Ryan Kovar: I look at things like DevOps which still today has a little bit of a carve out separate. In a lot of organizations, DevOps is a slightly different place than maybe traditional security and, well, certainly than security, and then possibly even different than traditional IT engineering or infrastructure. In today’s society or today’s world of technology, a lot of the security issues that organizations are facing are in their DevOps pipeline, and so I find it fascinating what Cory said. The SEC has this term, material finding. You have to report a material finding.

    Now, I as a security professional may very well know what that is, but, frankly, the DevOps world has gone feral to a point where the security organizations are not a part of it and so the DevOps team are the only ones who understand what the security implications are. They’re the ones finding it and, because they’re in a DevOps mind frame, they’re not stopping and saying, “Let’s create an incident. Let’s walk this through.” No. They’re just fixing it and they’re moving it on. That’s how the cloud works. That’s how DevOps works.

    There is this essential need for us as an industry to really start reaching more across because we are being outpaced by DevOps home growing their own SecOps without oversight, without the wisdom of the security world, but we can’t stay in the way of the business, which is why that’s so important for us to go across and understand this because now there’s regulatory requirements and the real world is people are going to move forward whether they like it or not, so get on that train.

    Cory Minton: Yeah/ it’s funny, Ryan, you say that. I was reading the SEC reports from JPMorgan Chase. Jamie Dimon, their CEO, said in the letter to shareholders he had two sort of funny juxtaposed statements that used the same phrasing, which I found interesting. He said two things he could not overemphasize, one was, “I cannot overemphasize the need for cybersecurity in our organization. Everything must be secure,” and then later in the letter he said, “I cannot overemphasize our need to deploy innovative technology,” which is exactly what you said. If my CICD pipeline is continuing to push updates, but I’m not applying those cybersecurity principles that, like you said, was incident review and forensics and actually looking into it, then we’ve missed the boat on this. One of the key parts of this SEC filing is you not only have to report the outage, but you actually have to from an annual reporting perspective talk about your processes that you’re using to ensure cybersecurity in those areas. That’s one of those that’s like the CEO is saying this to shareholders. We better pay attention.

    Ryan Kovar: The CEO of a bank that touches 20% of every dollar in the world every day, right?

    Cory Minton: We cannot overemphasize.

    Ryan Kovar: I love that example because I look at DevOps as a perfect world where I have no fear at all that the DevOps team will identify and remediate security issues very quickly and then continue to do so because they’re not looking at the larger picture. Why did this get ingested? Why do we have this happening? Why is this CRO a threat actor you may not know anything about a threat actor. You may not understand that Tampa strawberry as per Microsoft has a really significant interest in your organization, and one of their TTPs is actually moving up the chain and actually jumping ahead for software supply chain.

    That is something that I would not expect a DevSecOps engineer to understand, but that is where the context of a larger security world, and that’s why this resiliency message, this is why it’s working across aisles. It becomes so important for the process. I honestly think the technology is usually the easiest part of this. The people and the process to implement and secure the technology, way harder, way less interesting than most people, except nerds like me, but much harder.

    Mitch Ashley: Well, and what you really bring to light is that the security specialists are going to understand the threat landscape much better. DevOps developers, et cetera, know their environment focused on the things that they are. You mentioned Jamie Dimon. Those two things, he’s expecting us, our CEOs are expecting us to figure out how we bring that together, how we make that happen so we don’t have feral organizations, so we don’t have processes that are brittle when things really fall apart or our environments get more complex. I mean, that’s an obvious thing. They are complex and they’re getting more complex, and no one understands the whole thing, so we’ve all got to pull together and say there are three dimensions to this problem, not one, and here’s the steps. We together figure out how we fix this.

    Great. Well, let’s talk about the technology perspective then. Obviously, Splunk being a great technology company, have been around for a long time. I remember finding Splunk on the showroom floor in the early days still doing black T-shirts then just like they are today, a great company that you all work for. Let’s talk about some of the technology side of what’s important and, again, thinking about not just internal people and processes, but also third parties that we’re using like the Splunks of the world in our organizations and how you can help with this challenge.

    Ryan Kovar: I’ll start really easily on this one, and then Cory will actually say something much better. People buy software to solve their problems. That’s it. That’s the easiest way when I pay people. I didn’t come from a sales background. I was operational. I was a threat hunter for the government, threat intelligence at DARPA, places like this. I didn’t think about why I bought software until I worked for a software vendor. It’s very easy. You buy software to make your life easier, to solve a problem faster or to solve it better.

    When I look at what Splunk does, a lot of our recent efforts are really around this incremental growth of just making people’s lives better, making people’s lives easier, cutting down the barriers to do their job faster because that’s what people need, whether it be security, whether it be observability, whether it be traditional IT engineering, that to me is really what we focus on here at Splunk which is why I’m still here after nine years. We make people’s lives better and we allow them to fulfill the mission that they’re actually being paid for rather than fighting the software that they’re buying and paying money for. That to me really is at the heart of what we do.

    Cory Minton: Yeah, and I think, if anybody’s seen some of the Splunk messaging in the market, we’ve really rallied behind this idea of resilience, and I think it actually sums up nicely the things that we do which I think, based on the conversation we’ve just had about the people and process, things that are challenging organizations, I think we’re in a really unique place to be a partner, to be one of those partner organizations and be the software that people buy to make their lives easier because we sit in this unique place of being one of the few organizations that actually unify, tear down the walls between security, IT operations and those feral DevOps teams that are out there building the next digital services because our corporate mission is around resilience, and we think of that as a simple statement of, “How do you keep all this digital business secure and up and running?” Simple statement, but, secure, not a simple thing to do, and up and running oftentimes equally on simple thing to do.

    Ryan Kovar: Sometimes at loggerheads.

    Cory Minton: Yeah. Exactly, a pearl of either sort of objective. There used to be this term we called a MOM. We all love MOM, monitor of monitors. It sits above. It sits at an enterprise level, at a higher level to give you visibility. I think that’s what folks have maybe struggled with that we see. Hybrid cloud is reality, right? There’s lots of clouds, not one cloud. Clearly, there’s leaders in terms of revenue, and each quarter they grow at different paces, but candidly, even Michael Dell said some years ago, “The cloud is not a place. It’s more of an operating model.” As you see SaaS being part of cloud journeys, yes, data centers are still run by organizations. They just are run now in a much more orchestration pattern similar to cloud providers, and that reality of lots of silos across different landscapes is creating challenges.

    If you use one tool that’s provided by said provider, then, yes, you can secure and keep up and running that one particular cloud real estate, but what about how it’s connected across the organization? How about the ways that applications traverse deployment centers? I think that’s what we really see as interesting is organizations are leaning on Splunk especially in the macroeconomic conditions where everybody’s focused less on growth and more on profitability, so they’re looking at like, hey, how do I reduce the number of tools?

    Ryan Kovar: More out of less.

    Cory Minton: Yeah. Exactly. Do more with less. They’re looking at it and going, “Wait a second, so I have this network environ, this router, so my data center. I’m sending the logs from that thing to nine different tools. Why am I doing that?” That’s nine times I’m paying for that bit of data to be stored and processed and analyzed by some number of tools, and organizations are looking at it and going, “Maybe that’s not smart. Maybe that same piece of data has nine questions being asked of it, and is there may be a fewer number of tools that could answer those nine questions effectively to give me the outcome without having to have all this sprawl?”

    It’s a unique conversation that we’re in today. Candidly, the tools consolidation, that rationalization of tools is one, looking at data. The data deluge hasn’t stopped. As we talk about sending data to nine different places, well, as that data becomes richer, it’s no longer just logs. Now we’re looking at metrics for real time. We’re looking at traces for spanning. How does an application actually impact multiple sort of environments? The data is getting rich and large. Is it all equal? Should we all be sending it to the same place? Should we be treating it with some valuation?

    Those are conversations that I’m having today that technology is helping solve, and Splunk is doing some really interesting work in the data rationalization and in that tools consolidation area that’s making it easier for customers to do more. As Ryan said, you bought a piece of software to make your life easier. We’re trying to go out and help folks figure out how to make more lives easier with the software they already bought.

    Ryan Kovar: I’m just required to hit all the buzzwords, so I’ll say ransomware now. It’s fascinating to me. Resilience, as in the concept of business leaders and cybersecurity, CISO leaders having to come together, I think, partly is really being driven by this threat of ransomware because the first time a cybersecurity threat has consistently and continually impacted every aspect of a business where 10 years ago, 15 years ago, oh, APT1 from China, exfil data. Well, that was the big issue. Well, by the time you heard about in the news, it’s already done. It’s actually relatively small scale. The impact is we were compromised, there was a breach, and now we have to deal with the public relations fallout, the stock drop, and also we have to make sure this doesn’t happen again.

    With ransomware, it’s actually we’ve encrypted your critical business systems and you can no longer do business, and now you have questions of do we pay the ransom? Well, now you need to involve the finance or you need to involve your chief legal officer. Oh, it’s publicly out there that we have this security issue. You have to involve the PR team, the public relations team. Oh, it’s actually knocked out our core business. Oh, well, now you’re involving the sales org. Now you’re involving the IT org. It’s coming across all these different places, so it’s builds back into what tools do you have that facilitate this and then, going back to the people part that we touched on earlier, this is where someone needs to be across all of them because they need to understand the impact and actually drive those changes that are needed and, because of ransomware, we actually have to deal with it. We can’t just pretend it doesn’t exist. CISOs can’t just be nerds in the closet playing cyber. They actually have to be able to speak eloquently to their business leaders, to the board of directors and others.

    Cory Minton: Yep. I’m just going to jump on the bandwagon of buzzword bingo. One of the things we’re seeing a ton of, too, is we talk about new digital workloads constantly being brought in, new innovation, we’re hiring consultants to bring in and actually help generative AI get real in our organizations, there’s a lot of conversation about is generative AI threat versus risk? There’s a whole kind of conversation there, but one that actually, as we’re moving past the hype and the AI washing nonsense that’s happened in the market, once we’re getting to real, a lot of CISOs have been talking about or talking to are trying to figure out how do I deploy those technologies internally? How do I go find the models that I can train against my data sets that would actually go help my teams do their jobs better?

    What I would say is is that’s another one of those areas where, as you’re innovating and bringing in new technology, treat it like another digital workload. You’re still going to have to detect when something goes wrong. You’re still going to have to investigate when something goes wrong in that generative AI hallucinates, it has an outage, whatever, and you’re still going to have to respond to those outages and take those learnings and put it back into the operational process of how do you run that system more effectively.

    I think people need help oftentimes finding those consistent patterns that we have to deploy to just keep things securely up and running regardless of what the workload is under the covers, and I candidly see that as one of the greatest areas for leaders that are looking to partner with a professional services organization or an assay, those outside consultancies. There’s a lot of value there in having them help bring in the patterns especially on proven technologies to short cycle, increase your time to value or, excuse me ,shorten your time to value when thinking about those new digital workloads and making sure that they’re secure and up and running.

    Mitch Ashley: I like where you’re taking this, which is one I wanted to wrap with, which is our organizations are expecting us to innovate, not just ourselves innovate, but also support the rest of the organization to be able to innovate. You talked about Ransomware. The newest thing is they’re not encrypting your data anymore. They’re just threatening to release it, so it’s not a matter of losing the data. It’s losing control over it. AI is the next, maybe it’s the next boon to phishing. We may not be able to tell the difference between an email from a generative AI LLM that’s been trained on how our CEO talks versus a well-scripted, predefined email.

    These new kind of threat models as well as new technologies force us to innovate and find new ways to do things. We have to help our people with tools, technologies, processes. I’d love to hear some more thoughts about what do we do to set ourselves up for success to be able to innovate in a way that’s going to help support where the organization is today and where it’s going?

    Ryan Kovar: Well, I’ll make a slightly controversial opinion here. I don’t think there’s ever been a single technology that outweighs the benefits for the offense or defense. I look at generative AI as something that is going to liberate security organizations from the shackles of mundanity. It’ll democratize what we can do in terms of analysis. It will also do the same for the adversaries. The only people who will lose are those who are not taking advantage of this technology on either side.

    I really do believe that we can all up-level as we go through with different technologies just as the adversaries are. When we invented a trench or trench warfare in World War I, they invented the tank. When we invented an airplane, they invented anti-aircraft. This is a tit-for-tat thing, and there’s always a leap at some point. People created a star fortress and that stymied invasions until someone invented the cannon. Sometimes, defense beats offense and sometimes offense beats off defense, but it goes back and forth as you go.

    I don’t necessarily see anything that’s a semantic change in how organizations will defend, but I do believe, if you’re not paying attention, if you’re not innovating, if you’re not selecting vendors, if you’re buying software, if you’re not buying vendors who aren’t innovating, you’re going to have a very difficult time in the next five years because your adversaries absolutely are.

    Cory Minton: Yeah, and I would go maybe less digital and say we as leaders cannot underestimate and cannot undervalue investments that we make in our people and our business culture today that create space for the learning and the development and the skills attainment that Ryan talked about, being critical. Leaders must create space for technologists, for practitioners, for leaders to go learn and to go study and have a mantra of spending some large amount of time on a regular basis, on a cadence basis, investigating, whether that’s going and attending conferences and networking and starting to understand what’s happening in the industry or it’s partnering with a technology organization and doing more hands-on workshops and learnings or it’s going next, when you’ve got a consultant in, spending time with those folks in the office picking their brain on what are the skills that they’re paying you for? Why are you here? What is so unique about what you’re doing, and how do I learn more of that?

    It comes from leaders making that a priority. If we’re going to have an organization that has people, process and technology that’s actually going to be resilient, that’s going to be able to survive in this next epoch of technology innovation, it’s going to have to be led by the best intelligence, which is still human intelligence, and that requires investment, and I think you got to put a stake in the ground and make that a priority today.

    Ryan Kovar: I love that part because, going back to our previous discussions about leaders who often have 10, 20, 30 years of experience, none of it is relevant. When I started my career as CISO, my only half, well, first off, CISOs didn’t exist. Second, the CTO or the person in charge of technology at that company had to pay attention to desktops, printers and servers on-prem, and then 10 years ago, 15 years ago, it was laptops, desktops, printers on-prem, printers in the home offices, a data center that they co-located to in this little cloud thing that they’re trying out, and then five or 10 years ago, it was all those plus software as a service, and now it’s, okay, well, we have a CICD pipeline and we’re driving APIs and our entire business is dependent upon this one piece of open source software that’s maintained by three guys in a former Yugoslavian Republic and, if it has an issue, we’re going to be done.

    Also, we M&Aed a company out of China. Every time they develop a vulnerability, that has to go to the Chinese government along with us. It’s a lot more complex. For those who are just resting on their laurels of “I’ve been doing this for a while” and not up-leveling their own education and hiring people who are subject matter levels as direct lieutenants, you’re going to have a bad day.

    Mitch Ashley: Bringing it back to the thinking that got us here is not the thinking that will get us to the next place, I’m paraphrasing Einstein here, but it’s also our own thinking. That’s the value of bringing in perspectives like yours with your partners, your suppliers, companies like Splunk, professional services. There are a lot of ways. I think we’re at the same junction point where we were with the cloud. Where we’re like, “Here’s the cloud. Is it just somebody else’s computer or is it a new way of doing things?” But what is that new way?

    We’re kind of in that place today with AI. Okay. Right. We know we can do generative AI stuff, but what does it really mean? How do we leverage it, to your point, Ryan? You have to be a participant on the field to really figure that out. You don’t have to go spend crazy money on the latest thing just because that’s what was in the airline magazine, but it is about figuring out and learning what you can do with it and what’s possible learning by yourself, but also with others.

    Well, thank you to both of you, to Cory Minton and Ryan Kovar, for joining us, and thanks to the Splunk team for bringing us together to have this conversation. We hope it’s been really beneficial to everybody who’s listening in. I know it’s been great for me as well. Thank you.

  • ngrok Adds Load Balancing to Programmable Network Service

    ngrok Adds Load Balancing to Programmable Network Service

    ngrok today added a global server load balancing capability to its service for managing ingress via an application programming interface (API).

    Alan Shreve, ngrok CEO, said the ngrok Global Server Load Balancing (GSLB) service reduces latency and increases overall resiliency by automatically rerouting traffic across a global network of points-of-presence ngrok manages.

    The overall goal is to provide a higher level of abstraction for managing network operations (NetOps) that can be programmatically invoked. NetOps takes the place of what are essentially the equivalent of Assembly programming tools to provision networking equipment, he added. The long arc of computing is moving toward higher levels of abstraction to make IT capabilities more accessible, noted Shreve.

    That approach eliminates the need for IT teams to assign network operations teams to manage appliances in favor of a service that also provides a demilitarized zone (DMZ) via a service that accesses reverse proxies and firewalls, noted Shreve. The ngrok service ensures unauthorized requests are instantly blocked.

    The ngrok service provides a programmable approach to managing network services that makes it simpler for DevOps teams to invoke them. Instead of waiting for a network operations team to provision networking equipment, an API makes it possible to integrate the provisioning of networking services within the context of a larger DevOps workflow.

    One of the challenges that IT teams continue to encounter is that while virtual machines can be spun up in minutes, the provisioning of network services can require anywhere from minutes to days, depending on the age of the networking equipment. The ngrok service doesn’t eliminate the need for network professionals, but it does make it simpler for them to collaborate with application development and deployment teams, said Shreve.

    It’s not clear if NetOps teams will one day simply become an extension of a DevOps team, but as applications become more distributed, the need to minimize network latency is becoming more critical. A service like ngrok makes it simpler for DevOps teams to ensure the appropriate level of bandwidth is available so distributed application performance doesn’t slow to a crawl.

    Of course, the bigger challenge will be bridging the cultural divide that currently exists between many NetOps and DevOps teams. Both groups have their own distinct nomenclature for describing services that, in many cases, the other simply doesn’t comprehend. As distributed application development accelerates, each organization will need to determine how to demolish those silos so both teams can more easily achieve a common goal.

    In the meantime, IT leaders might to reconsider their strategy as networking services become increasingly programmable via APIs. Most modern networking equipment today is software-defined in the sense that network operations teams have access to graphical tools to provision networking equipment. However, as APIs become more accessible, the need for those graphical tools may not be nearly as high as it is today as networking becomes just another service to be managed as code.

  • Latest eBPF Advances Are Harbingers of Major Changes to IT

    Latest eBPF Advances Are Harbingers of Major Changes to IT

    The open source community advancing the extended Berkeley Packet Filter (eBPF) gathered at a virtual eBPF Summit today that featured a demonstration of eBPF being used in a Windows environment. eBPF enables applications to run within a sandbox in the Linux microkernel.

    Microsoft has signaled its intent to employ eBPF within Windows like the latest version of Linux already does today. That capability makes it possible to, for example, run networking, security, observability and storage software closer to the kernel to dramatically improve performance versus running that software in user space.

    Liz Rice, chief open source officer for Isovalent, a provider of networking software based on eBPF, said that while it’s clear eBPF will have a major impact on IT, the transition to eBPF won’t be a single event. Instead, it will be more of an evolutionary process that will occur as organizations upgrade their operating systems and the various tools and applications they employ.

    The eBPF Foundation, an arm of the Linux Foundation, was formed to expand the use of eBPF across all operating systems. Members of the eBPF Foundation include Facebook, Google, Isovalent, Microsoft and Netflix.

    In effect, eBPF changes the way operating systems are designed. It bridges the boundary between kernel and user space by enabling developers to combine and apply logic across multiple subsystems that were historically completely independent. That approach enables, for example, a security tool to scale to the point where it can identify threats at much higher levels of throughput at a time when brute-force cyberattacks are overwhelming existing cybersecurity tools.

    Similarly, networking, storage and observability tools should be able to run at much higher levels of scale. That’s especially critical for DevOps teams embracing observability platforms that need to dynamically process massive amounts of data in near-real-time.

    Less clear is the degree to which eBPF might one day drive convergence across distinct categories once multiple tools and applications run at the microkernel level.

    IT organizations would be well-advised to ask their vendors when they plan to support eBPF as they plan their upgrade cycles. It’s not likely IT teams are going to want to upgrade every tool and application that can benefit from eBPF all at once. In fact, updates to these applications that don’t support eBPF might even be postponed in the knowledge that a much larger and arguably more critical update is on the immediate horizon.

    In the meantime, as the number of organizations running the latest versions of Linux continues to increase, more hands-on experience with eBPF will be gained. IT teams may not need to concern themselves with what is occurring in the microkernel of the operating systems, but they should at least understand how eBPF ultimately helps reduce the total cost of running IT and how it makes the Linux microkernel more programmable. Those IT teams, however, will soon discover how stark a difference in performance there will be.

  • Nitric Adds Support for Additional Languages to IaC Alternative

    Nitric Adds Support for Additional Languages to IaC Alternative

    Nitric today announced it has added support for the Go and Kotlin programming languages to its application framework. The framework automatically creates code to provision infrastructure based on the way the code used to write an application has been developed.

    Rak Siva, vice president of engineering for Nitric, said that approach eliminates the need to rely on developers using infrastructure-as-code (IaC) tools to configure platforms.

    Nitric previously supported JavaScript, Python and C# along with frameworks such as Express, Fastify and Nest.js to enable infrastructure to be automatically configured based on an intent expressed in the codebase enabled by a software development kit (SDK). In effect, Nitric infers how to optimally configure infrastructure. DevOps teams can then choose to either rely on the configurations created by the Nitric framework or customize it to add support for their own policies, said Siva.

    While IaC tools have played a major role in streamlining the application development and deployment process, they are also the root cause of many cloud security incidents. Developers that use these tools typically have limited cybersecurity expertise, so the probability that cloud infrastructure will be misconfigured in a way that cybercriminals can exploit is high.

    Nitric is attempting to take the concept of managing infrastructure-as-code to the next logical level by automating the process so it can be consistently applied and managed, said Siva. That approach provides the added benefit of improving the overall developer experience by eliminating a task that most developers view as a means to a larger end, he added.

    It’s not clear how much of the cloud infrastructure that DevOps teams are relying on to build and deploy applications is misconfigured, but given the varying levels of skill among developers, there are very few organizations that have not encountered a cloud security issue. The challenge is not so much the security of the underlying platforms as it is the shared responsibility model relied on to secure these platforms. Cloud security platform providers assume that organizations that programmatically invoke infrastructure have the expertise required to securely provision it. However, very few organizations can achieve that goal 100% of the time, which results in organization having to make considerable investments in additional tools and platforms to monitor and enforce cloud security policies.

    As more organizations focus their efforts on securing software supply chains, many of them will be revisiting how cloud infrastructure is provisioned. In fact, one of the reasons platform engineering is gaining traction as a DevOps methodology is to improve the developer experience and strengthen application security.

    Cultural change, of course, is difficult. IaC tools such as Terraform are deeply embedded within many DevOps workflows, so any alternative approach to provisioning infrastructure will require time to take hold. However, as regulations that hold organizations more accountable for application security become increasingly more stringent, it’s now a question of when rather than if change will come to how cloud infrastructure is provisioned.

  • JFrog Extends DevOps Reach to Manage and Secure AI Models

    JFrog Extends DevOps Reach to Manage and Secure AI Models

    At its swampUP 2023 conference, JFrog today added additional security capabilities to its continuous integration/continuous development (CI/CD) environment as well as making it possible to manage machine learning models within the context of a DevOps workflow.

    The latest edition of the JFrog Software Supply Chain Platform adds an integrated static application security testing tool (SAST) to scan source code alongside the existing JFrog Xray tool for scanning binaries.

    JFrog has also added an Open-Source Software (OSS) Catalog to the JFrog Curation service to make it simpler for DevOps teams to discover specific packages that have been determined to be secure. JFrog is also adding Release Lifecycle Management (RLM) capabilities that enable aggregating software artifacts into an immutable software package that serves as the single source of truth as multiple iterations of an application are developed. JFrog RLM uses anti-tampering systems, compliance checks and evidence capture to collect data and insights to ensure the immutability of signed binaries.

    Finally, ML Model Management capabilities, available in beta, are being added to the platform to make it possible to store AI models, run AI models from Hugging Face, a provider of open source AI models, as a proxy in cache via an application programming interface (API) and apply DevSecOps best practices to govern, secure and ensure licensing compliance.

    JFrog CTO Yoav Landman said that as DevOps teams become more familiar with AI models, it’s becoming increasingly clear they are managed like any other software artifact incorporated into a DevOps pipeline. He noted that the approach provides the added benefit of helping to ensure AI models have not been poisoned by malicious data and prompts that would deliberately cause a hallucination.

    Most AI models today are created by data scientists using machine learning operations (MLOps) best practices that each organization defines. In some instances, those processes include what is known as a feature store that enables data scientists to track which components are being used at any given time. However, once a model is completed, it needs to be integrated into an application—a process that, from a DevOps perspective, is best facilitated by storing an AI model in a Git repository alongside other components that need to be managed.

    It’s still early days as far as the convergence of DevOps and MLOps practices are concerned. Going forward, as it becomes apparent almost every application will be invoke some type of AI model, the need to integrate these workflows is becoming a more pressing concern.

    It’s now only a matter of time before AI models become a pervasive element of the software development life cycle as they are continuously retrained and updated. It will generally fall to DevOps teams to manage the deployment and updating of those AI models in collaboration with the data science teams that train them. The immediate challenge, of course, will be to bridge the significant cultural divide between two very different types of IT professionals.

  • Operational Intelligence: AI-Powered SRE Measurements and Observability

    Operational Intelligence: AI-Powered SRE Measurements and Observability

    In my recent blog, Revolutionizing the Nine Pillars of SRE with AI-Engineered Tools, I indicated AI can help analyze vast amounts of data from monitoring and observability systems, identifying patterns and correlations that may be difficult for humans to detect. In this blog, I explain, in more detail, how AI-engineered tools can be used to improve the site reliability engineering (SRE) measurements and observability pillar of practice.

    AI Use Cases for SRE Measurements and Observability

    Making Applications Observable: AI tools like Dynatrace can provide real-time observability of applications, identifying trends and anomalies that might not be evident from raw data. Challenges here include the potential for overwhelming amounts of data, which can be mitigated by AI’s capacity to analyze and prioritize this data effectively.

    Making Infrastructure Observable: Tools like New Relic and Datadog use AI to provide insights into the health and performance of infrastructure. The challenge here can be the complexity of modern, cloud-based infrastructures, but AI tools can model these systems and analyze their behavior to provide actionable insights.

    Monitoring Applications in Production: AI can be used to monitor applications in real-time, identifying potential issues before they impact users. Tools like Splunk use AI to monitor application logs, detect anomalies and automatically alert relevant teams. The key challenge here is filtering the signal from the noise, which AI can help by learning which anomalies are significant.

    Monitoring Production Infrastructure: AI can monitor infrastructure health and performance, predict maintenance needs and identify potential bottlenecks. Tools like Zenoss use machine learning to predict infrastructure failures before they occur. One of the main challenges here is the diversity of infrastructure components, but AI can handle multiple data sources and types and integrate this data into a comprehensive overview.

    Tracing and Logging: AI can help manage the vast amounts of data generated by modern logging and tracing systems, identifying patterns and correlations that might be difficult for humans to detect. Tools like Logz.io use AI to analyze log data, detect anomalies and provide actionable insights. One challenge can be the sheer volume of log data, but AI can handle this by focusing on patterns and anomalies and learning over time which issues are significant.

    Production Dashboard Design: AI can help design more effective dashboards by identifying which metrics are most relevant and should be highlighted. Tools like Grafana use machine learning to optimize dashboard layouts based on usage data. The challenge here is determining what information is most relevant, but AI can learn this from usage patterns and user feedback.

    Performance Forecasting: AI can help with predicting system performance based on current and historical data, helping to prevent performance issues before they arise. Tools like Virtana use AI to provide accurate forecasts of system performance, helping teams to plan for future capacity needs. The challenge here is the variability of performance data, but AI can handle this by identifying patterns and trends in the data.

    Automated Root Cause Analysis: AI can be instrumental in quickly identifying the root cause of issues in complex, distributed systems. Tools like RunWhen and Moogsoft use AI to correlate events across different systems and tools, significantly reducing the time to identify the root cause. The challenge here is the complexity of modern systems, but AI can handle this by analyzing data from multiple sources and identifying correlations that might not be evident to humans.

    RoadMap for SREs to Transform Measurements and Observability With AI

    Here is a practical roadmap that organizations can use to transition to AI tools for SRE measurements and observability.

    Understand the Current State: First, an organization needs to fully comprehend its existing monitoring and observability practices. This includes understanding what tools are currently in use, what metrics are being monitored and where the gaps in visibility exist.

    Define Goals and Objectives: The next step is to define what the organization hopes to achieve by implementing AI in its monitoring and observability practices. This could include goals like improving system performance, reducing downtime or predicting system failures before they occur.

    Identify the Right Tools: Evaluate different AI tools to help achieve these goals. This includes looking at AI-powered monitoring tools like Dynatrace, New Relic and Datadog, log analysis tools like Logz.io and Splunk and AI-driven observability platforms like Moogsoft and Zenoss.

    Test and Iterate: Implement the chosen tools on a small scale initially, using a test environment if possible. Monitor the results and adjust your approach as necessary. This step might involve tweaking the AI models, adjusting the data inputs or even trying a different tool if the initial choice doesn’t provide the expected results.

    Training and Upskilling: Ensure your teams are well-versed in using the chosen AI tools. This could include formal training sessions, workshops, or even bringing in external experts to provide guidance. This will help your team to effectively use these tools and get the maximum benefit from them.

    Full-Scale Deployment: Once testing has been successful and the team is comfortable with the tools, deploy them across your production environment. Continue to monitor their performance and make adjustments as necessary.

    Continual Evaluation and Adjustment: AI tools learn and improve over time. It’s important to regularly evaluate their performance and make adjustments as needed. This could involve feeding them new data types, adjusting their models or retraining them based on the latest performance data.

    Summary

    Harnessing the power of AI to revolutionize your SRE practices can propel your organization to new heights. AI tools can significantly enhance the measurements and observability pillar of SRE, providing unparalleled insights into application and infrastructure performance, enabling more efficient monitoring practices and ultimately driving improved system reliability and performance. These tools, such as Dynatrace, New Relic, Datadog, Logz.io and Moogsoft, can transform how your organization perceives and interacts with its technological environment, providing crucial visibility and actionable intelligence.

    Imagine a world where your monitoring systems not only alert you about a problem but also predict potential system failures before they occur and even recommend preventive actions. With the implementation of AI in your SRE practices, this can become a reality. By following a well-defined roadmap that begins with understanding your current state, defining your objectives, selecting the right AI tools, iteratively testing and upskilling your teams, you can pave the way for a future where improved system performance and reduced downtime are the norms. AI-powered SRE practices today can unlock a new era of operational efficiency and system reliability.

  • Embracing Chaos With AI: Reinventing SRE’s Anti-Fragility Practices

    Embracing Chaos With AI: Reinventing SRE’s Anti-Fragility Practices

    In my recent blog, Revolutionizing the Nine Pillars of SRE with AI-Engineered Tools, I indicated that AI can contribute to the development of more resilient systems by analyzing system behavior under stress and identifying vulnerabilities. In this blog, I explain, in more detail, how AI-engineered tools can improve the SRE anti-fragility pillar of practice.

    AI Use Cases for SRE Anti-Fragility

    Fire Drills: AI can assist in creating more effective and dynamic fire drills by predicting potential failures based on patterns and anomalies in data. Tools like Gremlin and ChaosIQ can be used to design, schedule and manage chaos experiments. However, resistance from team members may be a challenge due to the potential for disruption. To overcome this, it’s important to communicate the benefits and ensure there are safeguards in place to prevent unnecessary disruption.

    Chaos Engineering – Infrastructure: By introducing controlled failures in the infrastructure, AI can learn and identify weaknesses that would not be apparent under normal conditions. Tools like Chaos Monkey and Netflix’s Simian Army can be helpful here. A significant challenge could be to deal with the resulting chaos and anomalies. Adequate training and support are required to ensure the teams can handle the chaos experiments and learn from them.

    Chaos Engineering – Applications: AI can introduce failures at the application level to understand how the system behaves and recovers. Tools like Litmus can help conduct these experiments. Challenges could include resistance from developers and potential impacts on the user experience. Having a well-defined scope and process and conducting these experiments in pre-production environments can help mitigate these risks.

    Chaos Engineering – Security of Production Environments: By simulating attacks and breaches, AI can help understand vulnerabilities in the security architecture. Tools like Game of Hacks and AttackIQ can simulate different types of attacks. The main challenge could be the potential exposure of vulnerabilities, making it essential to have robust security measures in place and conduct these exercises in a controlled environment.

    Identifying Reliability Weaknesses in Production Logs: AI can analyze vast amounts of log data to identify patterns or anomalies that may indicate reliability issues. Tools like Splunk and Logz.io can help here. However, the sheer volume and complexity of log data can pose a challenge. To overcome this, robust data management strategies and powerful AI-driven analytics tools should be deployed.

    Predicting System Failures: AI can analyze system usage and performance data to predict potential system failures before they occur. Tools like Dynatrace and New Relic can be useful in this scenario. A major challenge here is the accuracy of predictions, which can be addressed by training the AI model with quality data and continuously refining it.

    Automated Rollbacks: AI can help in the automated rollback of deployments if chaos experiments or real-time monitoring reveal major issues. Spinnaker is one tool that can help in this aspect. The challenge is to ensure accurate triggers for rollbacks without causing unnecessary disruption. This can be mitigated by proper testing and calibration of the AI system.

    RoadMap for SREs to Transform Incident Response to Using AI

    Here is a practical roadmap for organizations that want to transform to using AI tools for SRE anti-fragility.

    Define Anti-Fragility Goals: Define what you want to achieve with your AI-driven anti-fragility measures. Is it to reduce system downtime, improve security or enhance overall system resilience? Defining these goals upfront will help guide the selection and implementation of AI tools.

    Identify Use Cases: As per the use cases listed above, identify those that are most relevant to your organization and your specific infrastructure and applications. Prioritize them based on your anti-fragility goals.

    Evaluate AI Tools: Assess various AI tools available in the market, such as Gremlin, ChaosIQ, Chaos Monkey, Litmus, Game of Hacks, AttackIQ, Splunk, Logz.io, Dynatrace, New Relic and Spinnaker. Look for those that align best with your defined use cases and overall goals.

    Pilot and Learn: Select one or two tools and use cases to start. Run pilot programs to understand the effectiveness of these tools and to gain valuable insights. This step also includes training your team to use these tools effectively.

    Integrate with SRE Practices: Once you have gained sufficient confidence in the chosen tools, integrate them with your existing SRE practices. Ensure that these tools can interact with your current monitoring, incident management and deployment tools.

    Iterate and Improve: Continually measure the impact of these AI tools on your systems’ anti-fragility. Use this feedback to refine your approach, re-adjust your goals, and explore other use cases or tools.

    Expand Scope: As your team becomes more comfortable with AI tools and chaos engineering practices, you can gradually expand the scope of AI-driven anti-fragility measures across more applications, systems and use cases.

    Summary

    The rising complexity of digital systems and increasing customer expectations necessitate a shift towards a more resilient and adaptive approach to system reliability. SRE’s anti-fragility pillar embraces this challenge, allowing systems not just to resist stress but to thrive under it. The incorporation of AI tools in the realm of anti-fragility brings this dynamic resilience to a new level, enhancing capabilities from conducting fire drills and chaos engineering to identifying reliability weaknesses in production logs. With AI-driven tools like Gremlin, ChaosIQ, and Splunk, SREs can effectively simulate failures, analyze intricate system patterns, and proactively fix potential issues to build stronger, more reliable systems.

    Embarking on a journey to harness AI for anti-fragility is an exciting endeavor that promises profound benefits. An achievable roadmap begins with setting clear goals, selecting relevant use cases, evaluating suitable AI tools, and running pilot programs. The key is to integrate these tools into existing SRE practices gradually and iteratively, continually assessing their impact and refining the approach based on the insights gained. This journey is a continuous process of learning, adapting, and expanding, fostering an environment where systems can thrive amidst chaos and uncertainty. The future of SRE is here, and it’s more robust and resilient than ever.

  • CDF Report: Steady Growth in Number of Jenkins Jobs Run

    CDF Report: Steady Growth in Number of Jenkins Jobs Run

    The Jenkins project that operates under the auspices of The Continuous Delivery Foundation (CDF) reported this week that over a two-year period, usage of pipelines grew 79%, with total workloads defined in the continuous integration/continuous delivery (CI/CD) platform growing 45% over the same period.

    Based on analysis conducted from June 2021 to June 2023, the report found the number of jobs defined increased from 27,105,176 jobs to 48,625,398 jobs per month. The number of all jobs defined on Jenkins increased from 50,785,205 to 73,746,418 jobs per month (45%).

    Sacha Labourey, a member of the governing board for the CDF and chief strategy officer at CloudBees, a provider of commercially supported instances of the open source Jenkins CI/CD platform, said it’s clear that more applications are being constructed by more organizations than ever using DevOps workflows.

    At present, the CDF reported there are now more than 600 active contributors to the Jenkins project. That level of support has made it possible for Jenkins to become a standard CI/CD platform. DevOps teams know there is a substantial open source community they can count on for support, said Labourey.

    The challenge now is how to centrally observe DevOps workflows to accelerate application development while at the same time giving developers more freedom to innovate. Platform engineering principles represent an opportunity to achieve that goal, as long as developers still have an opportunity to innovate and collaboratively address bottlenecks in DevOps workflows as they occur.

    The platform engineering team should be observing processes and proactively making recommendations and define a set of guardrails that ensure DevSecOps best practices are followed, he added.

    Otherwise, application developers are likely to view platform engineering as another effort by centralized IT teams to take power away from the developer, noted Labourey.

    Striking that balance is especially critical as the number of cloud-native applications being built and deployed alongside classic monolithic applications has significantly increased, he added. DevOps teams need to be able to centralize observability of what are two distinct styles of application development that tend to occur at a significantly different pace from one another, Labourey said.

    Hopefully, advances in artificial intelligence (AI) will make it easier for DevOps teams to strike the right balance between management and innovation. In the meantime, however, as the pace of application development continues to accelerate, each DevOps team will need to determine how best to evolve their existing workflows. That’s especially critical at a time when economic headwinds are requiring organizations to find ways to increase existing developer productivity rather than hire or contract additional developers.

    In the meantime, it’s apparent that DevOps, as a methodology for building and deploying applications faster, is here to stay. The next major challenge is finding a way to achieve that goal at levels of scale that require automation everywhere from the network edge to the cloud.

  • Apica Acquires LOGIQ.AI to Unify Observability

    Apica Acquires LOGIQ.AI to Unify Observability

    Apica, a provider of a synthetic monitoring and observability platform, has just raised an additional $10 million in funding and has agreed to acquire LOGIQ.AI. LOGIQ.AI is a provider of a data fabric infused with machine learning algorithms.

    Jason Haworth, chief product officer for Apica, said the data fabric developed by LOGIQ.AI will be incorporated into the Apica Ascent platform in the third quarter. The LOGIQ.AI addition will apply artificial intelligence (AI) in a way that both optimizes and enriches data collected by the company’s observability platform.

    At the core of the Apica Ascent platform is an indexing engine built on top of a Kubernetes platform that aggregates data such as logs, traces and network packets from multiple sources. As part of that process, the platform reduces storage costs by trimming excess data that can be stored in a data lake it provides or a third-party data lake a customer prefers.

    At the same time, Apica also enables DevOps teams to both normalize data collected from multiple sources and enrich it to add additional context, said Haworth.

    The overall goal is to provide a unified view of all IT data to accelerate faster root cause analysis and reduce the total cost of collecting the data required, he added.

    Apica is making a case for an observability platform that goes beyond collecting data from DevOps platforms. As IT environments become more complex, IT teams need to be able to correlate data collected from a wide range of data sources to accurately determine the root cause of an issue, said Haworth.

    The challenge is managing the massive amount of data that needs to be continuously collected, stored and analyzed to achieve that goal, he added. The acquisition of LOGIQ.AI will enable Apica to embed a data fabric that employs machine learning algorithms and other data science techniques to make it feasible to observe complex IT environments at scale, noted Haworth.

    It’s still early days as far as observability is concerned, but it’s clear that IT teams need to move beyond monitoring a pre-determined set of metrics to manage complex and dynamic application environments. The rate of change to application environments as updates are made requires an ability to query data to determine the root cause of issues that could be caused by any number of dependencies.

    The issue is that even when IT teams have access to an observability platform, they may not have the knowledge and expertise required to craft the queries needed to determine the root cause of an issue. Machine learning algorithms will play a major role in enabling IT teams to proactively discover issues that they can investigate using the suggested queries.

    It’s only a matter of time before most IT organizations unify observability data within a single platform. That will make it simpler for them to collaborate across teams and should result in fewer disruptions as IT issues are discovered more quickly. After all, the best kind of IT incident is the one that never happened or that was too trivial for anyone outside of IT to even notice.

  • Strategic Roadmap for SREs: Software Deployments With AI

    Strategic Roadmap for SREs: Software Deployments With AI

    In my recent blog Revolutionizing the Nine Pillars of SRE with AI-Engineered Tools, I indicated that SREs can use AI and machine learning to optimize deployment strategies and minimize risks. In this blog, I explain, in more detail, how AI-engineered tools can be used to improve the pillar that I call deployments.

    SRE Use Cases for AI-Assisted Deployments

    Here are use cases for SREs to use AI to assist with software deployments.

    Pre-Deployment Preparation Activities: AI can analyze the deployment package to assess dependencies, predict potential conflicts and evaluate the readiness of the package for deployment. AI can also assist in configuration management, ensuring the correct configurations are applied based on the deployment environment. Tools like Harness Continuous Delivery use machine learning to predict the success of deployments based on previous patterns and trends.

    Deployment Decision Process Automation: AI can use data from past deployments, current system status and predictive analytics to decide the best time and method for deployment. It can assess the risk associated with the deployment and help in decision-making. Tools like IBM’s UrbanCode Deploy leverage AI to optimize deployment processes based on historical data.

    Deployment Orchestration Using Blue-Green: AI can help orchestrate Blue-Green deployments by managing the routing rules, monitoring the performance of the new environment and deciding when to switch traffic. Tools like Spinnaker, integrated with AI capabilities, can automate this process more effectively.

    Deployment Orchestration Using Canary: In Canary deployments, AI can analyze the performance of the canary instances in real-time, compare it with the baseline and make data-driven decisions about whether to proceed with the full deployment. Canary deployments can benefit from AI tools like Flagger, which automatically advance or roll back deployments based on key metrics.

    Deployment Orchestration Using Feature-Flag Rollouts: AI can help manage complex feature-flag rollouts, analyzing user behavior and system performance to optimize the rollout strategy. Tools like LaunchDarkly provide feature flag management and integrate machine learning for more sophisticated flag controls.

    In-Production Testing: AI can enable more effective testing in production by identifying unusual system behavior, predicting potential issues and recommending areas for focused testing. Chaos engineering tools like Gremlin integrate AI to automate the identification of system weaknesses.

    Decision-Making and Orchestration of Rollbacks: AI can monitor the health of the system post-deployment, making data-driven decisions about when a rollback is necessary. AI can also help orchestrate the rollback process to minimize disruption. Harness Continuous Delivery provides AI-driven rollback capabilities.

    Monitoring Performance of Deployed Artifacts: AI can analyze the performance of deployed artifacts, compare it with baseline performance, and predict potential performance issues. Tools like Dynatrace use AI to provide real-time performance monitoring and anomaly detection.

    RoadMap for SREs to Transform Deployments to Using AI

    Here is a practical roadmap for SREs to transition to using AI tools for deployment.

    Needs Assessment: Begin by understanding the current state of your deployment processes and identify areas where AI can bring the most benefits. This could be in deployment decision automation, deployment orchestration, in-production testing, rollback decision-making or performance monitoring.
    Tool Selection: Research AI-enabled tools that are available in the market that could fulfill your specific needs identified in step one. Some of these tools could include Harness Continuous Delivery, IBM’s UrbanCode Deploy, Spinnaker, Flagger, LaunchDarkly or Gremlin. Remember to evaluate these tools based on their features, scalability, compatibility with your existing systems and cost.
    Pilot Program: Implement a pilot program with the selected AI tools on a small scale or in a non-critical environment. This will allow you to evaluate the tool’s effectiveness, understand its strengths and weaknesses and refine its configuration and usage in your environment.
    Training: Ensure your SRE team is adequately trained to use the selected AI tools. This could involve formal training sessions, self-guided learning resources or hands-on practice in a sandbox environment. Remember, the tool is only as good as the person using it.
    Gradual Rollout: Once you’re confident in the tool’s effectiveness and your team’s ability to use it, begin a phased rollout of the tool in your production environment. Monitor closely and be ready to adjust as necessary.
    Continuous Improvement: Regularly review the tool’s effectiveness and look for ways to further optimize its usage. Stay informed about updates to the tool and industry best practices, and continually adjust your processes to stay ahead of the curve.

    Summary

    AI’s role in enhancing SRE practices, specifically software deployments, is revolutionizing the efficiency and reliability of deployment processes. From pre-deployment preparation to monitoring the performance of deployed artifacts, AI tools can automate complex tasks, make more informed decisions and improve the speed and reliability of deployments. Whether you’re using Blue-Green, Canary or feature-flag rollouts or you’re navigating decision-making for rollbacks, AI tools like Harness Continuous Delivery, UrbanCode Deploy, Spinnaker, Flagger, LaunchDarkly and Gremlin can be game-changers.

    However, transitioning to AI-driven deployments isn’t an overnight switch; it’s a journey that requires strategic planning and execution. Starting with a thorough needs assessment, choosing the right tools and implementing them via a pilot program is the first leg of this journey. Training your SRE team on these tools and rolling them out gradually in your production environment will ensure a smooth transition. And the journey doesn’t end there. Regular tool effectiveness reviews and staying up-to-date with the latest in the tool’s features and best industry practices will ensure that you’re getting the most out of AI in your deployment processes. Embrace the AI revolution in SRE to stay ahead of the curve in the fast-paced world of continuous software deployments.