Tag: devops culture

  • Survey Surfaces Major Observability Challenges

    Survey Surfaces Major Observability Challenges

    A recent survey found the biggest obstacles to observability were the pace of technology change, IT blind spots and the complexity of modern applications. The survey polled 300 IT professionals to identify the biggest challenges they faced in achieving observability. The accelerating pace of technology change (72%) topped the list, followed by IT blind spots and the growing complexity of modern applications (58% each), and lack of funding (52%).

    The survey, conducted by SolarWinds, also found the typical enterprise suffered nine brownouts or outages every month, each of which lasted around twelve hours. The full cost of outages and brownouts averaged $13.7M annually for these enterprises, according to the SolarWinds report.

    Jeff Stewart, field CTO and vice president, global solutions engineering at SolarWinds, said the survey makes it clear that a lack of observability is having a material impact on businesses. Organizations are now moving beyond monitoring predefined metrics to embrace observability platforms that make it simpler to investigate the root cause of issues that adversely impact businesses; these businesses are more dependent on IT than ever, he added.

    In fact, the primary reason organizations are investing in observability is to improve their customer experience (96%), enable faster innovation (71%), reduce time spent solving (71%) and detecting (60%) issues and improve operational efficiency (55%), the survey found.

    While it’s still early days as far as achieving observability is concerned, a lot of progress is starting to be made as organizations realize that complex IT environments require a different management approach. IT teams can no longer afford to navigate multiple monitoring tools in what often turns out to be a vain effort to prevent brownouts and downtime, noted Stewart.

    In addition, many IT teams are becoming increasingly aware that the telemetry data collected via an observability platform provides the foundation for infusing IT operations with artificial intelligence (AI) capabilities, he added. The ultimate Holy Grail is to one day achieve autonomous operations, noted Stewart.

    There is, of course, no shortage of observability platforms, but Stewart contends that providers of monitoring tools such as SolarWinds are in the best position to help IT teams make the observability transition. Rather than ripping and replacing a monitoring platform, a more seamless transition that results in less disruption is preferable, he noted.

    Regardless of approach, observability platforms that enable IT teams to launch queries across complex distributed computing environments are becoming a necessity. Distributed computing environments often have more dependencies than IT teams can manually track. It’s only a matter of time before the most talented DevOps professionals start to migrate to organizations that provide them with the management tools and platforms required to succeed. Otherwise, the stress of managing a complex IT environment will only result in higher rates of burnout among DevOps professionals.

    It may be a while before observability platforms are pervasively employed across IT environments, but as it becomes clear that brownouts and downtimes result in a decline in revenue, business leaders will understand what’s required to maximize profits. In many cases, the answer will be an observability platform that surfaces the insights needed to proactively manage modern IT environments.

  • Splunk: Creating a Resilient and Dynamic Organization

    Splunk: Creating a Resilient and Dynamic Organization

    Mitch Ashley: I have the pleasure of being joined by Ryan Kovar, Ryan is a distinguished tech security technologist and leader of SURGe with Splunk, and Cory Minton who’s field CTO for The Americas with Splunk. Welcome, guys.

    Ryan Kovar: Thank you.

    Mitch Ashley: Good to be chatting with you both. A topic that’s very top of mind for CISOs or IT leaders, of course, is how do we create a resilient and dynamic organization that can keep pace with the business and the change that we see in the technology landscape, whether it’s our own infrastructure, own application portfolio, cloud, et cetera, all of those things, but also, of course, the tax services that are evolving and changing as what the bad guys, the threat actors, are doing. We’ve also not only got to have a great and resilient technology stack, but organizationally, process-wise, all of those things have to fit together into a cohesive strategy.

    We’re here to talk about that, thinking about it as an IT leader, whether you’re in security or IT or combination of both, and discussing your all expertise experience, but also you talk to a lot of customers as leaders in Splunk and the kind of things that you do. Cory, it’d be great to have you kick things off. Maybe we should start with if you want to start with the people domain or maybe you want to set it up a little bit differently. I’d love to hear your initial thoughts on this.

    Cory Minton: Yeah. No people’s perfect. I think the people process technology lens on tackling any sort of problem for leaders today is an appropriate framework. I’m happy to talk about the people portion of building great cybersecurity and IT organizations that, like you said, deliver resilience.

    Ryan Kovar: For me, at least when we think about this, you said IT leaders, and I think one of the big changes I’ve seen across for cyber for CISOs is it’s no longer IT or security. It is business leaders. When we started talking about people, that to me starts resonating because that is a cognitive change in how CISOs think of themselves and they think of their value, which we often get stuck in the technology part of people process and technology because a lot of us are technologists at heart, but I think the big change that Cory and I have seen when we start talking to CISOs and CIOs is this convergence of skills and the need to support the business differently.

    Cory Minton: Yeah, and it’s an interesting sort of people market, too. I think leaders have to think about the fact that, yeah, in technology, there has been some turmoil in some of the big tech companies, but there’s still a lot of really great talent out there. I think that there’s choice. With unemployment so low, and there’s, what, like 16 million jobs unfilled currently in the US and a lot of those being in the tech sector, people have choice and they have a place where they can go.

    I think, leaders, if they’re going to build a talent pipeline and build a great organization, they have to find folks where they are and bring them in and connect them to a value, a mission and a vision that they get excited about. Certainly, securing digital services and protecting against bad actors is exciting in and of itself for many folks, but actually connecting those people to how it affects the outcomes that matter to the business and that whole value creation process, I think, is a real critical skill for folks to understand today as they, again, are out communicating and interacting in the network, building networks of talent pools. I think that clear vision of why it matters and why the work that somebody would do with you matters is critically important.

    Ryan Kovar: That reminds me of the cliche, of course, that people don’t quit jobs, they quit leaders. When you flip that around, one of my mentors, Susan St. Ledger, told me, you can evaluate the success of a leader by how many people follow them to another company. I know right now, in this market, what I’m finding is people are staying longer because they like the people they work for or they like their leadership team, they like the culture. The money, even if they’re not making as much as they were last year because of inflation, the people I know who stuck around longest are the people happy with the people that they work for and work with.

    Mitch Ashley: It’s interesting when we talk about people. I’ve worked in part of my career where the sort of higher the purple unicorn that doesn’t exist, that got eight skills, that needs 20 years of experience for things that have only been around for three. We set our subs up to hire these phenomenal people of which there may only be a few in the world that are like that, but it’s about growing our people, but also growing our organizations, developing them. It’s not just hiring people. It’s building a team. It’s building an organization that’s got the right talent at the right time at the right place to match with the business needs, and that changes. That means you can’t always hire for it. You’re partnering for it. You’re working with companies like Splunk to bring in skills and expertise maybe you need for the moment, you need for a project, you need for strategy, all of those things. Love to hear your perspective about broaden what we think about people and how we incorporate that part of it into our strategy.

    Cory Minton: Yeah. I’ll say, top to bottom, start at the executive level down to practitioners. Everybody has a partner. Everybody has a consulting partner typically that they bring in to, like you said, fill those gaps where it’s a new skill, it’s a new technology, it’s a new capability that they’re trying to deploy for the reasons that they’ve chosen to do so. They’re valuable, but you may not have those skills internally. Training takes time and, oftentimes, just finding that partner that can help you solve that particular problem is incredibly important. I think, as the security landscape and, frankly, the tools being used to deliver and develop the latest digital services continues to get more complex, expecting to hire that skillset completely, probably not realistic especially as digital transformation objectives have an ebb and a flow of momentum and the amount of work being done on particular projects.

    Yeah, I see it as one of the top areas where, when I’m talking to CIOs and CTOs and CISOs, when they think about, “Hey, we want to deploy this new capability within Splunk,” or, “We’re, frankly, trying to figure out how to integrate better with our observability and IT teams to derive more value from the tools we’ve already purchased,” it’s oftentimes that people conversation of, “How do I actually leverage your talent to show us what good looks like and bring the experience from other companies similar to us that have already been down this path to, call it the cheat code, how do I do the thing that you guys already know how to do? How do I buy that capability?” Oftentimes, it’s through people.

    Ryan Kovar: My experience, I’ll go the other side of the halo and horns there down to just the nitty-gritty hiring. Recently building out my team, I intentionally carved out three slots for entry level, early career folks into our cybersecurity team, which is pretty rare for Splunk and pretty rare for a lot of security research teams, but the way I did that was really rewriting the job description to be embracive of second-career people. I actually really dislike the term “there’s a problem with the pipeline for hiring”. I think that’s completely false. The problem is not in the pipeline. The problem is we have a valve on incredible talent and gate-keeping who actually gets into the pipeline. We do see a problem with the pipeline when I start looking for folks who are underrepresented in cyber especially 15, 20 years down.

    What I don’t see is a problem of, basically, the reservoir. The reservoir is healthy. The reservoir is huge, but people either don’t feel like they’re welcome or they self-select out especially of cybersecurity. For me, a lot of the people problems that we have, if we can start working on the problems of today, future us, as a security leader, will be very happy if present me can help unlock that reservoir by turning open the valve by creating job descriptions and roles that are less around years of experience and understanding technologies that I can Google or generative AI my way through today.

    I think there’s just a lot of flexibility especially for folks who have critical thinking and communication skills. I can teach you TCP/IP. I can’t teach you how to communicate. I can’t teach you how to synthesize information. I need you to walk in the door with that. Something for me around people and these hiring gaps that we see is really about being more embracive of nontraditional cybersecurity and IT roles and then also facilitating them in ways they can succeed.

    Mitch Ashley: Well, and it’s also about hiring for what we need today, but also where the ball is going to be downfield when the ball lands. Part of that is hiring people who have demonstrated their ability to learn, adapt. Yesterday, they were TCP/IP expert. Today, They’re the security threat landscape expert. How did they do that? How did they get there? They learned it. It may have been on the job, but there’s also a lot of self-motivation and just skill in that learning, and that repetitive learning skill is I think something you can harness or leverage to accelerate their career as well as what you need.

    Cory Minton: Absolutely. It’s one of those things that, always, some folks get turned off. As Ryan said, sometimes there’s a valve, and one of those valves is like, “You must have a college education.” While I don’t necessarily believe that a college education is always the right thing, I think what it proves if you have one is that, as you said Mitch, you’re able to go through a structured learning process in an organization that’s institutional, understand dynamics and things and achieve an objective which was set before you that had some measurable outcomes that you had to deliver, which I think is something true for all of us in a corporate responsibility job as we have to operate in an institutional environment. We have to consistently learn new things. We have to interact with people around us. I think college is a good measure, but, as Ryan said, oftentimes, a second career is maybe even a more powerful measure of somebody who’s done that successfully.

    Like Ryan said, the technical skills are learnable and, frankly, the fun part is some of the technical skills are actually getting obfuscated by advancements in technology. As we think about things like, Ryan, you mentioned, generative AI, we can make the joke, but, candidly, wouldn’t you rather hire somebody who’s got incredible cybersecurity skills and understands the landscape more so than somebody who’s just really good at crafting queries because then, if you have somebody that has that domain expertise as technologies like generative AI and other sort of assistive technologies continue to evolve, then the domain expertise becomes the most important because then you start interacting on a natural language way and you don’t have to have those same technical skills to get there, which I think is probably one of the most interesting uses for generative AI is actually bringing out the barriers for technical ability to execute in a job like security or IT.

    Ryan Kovar: Prompt engineering will be one of the most significant requirements for entry-level and mid-level jobs by next year, in my opinion, categorically.

    Mitch Ashley: We’ve already been through one generation. It’s called search engines. Now, we’re doing it with generative AI.

    Ryan Kovar: Yeah. Yeah. No. We’re talking about this, but the reality is I’ve been doing cybersecurity and IT since 1999. Cory, there’s enough gray there. I’m sure you’re about the same generation. When I started, there was no Google and you had to read the Microsoft TechNet documentation, and then there was Google, and I put 65 CD binder in the trash and said, “Never again.”

    The fun thing that I always tell people that I’m mentoring or advocating for in cybersecurity is I’ve been doing this for 24 years and, of the 24 years, I have about four years of knowledge that’s relevant. I have 24 years of wisdom, but four years of knowledge that’s relevant. Everything else, no one cares that I can fix Exchange 5.5 driven pub EDB and I know how to defrag a Windows NT 4.0 server. That doesn’t matter. That’s one of the great things about cybersecurity and IT in the general is that you can become a subject matter expert in something very quickly and not have the bias of age.

    Mitch Ashley: Well, let’s turn our lens to the process part of it, and we’ve talked a lot about people and we could talk a lot more about it. There’s some great conversations that we’ve had already about that. We’ve got to have processes that are well-oiled to highly tuned, but can be responsive of incident management when things happen. The organization now has to operate cross-functionally, the stove-pipes, the things that we’ve lived with for so long. Now, we have to operate cross-functionally. We’re trying to tear those down, but our processes have to work across those.

    I’m interested in your all thoughts about how are organizations, what do they best do to adapt to what we need today so we can be speedier, respond more quickly and more reliably to threats or incidents or needs of the business?

    Cory Minton: Yeah. Do you want to to hit it, Ryan?

    Ryan Kovar: No. Please, Cory, go ahead.

    Cory Minton: Okay. I was actually going to say it’s actually not even an option anymore. We talk about that they need to do it. It’s not an option. If you look at some of the SEC’s recent rulings on the disclosure of material and incidents that happen for publicly traded companies in the US, you have to report that. You have to have that cross-functional view of how if a cybersecurity incident happens or an IT sort of incident outage or breach, any of these categorical things that happen, if they have a material impact on operations that would affect shareholder value, then you must disclose those things, and so now the impetus is on every publicly traded company to get this figured out really well. It’s got executive buy-in now that we’re going to tear down the walls between security and IT operations and our engineering teams developing our next digital capabilities because, anywhere across that spectrum that we have a process breakdown, whether it’s externally caused or internally caused, if it’s material, we better report it.

    These are no longer like, hey, it’s a good idea. This is like “test that theory and report the results” kind of stuff that the SEC will send somebody to jail over. We’ve already seen some convictions on previously publicly traded companies on misreporting things. It’s no longer a game. We’re serious about this. I think, from a process perspective, one of the things I’m seeing is reaching across the aisle. CISOs are talking to CIOs and CTOs more so now than I’ve seen in the last couple of years because they have to understand those impacts and they’re looking to organizations that actually already do some of them within the company that are already being trusted by different pockets of the organization. They’re looking externally for that guidance and help, whether it’s from consulting partners that are, hey, advising on security operations or advising on software development capabilities or technology partners.

    I mean, even the CISA, the Cloud Infrastructure Security Agency, their recent strategy update talked about one of the key pillars of their annual strategy for resilience included technology partners that were going to help them achieve that resilience. I think we have the impetus. The measurement is required now. There’s no longer a game, and I think executives are starting to understand it and they’re looking external to say, “Who can help me deal with this?” No matter where we land on what is material and some of the legal questions on how it’s implemented, we still have to respond and we still have to report some of those capabilities. Now is the time to start reaching across the aisle and starting to ask questions of your partnerships that you have in the org on how can you help us solve this problem.

    Ryan Kovar: I look at things like DevOps which still today has a little bit of a carve out separate. In a lot of organizations, DevOps is a slightly different place than maybe traditional security and, well, certainly than security, and then possibly even different than traditional IT engineering or infrastructure. In today’s society or today’s world of technology, a lot of the security issues that organizations are facing are in their DevOps pipeline, and so I find it fascinating what Cory said. The SEC has this term, material finding. You have to report a material finding.

    Now, I as a security professional may very well know what that is, but, frankly, the DevOps world has gone feral to a point where the security organizations are not a part of it and so the DevOps team are the only ones who understand what the security implications are. They’re the ones finding it and, because they’re in a DevOps mind frame, they’re not stopping and saying, “Let’s create an incident. Let’s walk this through.” No. They’re just fixing it and they’re moving it on. That’s how the cloud works. That’s how DevOps works.

    There is this essential need for us as an industry to really start reaching more across because we are being outpaced by DevOps home growing their own SecOps without oversight, without the wisdom of the security world, but we can’t stay in the way of the business, which is why that’s so important for us to go across and understand this because now there’s regulatory requirements and the real world is people are going to move forward whether they like it or not, so get on that train.

    Cory Minton: Yeah/ it’s funny, Ryan, you say that. I was reading the SEC reports from JPMorgan Chase. Jamie Dimon, their CEO, said in the letter to shareholders he had two sort of funny juxtaposed statements that used the same phrasing, which I found interesting. He said two things he could not overemphasize, one was, “I cannot overemphasize the need for cybersecurity in our organization. Everything must be secure,” and then later in the letter he said, “I cannot overemphasize our need to deploy innovative technology,” which is exactly what you said. If my CICD pipeline is continuing to push updates, but I’m not applying those cybersecurity principles that, like you said, was incident review and forensics and actually looking into it, then we’ve missed the boat on this. One of the key parts of this SEC filing is you not only have to report the outage, but you actually have to from an annual reporting perspective talk about your processes that you’re using to ensure cybersecurity in those areas. That’s one of those that’s like the CEO is saying this to shareholders. We better pay attention.

    Ryan Kovar: The CEO of a bank that touches 20% of every dollar in the world every day, right?

    Cory Minton: We cannot overemphasize.

    Ryan Kovar: I love that example because I look at DevOps as a perfect world where I have no fear at all that the DevOps team will identify and remediate security issues very quickly and then continue to do so because they’re not looking at the larger picture. Why did this get ingested? Why do we have this happening? Why is this CRO a threat actor you may not know anything about a threat actor. You may not understand that Tampa strawberry as per Microsoft has a really significant interest in your organization, and one of their TTPs is actually moving up the chain and actually jumping ahead for software supply chain.

    That is something that I would not expect a DevSecOps engineer to understand, but that is where the context of a larger security world, and that’s why this resiliency message, this is why it’s working across aisles. It becomes so important for the process. I honestly think the technology is usually the easiest part of this. The people and the process to implement and secure the technology, way harder, way less interesting than most people, except nerds like me, but much harder.

    Mitch Ashley: Well, and what you really bring to light is that the security specialists are going to understand the threat landscape much better. DevOps developers, et cetera, know their environment focused on the things that they are. You mentioned Jamie Dimon. Those two things, he’s expecting us, our CEOs are expecting us to figure out how we bring that together, how we make that happen so we don’t have feral organizations, so we don’t have processes that are brittle when things really fall apart or our environments get more complex. I mean, that’s an obvious thing. They are complex and they’re getting more complex, and no one understands the whole thing, so we’ve all got to pull together and say there are three dimensions to this problem, not one, and here’s the steps. We together figure out how we fix this.

    Great. Well, let’s talk about the technology perspective then. Obviously, Splunk being a great technology company, have been around for a long time. I remember finding Splunk on the showroom floor in the early days still doing black T-shirts then just like they are today, a great company that you all work for. Let’s talk about some of the technology side of what’s important and, again, thinking about not just internal people and processes, but also third parties that we’re using like the Splunks of the world in our organizations and how you can help with this challenge.

    Ryan Kovar: I’ll start really easily on this one, and then Cory will actually say something much better. People buy software to solve their problems. That’s it. That’s the easiest way when I pay people. I didn’t come from a sales background. I was operational. I was a threat hunter for the government, threat intelligence at DARPA, places like this. I didn’t think about why I bought software until I worked for a software vendor. It’s very easy. You buy software to make your life easier, to solve a problem faster or to solve it better.

    When I look at what Splunk does, a lot of our recent efforts are really around this incremental growth of just making people’s lives better, making people’s lives easier, cutting down the barriers to do their job faster because that’s what people need, whether it be security, whether it be observability, whether it be traditional IT engineering, that to me is really what we focus on here at Splunk which is why I’m still here after nine years. We make people’s lives better and we allow them to fulfill the mission that they’re actually being paid for rather than fighting the software that they’re buying and paying money for. That to me really is at the heart of what we do.

    Cory Minton: Yeah, and I think, if anybody’s seen some of the Splunk messaging in the market, we’ve really rallied behind this idea of resilience, and I think it actually sums up nicely the things that we do which I think, based on the conversation we’ve just had about the people and process, things that are challenging organizations, I think we’re in a really unique place to be a partner, to be one of those partner organizations and be the software that people buy to make their lives easier because we sit in this unique place of being one of the few organizations that actually unify, tear down the walls between security, IT operations and those feral DevOps teams that are out there building the next digital services because our corporate mission is around resilience, and we think of that as a simple statement of, “How do you keep all this digital business secure and up and running?” Simple statement, but, secure, not a simple thing to do, and up and running oftentimes equally on simple thing to do.

    Ryan Kovar: Sometimes at loggerheads.

    Cory Minton: Yeah. Exactly, a pearl of either sort of objective. There used to be this term we called a MOM. We all love MOM, monitor of monitors. It sits above. It sits at an enterprise level, at a higher level to give you visibility. I think that’s what folks have maybe struggled with that we see. Hybrid cloud is reality, right? There’s lots of clouds, not one cloud. Clearly, there’s leaders in terms of revenue, and each quarter they grow at different paces, but candidly, even Michael Dell said some years ago, “The cloud is not a place. It’s more of an operating model.” As you see SaaS being part of cloud journeys, yes, data centers are still run by organizations. They just are run now in a much more orchestration pattern similar to cloud providers, and that reality of lots of silos across different landscapes is creating challenges.

    If you use one tool that’s provided by said provider, then, yes, you can secure and keep up and running that one particular cloud real estate, but what about how it’s connected across the organization? How about the ways that applications traverse deployment centers? I think that’s what we really see as interesting is organizations are leaning on Splunk especially in the macroeconomic conditions where everybody’s focused less on growth and more on profitability, so they’re looking at like, hey, how do I reduce the number of tools?

    Ryan Kovar: More out of less.

    Cory Minton: Yeah. Exactly. Do more with less. They’re looking at it and going, “Wait a second, so I have this network environ, this router, so my data center. I’m sending the logs from that thing to nine different tools. Why am I doing that?” That’s nine times I’m paying for that bit of data to be stored and processed and analyzed by some number of tools, and organizations are looking at it and going, “Maybe that’s not smart. Maybe that same piece of data has nine questions being asked of it, and is there may be a fewer number of tools that could answer those nine questions effectively to give me the outcome without having to have all this sprawl?”

    It’s a unique conversation that we’re in today. Candidly, the tools consolidation, that rationalization of tools is one, looking at data. The data deluge hasn’t stopped. As we talk about sending data to nine different places, well, as that data becomes richer, it’s no longer just logs. Now we’re looking at metrics for real time. We’re looking at traces for spanning. How does an application actually impact multiple sort of environments? The data is getting rich and large. Is it all equal? Should we all be sending it to the same place? Should we be treating it with some valuation?

    Those are conversations that I’m having today that technology is helping solve, and Splunk is doing some really interesting work in the data rationalization and in that tools consolidation area that’s making it easier for customers to do more. As Ryan said, you bought a piece of software to make your life easier. We’re trying to go out and help folks figure out how to make more lives easier with the software they already bought.

    Ryan Kovar: I’m just required to hit all the buzzwords, so I’ll say ransomware now. It’s fascinating to me. Resilience, as in the concept of business leaders and cybersecurity, CISO leaders having to come together, I think, partly is really being driven by this threat of ransomware because the first time a cybersecurity threat has consistently and continually impacted every aspect of a business where 10 years ago, 15 years ago, oh, APT1 from China, exfil data. Well, that was the big issue. Well, by the time you heard about in the news, it’s already done. It’s actually relatively small scale. The impact is we were compromised, there was a breach, and now we have to deal with the public relations fallout, the stock drop, and also we have to make sure this doesn’t happen again.

    With ransomware, it’s actually we’ve encrypted your critical business systems and you can no longer do business, and now you have questions of do we pay the ransom? Well, now you need to involve the finance or you need to involve your chief legal officer. Oh, it’s publicly out there that we have this security issue. You have to involve the PR team, the public relations team. Oh, it’s actually knocked out our core business. Oh, well, now you’re involving the sales org. Now you’re involving the IT org. It’s coming across all these different places, so it’s builds back into what tools do you have that facilitate this and then, going back to the people part that we touched on earlier, this is where someone needs to be across all of them because they need to understand the impact and actually drive those changes that are needed and, because of ransomware, we actually have to deal with it. We can’t just pretend it doesn’t exist. CISOs can’t just be nerds in the closet playing cyber. They actually have to be able to speak eloquently to their business leaders, to the board of directors and others.

    Cory Minton: Yep. I’m just going to jump on the bandwagon of buzzword bingo. One of the things we’re seeing a ton of, too, is we talk about new digital workloads constantly being brought in, new innovation, we’re hiring consultants to bring in and actually help generative AI get real in our organizations, there’s a lot of conversation about is generative AI threat versus risk? There’s a whole kind of conversation there, but one that actually, as we’re moving past the hype and the AI washing nonsense that’s happened in the market, once we’re getting to real, a lot of CISOs have been talking about or talking to are trying to figure out how do I deploy those technologies internally? How do I go find the models that I can train against my data sets that would actually go help my teams do their jobs better?

    What I would say is is that’s another one of those areas where, as you’re innovating and bringing in new technology, treat it like another digital workload. You’re still going to have to detect when something goes wrong. You’re still going to have to investigate when something goes wrong in that generative AI hallucinates, it has an outage, whatever, and you’re still going to have to respond to those outages and take those learnings and put it back into the operational process of how do you run that system more effectively.

    I think people need help oftentimes finding those consistent patterns that we have to deploy to just keep things securely up and running regardless of what the workload is under the covers, and I candidly see that as one of the greatest areas for leaders that are looking to partner with a professional services organization or an assay, those outside consultancies. There’s a lot of value there in having them help bring in the patterns especially on proven technologies to short cycle, increase your time to value or, excuse me ,shorten your time to value when thinking about those new digital workloads and making sure that they’re secure and up and running.

    Mitch Ashley: I like where you’re taking this, which is one I wanted to wrap with, which is our organizations are expecting us to innovate, not just ourselves innovate, but also support the rest of the organization to be able to innovate. You talked about Ransomware. The newest thing is they’re not encrypting your data anymore. They’re just threatening to release it, so it’s not a matter of losing the data. It’s losing control over it. AI is the next, maybe it’s the next boon to phishing. We may not be able to tell the difference between an email from a generative AI LLM that’s been trained on how our CEO talks versus a well-scripted, predefined email.

    These new kind of threat models as well as new technologies force us to innovate and find new ways to do things. We have to help our people with tools, technologies, processes. I’d love to hear some more thoughts about what do we do to set ourselves up for success to be able to innovate in a way that’s going to help support where the organization is today and where it’s going?

    Ryan Kovar: Well, I’ll make a slightly controversial opinion here. I don’t think there’s ever been a single technology that outweighs the benefits for the offense or defense. I look at generative AI as something that is going to liberate security organizations from the shackles of mundanity. It’ll democratize what we can do in terms of analysis. It will also do the same for the adversaries. The only people who will lose are those who are not taking advantage of this technology on either side.

    I really do believe that we can all up-level as we go through with different technologies just as the adversaries are. When we invented a trench or trench warfare in World War I, they invented the tank. When we invented an airplane, they invented anti-aircraft. This is a tit-for-tat thing, and there’s always a leap at some point. People created a star fortress and that stymied invasions until someone invented the cannon. Sometimes, defense beats offense and sometimes offense beats off defense, but it goes back and forth as you go.

    I don’t necessarily see anything that’s a semantic change in how organizations will defend, but I do believe, if you’re not paying attention, if you’re not innovating, if you’re not selecting vendors, if you’re buying software, if you’re not buying vendors who aren’t innovating, you’re going to have a very difficult time in the next five years because your adversaries absolutely are.

    Cory Minton: Yeah, and I would go maybe less digital and say we as leaders cannot underestimate and cannot undervalue investments that we make in our people and our business culture today that create space for the learning and the development and the skills attainment that Ryan talked about, being critical. Leaders must create space for technologists, for practitioners, for leaders to go learn and to go study and have a mantra of spending some large amount of time on a regular basis, on a cadence basis, investigating, whether that’s going and attending conferences and networking and starting to understand what’s happening in the industry or it’s partnering with a technology organization and doing more hands-on workshops and learnings or it’s going next, when you’ve got a consultant in, spending time with those folks in the office picking their brain on what are the skills that they’re paying you for? Why are you here? What is so unique about what you’re doing, and how do I learn more of that?

    It comes from leaders making that a priority. If we’re going to have an organization that has people, process and technology that’s actually going to be resilient, that’s going to be able to survive in this next epoch of technology innovation, it’s going to have to be led by the best intelligence, which is still human intelligence, and that requires investment, and I think you got to put a stake in the ground and make that a priority today.

    Ryan Kovar: I love that part because, going back to our previous discussions about leaders who often have 10, 20, 30 years of experience, none of it is relevant. When I started my career as CISO, my only half, well, first off, CISOs didn’t exist. Second, the CTO or the person in charge of technology at that company had to pay attention to desktops, printers and servers on-prem, and then 10 years ago, 15 years ago, it was laptops, desktops, printers on-prem, printers in the home offices, a data center that they co-located to in this little cloud thing that they’re trying out, and then five or 10 years ago, it was all those plus software as a service, and now it’s, okay, well, we have a CICD pipeline and we’re driving APIs and our entire business is dependent upon this one piece of open source software that’s maintained by three guys in a former Yugoslavian Republic and, if it has an issue, we’re going to be done.

    Also, we M&Aed a company out of China. Every time they develop a vulnerability, that has to go to the Chinese government along with us. It’s a lot more complex. For those who are just resting on their laurels of “I’ve been doing this for a while” and not up-leveling their own education and hiring people who are subject matter levels as direct lieutenants, you’re going to have a bad day.

    Mitch Ashley: Bringing it back to the thinking that got us here is not the thinking that will get us to the next place, I’m paraphrasing Einstein here, but it’s also our own thinking. That’s the value of bringing in perspectives like yours with your partners, your suppliers, companies like Splunk, professional services. There are a lot of ways. I think we’re at the same junction point where we were with the cloud. Where we’re like, “Here’s the cloud. Is it just somebody else’s computer or is it a new way of doing things?” But what is that new way?

    We’re kind of in that place today with AI. Okay. Right. We know we can do generative AI stuff, but what does it really mean? How do we leverage it, to your point, Ryan? You have to be a participant on the field to really figure that out. You don’t have to go spend crazy money on the latest thing just because that’s what was in the airline magazine, but it is about figuring out and learning what you can do with it and what’s possible learning by yourself, but also with others.

    Well, thank you to both of you, to Cory Minton and Ryan Kovar, for joining us, and thanks to the Splunk team for bringing us together to have this conversation. We hope it’s been really beneficial to everybody who’s listening in. I know it’s been great for me as well. Thank you.

  • From DevOps to the Future: The Evolutionary Leap to Platform Engineering

    From DevOps to the Future: The Evolutionary Leap to Platform Engineering

    The paradigm of technology development never ceases to amaze. Over the past few decades, we’ve witnessed an unbroken chain of progress from traditional IT to Agile methodologies, from cloud computing to microservices and now, another leap is on the horizon. Enter the age of the platform engineer, a role that is swiftly emerging as a logical evolution of DevOps culture.

    From the Roots: Understanding DevOps

    DevOps revolutionized the way teams design, build and deliver software. With its advent, we saw the amalgamation of developers and operators into a single team, fostering a culture of collaboration and integration. The core idea was to tear down the traditional silos and ensure seamless interaction between different stakeholders in the development process.

    This unique approach has proven its mettle over time, bringing about shorter development cycles, higher deployment frequency, more dependable releases and quicker resolution of issues. DevOps set the stage for continuous delivery, integration and monitoring, thus speeding up the digital transformation of many organizations.

    The Evolution: Birth of the Platform Engineer

    As we delve further into the era of cloud-native technologies, containers and service meshes, the demands on the DevOps methodology have evolved. To cater to the increasing complexity and scale of modern infrastructures, we witness the emergence of a new breed of professionals, aptly titled platform engineers.

    This role is not just a renaming of the traditional DevOps engineer; it is an evolution. Platform engineers build upon the foundational practices of DevOps but go beyond by embracing a holistic view of the technology stack. They focus on creating platforms that provide a unified, consistent, and efficient environment for development and operations teams to work within.

    Platform Engineering: The Core Components

    Platform engineers possess a skillset that spans software development, infrastructure management, system operations, and, importantly, a deep understanding of the business context. They are entrusted with the design, construction and management of the underlying platforms supporting the deployment and scaling of applications.

    A significant part of the platform engineer’s role involves creating self-service infrastructures that empower teams to deliver software independently and efficiently. This can involve building and managing internal toolchains, maintaining cloud-native infrastructures, designing service observability systems and ensuring high levels of automation across the board. 

    The Future is Here: Embracing Platform Engineering

    The rise of the platform engineer signifies a shift towards more scalability, resilience and efficiency in managing modern tech ecosystems. It’s about leveraging technology to its fullest extent and creating a platform that is more than the sum of its parts. 

    As the technology landscape continues to evolve, the transition from DevOps to platform engineering may soon become the norm. It’s an exciting prospect, promising a future where innovative platforms can seamlessly integrate and orchestrate the complexities of the modern digital world. After all, in this era of rapid digital transformation, those who adapt will thrive. 

    In conclusion, the platform engineer role is not replacing DevOps but evolving from it, extending the principles that made DevOps so successful to meet the challenges and opportunities of the new tech era. As we continue this exciting journey into the future of software development and delivery, there’s no doubt that platform engineering will play an increasingly important role. 

  • Five Great DevOps Job Opportunities

    Five Great DevOps Job Opportunities

    devopsy.kinsta.cloud is now providing a weekly DevOps jobs report through which opportunities for DevOps professionals will be highlighted to better serve our audience.

    Our goal in these challenging economic times is to make it easier for DevOps professionals to advance their careers.

    Of course, the pool of available DevOps talent is still relatively constrained, so when one DevOps professional takes on a new role it tends to create an opportunity for others.

    The five job postings shared this week are selected based on the company looking to hire, the vertical industry segment and, naturally, the pay scale being offered.

    We’re also committed to providing additional insights into the state of the DevOps job market. In the meantime, for your consideration:

    Indeed.com

    PEAR Health Labs
    Remote
    DevOps Manager
    $150,000 to $175,000

    Dice.com

    SR Partners
    Dallas, Texas
    DevOps Architect
    $140,000 to $160,000

    SimplyHired

    Reynolds & Reynolds
    Tallahassee, Florida
    Senior Software Architect
    $106,000 to $135,000

    LinkedIn

    Genesys
    Oakdale, Connecticut
    Senior Software Engineer: DevOps
    $93,600 to $194,400

    CareerBuilder.com

    Patterned Learning AI
    Remote
    DevOps Engineer
    $60,000 to $70,000

  • Revolutionizing DevOps Culture Transformations With AI-Engineered Tools

    Revolutionizing DevOps Culture Transformations With AI-Engineered Tools

    In my recent article Revolutionizing the Nine Pillars of DevOps with AI-Engineered Tools, I explained that DevOps thrives in a culture of openness, transparency and shared responsibility. It’s about breaking down the silos that can exist between different teams (like development, operations, security and QA) and promoting effective communication and collaboration. AI-powered tools can improve collaboration through smart recommendations, fostering more effective communication and knowledge sharing.

    Collaboration and Communication: AI-powered chatbots can facilitate quick and easy communication across teams, enabling faster decision-making and problem-solving. AI-powered project management tools can improve visibility into tasks and dependencies, promoting better coordination among team members.

    Continuous Learning and Improvement: AI-based monitoring tools can help teams learn about system performance in real-time, identifying trends and patterns that can inform better decision-making. AI can also help in creating personalized learning experiences, identifying skills gaps and suggesting relevant training resources, promoting a culture of continuous learning.

    Shared Responsibility and Ownership: AI can improve visibility into the system’s state, helping all team members understand how their work contributes to overall system performance. This can promote a sense of shared ownership and responsibility. AI can also assist in automating the allocation of tasks based on an individual’s skills and availability, promoting a more equitable distribution of responsibilities.

    Automation and Efficiency: AI is pivotal in automating routine tasks, such as testing, deployment and monitoring. This can significantly improve efficiency and allow teams to focus on strategic, high-value tasks. AI can also help automate the detection and resolution of common issues, reducing downtime and improving system reliability.

    Feedback and Adaptability: AI-powered analytics tools can provide real-time feedback on system performance, enabling teams to quickly adapt their approach as needed. AI can also help in analyzing customer feedback and identifying key themes and sentiments that can inform product development.

    Risk-Taking and Innovation: AI can help simulate the potential impacts of different decisions, allowing teams to take calculated risks. AI can also drive innovation by helping teams identify new opportunities, predict trends and develop more creative solutions.

    Roadmap to AI-Assisted DevOps Culture

    A practical roadmap for transforming an organization’s DevOps culture using AI-engineered tools may look something like this:

    1. Define Goals and Evaluate the Current State: Identify what you want to achieve by integrating AI into your DevOps practices. This could be improving deployment times, reducing errors, increasing productivity and more. Evaluate your current DevOps practices and tools to understand where AI could bring the most value.

    2. Identify AI Opportunities and Select Tools: Look for areas where AI can enhance collaboration and communication. This could involve AI-powered chatbots for team communication or project management tools with AI capabilities. Identify opportunities for AI to support continuous learning and improvement. AI-powered analytics and monitoring tools can provide valuable insights for this. Determine where AI can help promote shared responsibility and ownership. AI-enabled visibility tools can assist here, too.

    Choose areas for AI-powered automation. This could include tasks like testing, deployment and issue resolution. Find avenues where AI can support feedback and adaptability. Tools that analyze system performance or customer feedback could be helpful here. Pinpoint where AI can assist in risk-taking and innovation. AI-powered simulation or predictive analytics tools could be useful.

    3. Implement AI Tools Gradually: Start by integrating AI tools into one or two areas of your DevOps process. Monitor the impact to ensure you’re driving the desired improvements. Gradually expand the use of AI tools to other areas of your DevOps practices.

    4. Train Teams on AI Tools: Provide adequate training to your teams so they can use these new AI tools effectively. Encourage a culture of learning and experimentation to help teams become comfortable with AI.

    5. Monitor and Adapt: Regularly monitor the impact of these AI tools on your DevOps practices. Are they improving speed, reducing errors, enhancing collaboration, etc.? Be prepared to adapt your approach based on the feedback and results you’re seeing. The integration of AI is a journey, not a one-time event.

    6. Continuously Improve: Continually look for new ways that AI can enhance your DevOps practices. AI is a rapidly evolving field, and new opportunities are emerging all the time. Remember that this roadmap is iterative and adaptive. AI integration should be seen as a part of the broader DevOps goal of continuous improvement.

    Benefits for Organizations, Teams and Individuals

    With the integration of AI-engineered tools into DevOps processes, organizations can expect significant improvements in efficiency, speed and quality. AI automation can drastically reduce manual, repetitive tasks, thereby lowering the chance of errors and freeing up resources for more strategic endeavors. This can lead to cost savings and a better allocation of resources. Additionally, AI can enable faster decision-making and problem-solving, accelerating the software development life cycle and making the organization more competitive. AI can also enhance product quality by providing valuable insights and analytics, enabling continual improvement.

    For Teams: AI can facilitate improved collaboration and communication, making workflows more seamless and efficient. AI-powered project management tools can provide better visibility into tasks, dependencies and progress, enabling smoother coordination. AI can also automate routine tasks, freeing up team members to focus on more challenging, high-value work. Additionally, AI can provide real-time feedback and insights, enabling teams to continuously learn and improve their processes. These improvements can lead to higher job satisfaction and productivity.

    For Individuals: Individuals stand to gain from AI integration in several ways. AI-powered tools can support skill development by helping individuals learn new technologies and best practices. Real-time feedback and insights can also foster personal improvement. Automation of routine tasks allows individuals to focus more on creative and strategic aspects of their work, leading to a more engaging and satisfying job experience. Furthermore, being part of an AI-driven DevOps environment can open opportunities for individuals to work on cutting-edge technologies and innovative projects, enhancing their career prospects.

    Measuring Success

    Choosing the right metrics is crucial for measuring the progress of your DevOps culture transformation with AI-engineered tools. Here are some important metrics to consider:

    Deployment frequency: This is a measure of how often new features, bug fixes or updates are deployed. An increase in deployment frequency can be a sign that AI tools are improving efficiency.
    Lead time for changes: This metric refers to the amount of time it takes from code being committed to it being deployed. AI tools can help to reduce this time by automating testing and deployment processes.
    Change failure rate: This is the percentage of changes that result in a failure or that require a rollback or hotfix. AI can help reduce this rate by improving testing accuracy and identifying potential issues earlier in the development process.
    Mean-time-to-recovery (MTTR): This is the average time it takes to recover from a failure. AI tools can help to reduce MTTR by quickly identifying the cause of failures and automating the recovery process.
    Customer satisfaction: AI can help improve the quality and reliability of the product, leading to increased customer satisfaction. Customer satisfaction can be measured using surveys or net promoter score (NPS).
    Efficiency metrics: Look at how much manual, repetitive work has been reduced by automation. This could be measured in terms of man-hours saved.
    Learning and development: Monitor the upskilling of your teams to using AI tools and technologies. This could be tracked through training completion rates, skill assessments, etc.
    Innovation rate: AI can free up teams to focus on innovation. Measure the number of new ideas or features being developed over time.

    Summary

    To transform DevOps culture with AI tools, organizations must set clear goals, identify AI opportunities, gradually implement those tools and monitor their impact. Training teams to use AI tools effectively and fostering a culture of continuous learning and improvement will enable this transformation and help DevOps teams thrive. This integration can significantly enhance efficiency, decision-making speed and product quality at an organizational level.

    On a team level, AI can streamline collaboration, automate routine tasks and provide real-time insights for continuous process improvement. For individuals, this transformation provides opportunities for skill development, personal growth and more satisfying job experiences. Monitoring progress with metrics like deployment frequency, lead time for changes, change failure rate, mean-time-to-recovery (MTTR), customer satisfaction, efficiency, learning and development and innovation rate ensures the transformation is delivering the expected benefits and contributing to continuous improvement.

  • How to Build Successful DevOps Teams

    How to Build Successful DevOps Teams

    The DevOps sector is still in urgent need of new talent, with 38% of tech companies stating that they struggle to find skilled developers in 2023. These same respondents named DevOps as the number-one area where they are feeling the pinch. Because of this, businesses must ensure they prioritize and nurture the right set of core skills when building DevOps teams.

    Below, we discuss the key qualities that businesses should look for when taking on new DevOps engineers and what leaders can do to get the most out of existing employees.

    Tech companies that prioritize the development of these skills will be rewarded with an intelligent, inquisitive and high-performing DevOps team, setting themselves apart from competitors.

    An Eagerness to Learn

    The DevOps sector is constantly evolving, so the talent organizations hire should be similarly open to evolution in the way they work.

    Emil Krawczak, DevOps engineer at STX Next, said this is a key trait necessary for success. “To be successful as a DevOps engineer, you need to have a natural thirst for knowledge and desire to develop, and also be highly persistent when faced with challenges. This area of IT is developing so quickly that it’s difficult to keep track of everything new. It is not enough to learn at work; you need to take steps to grow your capabilities outside of the day-to-day, as well.”

    An Analytical and Strategic Approach

    System administration and development skills are essential, but they aren’t enough if developers don’t keep the bigger picture in mind at all times. Therefore, the ability to think strategically and the ability to apply analytical skills are invaluable for DevOps engineers.

    Piotr Pawłowski, DevOps specialist at The One Question, explained how these traits made it easier for engineers to tackle a DevOps project. “Being successful in DevOps means having an analytical mind and taking a strategic approach. First and foremost, you should be to analyze the project and decide the right tools and approaches. You might know a lot about different tools and cloud solutions, but if you don’t know how to use them, you will not be successful in the world of DevOps.”

    Putting DevOps Theory Into Practice

    The ability to put theory into practice is where true progress is made. With that in mind, what DevOps employees glean from training, education, books and research has little relevance unless it is supplemented by practical experience.

    Versatility is an integral skill in the industry, and successful DevOps pros should be able to use—or at least have working knowledge of—a broad range of tools. They should also specialize in at least one area of Dev or Ops. Python is a great example; Python is highly effective in the automation process and the language should be viewed as a key stepping stone in growing DevOps engineering skillsets.

    Identifying Transferrable Skills

    Employers searching for new talent should look for relevant skills in other IT disciplines that could help a candidate make the switch to DevOps. For example, experience in SysOps or a specific programming language can be extremely useful when opting for a change of focus.

    “DevOps unites the world of admins (SysOps) with the world of programmers/developers (Dev). The easiest way to get into DevOps is to have experience in both of these areas,” Krawczak said. “As an admin, you can build on your experience in scripting, and as a developer, you can leverage your experience in setting up local infrastructure for testing.”

    A Clear and Focused Approach to Specialisation

    Marcelo Grebois, senior DevOps Engineer at Toptal, said businesses should keep things as simple as possible when establishing their DevOps approach. “Try to make things easy. Always think about automation, keep all infrastructure ‘as-code’ and stay up-to-date on tools and technologies.”

    Roman Kamiński, cloud system architect at Atos, agreed. “Using automation tools such as Ansible and Terraform, for example, stops developers from getting left behind while helping them adjust to an ever-changing industry.”

    Developing the Next Generation

    It’s important that tech companies invest in the growth of DevOps engineers and help them focus on the skills that will maximize their potential. The organizations that put time and resources into nurturing the next generation of talent will benefit from a larger pool of well-trained engineers. In the long term, this will ease the DevOps skills shortage and make for a more prosperous IT department.

  • Improving DevX Gives Your Business a Boost

    Improving DevX Gives Your Business a Boost

    As organizations transform and grow, different methodologies and development tools can come together, creating an extra level of complexity for your mainframe application development teams. In a perfect world, developers would be able to focus on delivering solutions without worrying about all the infrastructure necessary to make that happen. Building a better developer experience, or DevX, can make that a reality.

    DevX is about meeting developers where they are, at every stage of their career, delivering compatibility with and connectivity to the tools and devices that they want to use so they can focus on what they do best: Coding and releasing high-quality mainframe applications faster and more frequently and fostering better productivity and more innovation.

    Automation is integral to removing friction for developers. A modern integrated development environment (IDE) with built-in automation allows developers to find and manage code more efficiently and with greater quality and move quickly and easily from coding to editing to debugging to testing.

    Modern tooling can simplify and accelerate how developers work by giving them a visual representation of their code layout in a single-pane-of-glass view. They can move quickly in a natural, easy way with more information at their fingertips while built-in automation offers assistance at the right moment without getting in the way.

    This can seem difficult when dealing with mainframe application development and longtime mainframe developers who are still working with tools from the 1980s haven’t directly benefitted from the intervening decades of improvements. What they might not realize is that the improvements made for other platforms are available to mainframe developers, too. By engaging them in your DevX modernization efforts, you can improve their experience—and that of their team members.

    Elements of a Productive Mainframe Developer Experience

    • A modern DevX adopts flexible agile processes that give developers predictable capabilities and enable smaller, iterative changes with faster feedback. It also syncs distributed and mainframe development teams to foster greater collaboration for additional process improvements.
    • A better DevX is platform-agnostic, creating a development environment where developers can focus on the functionality of the applications they’re building instead of the deployment mechanism, leveling the playing field for every generation of developer.
    • With an improved DevX, developers can achieve the full potential of DevOps. Implementing DevOps for source code management (SCM) functions allows developers to understand the scope of any changes needed before they start coding, which helps speed approvals through concurrent work streams.
    • The single biggest boost to productivity—and an integral component of DevX—is shift left automated testing that enables developers at all skill levels to test new code immediately and find and fix bugs earlier in the development cycle when they’re easier to fix—and before they reach the end user. Continuous testing also automatically prepares test data within the DevOps toolchain.
    • An improved DevX delivers operational insights, so developers aren’t just catching issues but also understanding and prioritizing them, working hand in hand with the operations team to make the overall system better. It also empowers newer developers to address and correct any issues that arise in legacy code without poring over hundreds of modules to find what they need to improve. DevX tools work together to give developers an opportunity to leverage operational data in a whole new way.

    Once you have the tools, your mainframe developers need to be supported by a culture and education that teaches them how to use them and understand the benefits they offer. If you implement new tooling that nobody will use, the transformation is for naught. It’s important to start with the user experience and find the right tools to address it rather than starting with a tool and forcing its use just because it’s new. Continuous improvement is an important part of this; otherwise, you could bring the mainframe developer experience from the 1980s up to today’s standards only to have it freeze in place. This is why metrics and analytics are so important.

    As you speed up delivery, you also want to make sure the quality improves. With an improved DevX, you’re consistently finding defects earlier in the life cycle, which results in safer, more efficient delivery of better-quality code. By tracking velocity with metrics such as user stories delivered in one sprint, you can gauge how those changes are making a difference. You can also demonstrate how automation helps eliminate human error, increases code drops, and improves business agility.

    If you’re giving your mainframe developers everything they need to do their jobs easily and more efficiently, they’re exponentially more likely to adopt the tool and advocate for it—because a better DevX improves the day-to-day employee experience for longstanding and next-generation developers. By removing the drudgery of tasks that take developers away from writing code, you’re unleashing their creative and enabling them to achieve even greater levels of innovation—and you’re also improving your hiring and retention practices.

    Improving the mainframe DevX pays dividends across your business, from your developers to your customers to your stakeholders, so you can bring more new features to market faster and enable a competitive response for revenue-generating solutions. The mainframe is here to stay. Make sure you’re building a modern development environment that empowers your developers to respond nimbly in the face of change and make change your advantage.

  • How LinkedIn is Investing in Developer Productivity and Satisfaction

    How LinkedIn is Investing in Developer Productivity and Satisfaction

    More and more companies are beginning to examine the role developer experience has in improving engineering productivity and morale. And one company often referenced as a model for developer productivity engineering is LinkedIn—its engineering leadership has invested heavily into reducing time-consuming developer tasks and constructing an internal platform to gauge developer satisfaction on the tools they use.

    Engineers at LinkedIn recently shared how the company has invested in measuring developer productivity and happiness as well as their strategies for automatically cherry-picking commits. I also met with LinkedIn engineering leadership for a closer look at how the organization is working to improve the developer experience across its internal platform.

    Below, we’ll take an inside look at LinkedIn Engineering’s efforts to enhance developer happiness. We’ll explore their approach to improving their internal software development trends and consider the benefits LinkedIn has gained throughout the process.

    A Look at LinkedIn Internal Developer Solutions

    Productivity is challenging to measure, said Jared Green, VP of engineering, head of developer productivity and happiness at LinkedIn. So, the company opted to take a different approach by including happiness as a variable in the equation. Thus, they measure not only how fast a workflow is but continually monitor how enjoyable the process is to ensure developers have a delightful experience.

    “We don’t just strive for productivity, but for productivity and happiness,” said Green. And to his point, on-the-job satisfaction can have real-world results—on average, happy workers are 13% more productive, according to a University of Oxford study.

    LinkedIn now oversees a custom-built internal platform for evaluating the sentiment of developer tools. The platform, known as iHub, is a holistic listening channel to gauge developer satisfaction across various tools. It collects real-time qualitative feedback from users and correlates it with objective quantitative telemetry. It then surfaces this data in a UI dashboard for team leads and individual contributors to view.

    But there’s an art to engendering the trust required to collect accurate feedback, says Max Kanat-Alexander, principal staff software engineer at LinkedIn. For example, individual contributors need to be sure the data won’t be used for performance reviews. Once that trust is established, folks are more willing to give genuine feedback, he said.

    Benefits of Investing in Developer Enablement

    The benefit of centralizing developer experience metrics with a shared platform means that tech leads can make data-backed decisions on how to improve their DevOps processes. “It’s really about thinking about giving people insights versus giving them data,” said Green. The idea is that frontline engineers can rapidly investigate any metric to expose problems and see trends. “The optics have given leaders ways to improve various parts of developer experience,” he said.

    Through their investment into developer experience, LinkedIn Engineering has discovered and streamlined hangups in the Git flow, exposed inefficiencies in their code review processes, and fixed specific developer build times. LinkedIn Engineering reported it has reduced time spent on certain time-consuming developer tasks by 99.6%.

    To construct iHub, LinkedIn Engineering wrote its own front end. The hub pre-computes data dimensions and stores them in an offline data store, increasing speed, said Kanat-Alexander. The system uses a sophisticated survey technology to collect the data and hides the complexity of manual collection in a sleek dashboard.

    LinkedIn managers have already leveraged iHub data to identify macro trends and resolve specific points of friction. For example, Kanat-Alexander describes how one director noticed that developer build time was worsening in their organization. They were able to isolate it to a single developer who wasn’t using remote development, which was spiking the metric. Leaders then assisted in unblocking them.

    Takeaways For Your Organization

    Most companies could benefit from a UI like iHub, explained Kanat-Alexander. But, you really must have a large engineering organization to see the benefits of the work. Plus, up to 80% of the legwork is simply data integrations—this involves hooking into systems, cleaning up data and deciding how to measure certain metrics. “Whatever you can measure, you can improve,” said Green.

    LinkedIn Engineering has flirted with the idea of open sourcing iHub, but it’s a bit too custom to LinkedIn to be valuable to other enterprises. “There’s a lot of variance in build technologies nowadays, and large-scale engineering systems are bespoke,” said Green. Implementing such a dashboard also requires a deep understanding of developer personas and how people work, which can be entirely unique for each organization.

    Although other organizations can’t fork LinkedIn’s efforts, there are some takeaways that could help others invest in their internal DX. First, Green recommended starting with the fundamentals. “Having a core of understanding about what you’re trying to move the needle on is important, as is coming up with ways to talk about it consistently.” This forethought should help establish what you want to measure.

    Turning DX Metrics Into Action

    Studies indicate that more companies are seeking to implement platform engineering or similar developer productivity engineering initiatives. The end result of these efforts should be to empower people and ensure they are satisfied with their work and not just a cog in the system.

    On that note, it’s critical that there are mechanisms for action to happen on the data that is collected, said Kanat-Alexander. Someone needs to monitor the insights and make changes to improve the metrics over time. “Then you can win hearts and minds.”

  • Breaking Down Silos: Applying Open Source Practices in the Workplace

    Breaking Down Silos: Applying Open Source Practices in the Workplace

    DevOps plays a prominent role in mainstream technology and companies’ ability to deliver results. While the definition of DevOps encompasses the complete range of characteristics it covers, in practice, current tech culture tends to focus solely on the tooling aspect, neglecting the equally essential focus on the people and process critical in delivering value to end users. While cultures are difficult to change and redefine, a transformation is needed to fully capitalize on the potential DevOps has to accelerate software development.

    Conway’s law–the concept that organizations ship software based on their organizational structure–outlines the need for visibility. This law revolves around the idea that in order for a software product to successfully operate, its developers must have established open pathways of communication to ensure each component works as a whole rather than as individual pieces. Unfortunately, today, many organizations still run into problems with how their teams operate when individuals own limited pieces of development.

    Take a retail platform, for example. In the development of the platform, there will likely be several sub-teams that focus on individual components of the design. Perhaps there’s a team for the shopping cart service, another for the catalog service, a team for the underlying data platform and one for user account services. The list goes on. Typically, those teams do not have visibility into each other’s work and operate in their own silo. As soon as teams start thinking about ‘their’ component or ‘their’ microservice, a dangerous tunnel-vision takes hold. In some cases, there may be standup meetings across teams for coordination efforts, but this takes time, energy and steals focus from the important task: Delivering value to end users.

    Alternatively, think about your favorite open source project. The inner workings were likely  designed asynchronously, openly, remotely and at a global scale. Given the remote nature of open source innovation, developer teams don’t need to be co-located in the same workspace. While physically operating separately, open source developers have mastered alternative methods to operate with transparency in such a way that external contributions aren’t just welcome, but encouraged.

    This same idea applies to proprietary software development, or InnerSource, a modern engineering approach that applies open source practices inside your organization to proprietary code. While a technological shift in practice, InnerSourcing is also a cultural revolution toward collaboration.

    Companies of all types can benefit from implementing InnerSource practices. Some enterprise organizations that operate using InnerSource methodologies today include 3M, Ford, KPMG, Mercedes-Benz, Nubank, Spotify and more.    

    A critical component in making this change is prioritizing team visibility. If other teams in your organization are unable to see what you’re working on, you’re inherently (intentionally or not) creating silos and limiting efficient collaboration. In practice, there will be scenarios where company-wide visibility may not be feasible due to external regulations, highly secure environments or sensitive projects; however, these moments of limited project visibility should be the exception rather than the default.

    When approaching InnerSource through an open source mindset, you will find that many DevOps principles still hold true. Here are a few strategies for implementing a collaborative open source culture when developing proprietary code:

    1. Use existing resources: When problem solving, using the existing materials available within your organization is a great place to start—it is possible someone has already tackled the issue you are addressing. If you find the right fit, you can reuse prior work and, if it has not been started, this is an opportunity to add value to your organization by creating a project that could help others in the future. When building upon someone’s work or starting a project, prioritizing visibility by providing an explanation of any technical requirements of the change will allow future project maintainers to determine whether it’s in the project scope, already a proposed change, on the backlog, or not a prioritized piece of work. This use of existing resources will help develop a collaborative culture over time.
    2. Documentation of work: Documentation is critical. Keeping track of the expected standards of the project and which tools or dependencies are required to build the project for local testing will help others in your organization know what to contribute. When in the early stages of contributing code to a project, documenting proposed changes without directly impacting the production version will allow you to receive feedback early on from the project maintainer about the direction you should take as they will be able to see your steps clearly outlined. To contribute code to a project, you can propose changes as a pull request, which doesn’t impact the production version but means you can get valuable feedback on the direction the project should take. Handling contributions by a pull request also ensures that branch protection rules apply and continuous integration practices are required.

    While implementing these practices may not result in immediate changes, over time they will result in a significant cultural shift. And while cultural transformations require continued conscious efforts, embracing modern software engineering practices–learned from the open source community–will enable teams to break barriers for organizational collaboration.

  • Things We Should Acknowledge, Part Three: Remotely Good

    Things We Should Acknowledge, Part Three: Remotely Good

    This installment of my off-and-on “Things We Should Acknowledge” series is going to focus on an area that most DevOps teams are skirting, but increasingly will not be able to: Remote work.

    For background, I have worked remotely for decades; my experience spans Fortune 100s, publishing companies and startups—and that’s just full-time work. I’ve worked for a ton of vendors and publications as a contractor, too.

    I will start by making clear my stance on remote work—a stance I’ve happily talked with people about since long before COVID-19 measures made remote work a regular thing. You either thrive working remotely or you wither. There is a small group in the center that can make it work with some effort, but inevitably that effort becomes routine and those in the middle become part of one of the two groups.

    Because people drift toward one of two camps, we get the currently popular stance on, well, everything, which is polarization. But we shouldn’t. Just like positions and people that should/should not be in meetings to resolve problems or people who should/should not have direct reports, we should consider positions and people that remote work is best for.

    And post-pandemic, the vast majority of DevOps roles can be performed effectively in a remote position. That leaves us with the people question. Mostly. Note: There are organizations and roles that need to be local; those are not a topic of this blog. The 99% of other roles/orgs are.

    This topic gets really ugly, pretty quick. If a significant portion of every organization goes remote, city centers in several metropolises will die. That is not conjecture; you can go look into commercial real estate in any information worker hub. It is in a massive slump (and calling it a ‘slump’ is being nice), with occupancy circling the drain and property values plummeting. That is largely because formerly high-demand real-estate is not needed in the volume it was.

    That should not be the concern of any given corporation. It is the corporation’s responsibility to see to shareholder value, not the real estate market (unless they are invested in real estate, which many large organizations are). But if orgs are keeping any offices downtown, it becomes an issue about safety. In other words, it’s about if the collapse of the market means the offices become less safe for workers, regardless of investment. This is not a decision-making bullet point, but it does inform actions. I have a ton more to say on this topic, but it’s a side bit, so for now, this is enough.

    For people who thrive working remotely—and a lot more people now know that about themselves than knew that five years ago—being forced into the office is a non-starter. I have been offered astounding positions at organizations on both coasts over the years, but the ones that came with the “…and you have to move here” demand—I immediately turned down. More people are doing that; some of them truly great employees.

    So, like it or not, DevOps teams are going to have to consider whether all being in the same building is more important than hiring the most qualified co-workers. That is a team/management/corporate decision and always has been. I consider any “We all do this one thing” stance—from “We’re an X shop” to “We don’t do remote” to be a negative, but I don’t have to make these decisions for an organization.

    It does need to be part of the conversation, though. Because the work environment is unlikely to return to pre-pandemic standards, no matter how much some want it to. The more organizations are willing to work with both remote and on-site employees, the more options those employees—your potential coworkers—have. That changes the dynamic, even in light of recent layoffs, because the new job hunt is not tied to “local” anymore.