Category: Blogs

  • Survey Surfaces Major Observability Challenges

    Survey Surfaces Major Observability Challenges

    A recent survey found the biggest obstacles to observability were the pace of technology change, IT blind spots and the complexity of modern applications. The survey polled 300 IT professionals to identify the biggest challenges they faced in achieving observability. The accelerating pace of technology change (72%) topped the list, followed by IT blind spots and the growing complexity of modern applications (58% each), and lack of funding (52%).

    The survey, conducted by SolarWinds, also found the typical enterprise suffered nine brownouts or outages every month, each of which lasted around twelve hours. The full cost of outages and brownouts averaged $13.7M annually for these enterprises, according to the SolarWinds report.

    Jeff Stewart, field CTO and vice president, global solutions engineering at SolarWinds, said the survey makes it clear that a lack of observability is having a material impact on businesses. Organizations are now moving beyond monitoring predefined metrics to embrace observability platforms that make it simpler to investigate the root cause of issues that adversely impact businesses; these businesses are more dependent on IT than ever, he added.

    In fact, the primary reason organizations are investing in observability is to improve their customer experience (96%), enable faster innovation (71%), reduce time spent solving (71%) and detecting (60%) issues and improve operational efficiency (55%), the survey found.

    While it’s still early days as far as achieving observability is concerned, a lot of progress is starting to be made as organizations realize that complex IT environments require a different management approach. IT teams can no longer afford to navigate multiple monitoring tools in what often turns out to be a vain effort to prevent brownouts and downtime, noted Stewart.

    In addition, many IT teams are becoming increasingly aware that the telemetry data collected via an observability platform provides the foundation for infusing IT operations with artificial intelligence (AI) capabilities, he added. The ultimate Holy Grail is to one day achieve autonomous operations, noted Stewart.

    There is, of course, no shortage of observability platforms, but Stewart contends that providers of monitoring tools such as SolarWinds are in the best position to help IT teams make the observability transition. Rather than ripping and replacing a monitoring platform, a more seamless transition that results in less disruption is preferable, he noted.

    Regardless of approach, observability platforms that enable IT teams to launch queries across complex distributed computing environments are becoming a necessity. Distributed computing environments often have more dependencies than IT teams can manually track. It’s only a matter of time before the most talented DevOps professionals start to migrate to organizations that provide them with the management tools and platforms required to succeed. Otherwise, the stress of managing a complex IT environment will only result in higher rates of burnout among DevOps professionals.

    It may be a while before observability platforms are pervasively employed across IT environments, but as it becomes clear that brownouts and downtimes result in a decline in revenue, business leaders will understand what’s required to maximize profits. In many cases, the answer will be an observability platform that surfaces the insights needed to proactively manage modern IT environments.

  • Generative AI: The 90% Rule

    Generative AI: The 90% Rule

    There is a concept we need to solidify that I will call the 90% Rule—until someone smarter than me comes up with a better name. The 90% Rule is simple: Generative AI will be able to do 90% of the work in areas that it is good at. Right now, that would be writing, but coming up soon will be image generation and coding and, not too long after that, video generation. For those of you buried in the space who are offended, I’m not rating it as advanced as it is for specialists; the point of generative AI is not to let AI specialists do things, but to let average people in the space do so. And that list is representative of the current space.

    This is just based upon the state of things at the moment I wrote this blog, but that seems to be the immediate order. I’m only talking generative AI here—AI for things like filtering test results is a completely different beast (that might just meet that 90% rule … Hmmm). I can tell generative AI to write a story, a script or code for me, and tweak the results. The amount of tweaking certainly depends on the topic and how much time I spent zeroing it in, but 90% seems right. There are those looking far ahead to a time that will herald the arrival of our AI overlords, but for the foreseeable future, we will not be ditching authors, coders and ops people. Our jobs will just change.

    Crikey—given the last few decades, a change to our jobs that means less work is not a terrible thing. Today, writers have to be publishers and marketers, coders have to be ops and ops have to be coders and the entire group has increasingly had to be security and architecture specialists, too.

    Let’s celebrate a tool that says, “Here’s a start, make it useful in your specific situation.” I know I will. I sometimes contract people to do initial research for me and send me links and short descriptions. I encourage them to use generative AI to do so. I do not contract people with my level of knowledge for this work, so they’re going to have things I need to read carefully and follow the links for—they’re not specialists, after all. Generative AI suffers from this syndrome, too, but that means there is no difference for me and they can work smarter, not harder. It’s not like they are writing content; merely providing introductory research through a scan of web properties.

    As time goes on, I expect they will become specialists. But not in the things I pay them to research—in making generative AI do what they want. I would argue the person I contract with the most often is better than me at it, just because she’s doing it regularly. That skill set will inevitably be transferrable, and I may lose her to a full-time position making the I in generative AI actually mean something, and I’m great with that. Meanwhile, I’m getting the background info I need to do my job(s).

    Embrace it. Aim for 90%. Turn out even more rockstar-level apps for the org, and maintain them with less effort. This is where we’re headed, and it’s entirely possible you’ll be less buried in work for some of that time. Win/win. Go out and enjoy. The business will thank you for being even more productive … then waste that extra time you created by implementing new projects. But that’s in the future; use AI in the short term to get some breathing room and still be the force behind the apps.

  • Oracle Previews Latest Java 21 Innovations

    Oracle Previews Latest Java 21 Innovations

    Oracle today made generally available a Java 21 edition and provided early access to more than 15 proposed enhancements to the Java development kit (JDK).

    Those enhancements were announced at an Oracle CloudWorld 2023 conference, including support for virtual threads to improve overall throughput and adding support for a vector application programming interface (API) that makes it simpler to build Java applications that involve artificial intelligence (AI) models.

    Georges Saab, senior vice president of Oracle Java Platform and chair of the OpenJDK governing board, said the goal is to streamline access to the latest innovations that will be added to future versions of Java.

    Specifically, via Project Amber, JDK 21 delivers improvements such as support for string Templates, record Patterns, pattern Matching for switch, unnamed Patterns and variables and unnamed classes and instance main methods.

    In addition to the Vector API, Project Panama adds a Foreign Function & Memory API. Project Loom, in addition to virtual threads, adds scoped values and structured concurrency support.

    There is also a more efficient garbage collection mechanism being proposed.

    Oracle is committing to offering long-term support for Java 21 for at least eight years to encourage organizations to start working with this version of Java today. Oracle is also making available a Java Playground, an online sandbox on the Oracle Cloud Infrastructure (OCI) service where developers can create Java code using this edition without needing a local runtime or integrated development environment (IDE).

    In general, Java has been experiencing a renaissance after more organizations began contributing to projects once the core platform became available under an open source OpenJDK license. In fact, the Java community is now adopting many concepts pioneered in other programming languages. That enables organizations to, for example, drive digital business transformation initiatives that depend on the modernization of legacy applications. As part of those efforts, the rate at which Java applications are being deployed in the cloud has substantially increased.

    Despite the number of alternative programming languages, many enterprise IT organizations remain committed to Java. Rather than opting to retrain developers, it’s simply been more cost-effective to enable developers to continue to build applications using a programming language they already know.

    Many organizations are currently in the process of shifting to Java 17. It’s not clear how many organizations will skip Java 17 and jump to Java 21 from previous versions of Java, but it’s clear Oracle is committed to making the lag time between moving from previous versions of Java to Java 21 much shorter.

    Most organizations will find themselves continuing to deploy Java applications everywhere from the network edge to the cloud. These applications will be written in a programming language that has persevered for more than two decades and continues to evolve and mature for the next era of application development.

  • Buildkite Acquires Packagecloud to Streamline DevOps Workflows

    Buildkite Acquires Packagecloud to Streamline DevOps Workflows

    Buildkite Pty Ltd today revealed it has acquired Packagecloud as part of an effort to tightly integrate package management tools with its continuous integration/continuous delivery (CI/CD) platform.

    Buildkite CEO Keith Pitt said the goal is to reduce the level of friction that exists between providers of third-party package management tools and CI/CD platforms that DevOps teams are today required to address themselves.

    There is no shortage of package management tools used to configure, install and update software, but the level of integration between these tools and CI/CD platforms has not historically been very high. The acquisition of Packagecloud will result in a package manager becoming a first-class citizen within the context of a CI/CD platform, said Pitt.

    Buildkite will maintain the existing support Packagecloud provides for platforms such as Jenkins, but the acquisition will streamline DevOps workflows and improve overall developer productivity, he added.

    In the months ahead, Buildkite will also add a range of tools to better enable DevOps teams to achieve that goal, said Pitt. In addition, the company plans to add tools to streamline the management of compliance processes as regulations pertaining to application development are becoming more stringent, he added.

    It’s not clear how much IT organizations are reevaluating their CI/CD platforms to improve developer productivity, but platform engineering as a methodology to centralize the management of DevOps workflows is gaining traction. Those efforts promise to reduce the total cost of implementing DevOps at scale and improve developer productivity by making the scaffolding required to build applications more readily accessible. Today, too many developers are spending time managing developer environments rather than writing actual code.

    The challenge, of course, is convincing developers—who tend to jealously guard their tool prerogatives—to embrace a platform engineering methodology and cede control over the development environment to a centralized team.

    One way or another, developer productivity needs to improve as more organizations encounter economic headwinds. Developers are the most expensive resource any organization has, so improving the rate at which high-quality code is created will always be imperative, noted Pitt.

    Today, organizations depend on software to drive revenue. Investors are assessing senior leaders within those organizations on their ability to deliver quality software in a timely fashion. Each new application and subsequent software update now has a direct impact on the profitability of the organization. As such, tolerance for legacy approaches to building software that require more time and effort is dropping.

    The issue, as always, is that switching from one software development platform to another is an expensive proposition, so many organizations may find themselves supporting a mix of legacy and modern platforms. The challenge becomes defining the pace at which that migration occurs as new applications are created and legacy applications are continuously updated.

  • What DevOps Teams Should Know About Phishing and the Supply Chain

    What DevOps Teams Should Know About Phishing and the Supply Chain

    The DevOps pipeline is a critical component in any organization, and it has broader implications that can affect other organizations as well. Some of the biggest cybersecurity breaches of recent years were supply chain attacks, in which attacks infiltrated the development systems of one organization to break into the systems of its customers downstream. Needless to say, this had catastrophic consequences for the organization under attack.

    According to Deloitte, 91% of cyberattacks begin with a phishing email, and supply chain attacks are no exception. By understanding this common threat and preventing it, DevOps organizations can significantly improve their resilience to these attacks. Of course, phishing awareness and prevention should come as part of a broader cybersecurity program that addresses additional supply chain risks. 

    What is Phishing?

    Phishing is a method used by cybercriminals to trick you into revealing sensitive information. They do this by creating a replica of a legitimate website or email and asking you to provide personal information such as passwords, credit card numbers, and social security numbers.

    Phishing attacks can take various forms. Some of the most common include email spoofing, where the attacker sends an email that appears to be from a trusted source, and website spoofing, where the attacker creates a fake website that looks identical to a legitimate one.

    Regardless of the method used, the goal is the same: To trick you into providing sensitive information. Phishing is a serious threat to both individuals and businesses. It’s not just about losing money; it can also lead to identity theft, damage to your reputation, and even legal consequences. Get more background in this detailed blog post about phishing prevention.

    Implications of Phishing for the Software Supply Chain

    Entry Point for Larger Attacks

    The supply chain is a prime target for phishing attacks. This is because it provides an entry point for larger attacks. For example, a phishing attack on a supplier could compromise their security, allowing the attacker to infiltrate organizations that use their software downstream.

    This kind of attack can have devastating consequences. It could lead to the theft of sensitive information, disruption of operations and even compliance risks.

    Information Theft

    Information theft is one of the most significant risks associated with phishing attacks on the supply chain. When a supplier falls victim to a phishing attack, the attacker can gain access to a wealth of valuable information belonging to the supplier and, in some cases, to its customers or partners. This could include everything from financial data to proprietary business information.

    The loss of such information can have severe implications. It could lead to financial loss, damage to the company’s reputation, and legal consequences.

    Disruption of Operations

    Another significant risk associated with phishing attacks on the supply chain is the potential for disruption of operations. A successful phishing attack could compromise a supplier’s systems or the systems of organizations that rely on that supplier, leading to downtime and delays in the supply chain.

    This kind of disruption can cause significant damage to a business. It could result in lost sales, damage to customer relationships and even the potential for business failure.

    Compliance Risks

    Lastly, phishing attacks on the supply chain can lead to compliance risks. Many industries are subject to strict regulations regarding data security. If a phishing attack leads to a data breach, the companies involved could face hefty fines and penalties.

    Additionally, a data breach could damage a company’s reputation, making it harder to do business in the future.

    Practical Steps to Secure the DevOps Pipeline

    1. Implementing Security Awareness Training

    The first and most crucial step in countering phishing risks is implementing security awareness training. This involves educating all team members about the various forms of phishing attacks and how to recognize them. Interactive sessions, workshops and webinars can be very effective in conveying this knowledge.

    Another vital aspect of security awareness training is teaching employees how to respond to suspected phishing attempts. They should know the steps to take, such as not clicking on any links in a suspicious email, reporting the incident to the IT department and deleting the email.

    2. Deploying Anti-Phishing Tools and Technologies

    While awareness and education are critical, they are not enough on their own to protect against phishing attacks. This is where anti-phishing tools and technologies come into play. These tools can help detect phishing attempts and prevent them from reaching the intended targets.

    There are several types of anti-phishing tools available, including email filters that detect and block phishing emails, browser add-ons that warn users when they attempt to visit a phishing website and AI-based tools that can recognize even sophisticated phishing attempts.

    3. Incorporating Security into the Software Development Life Cycle (SDLC)

    Incorporating security into the software development life cycle (SDLC) is another crucial step in countering phishing risks. This means considering security at all stages of software development, from planning and design to coding, testing and deployment.

    Security should be a primary consideration when designing software systems. This includes designing systems to be resilient to phishing attacks, for example, by using secure coding practices to prevent common vulnerabilities that phishers can exploit.

    During the coding and testing stages, it’s crucial to conduct regular security audits and vulnerability assessments to identify any potential weaknesses that could be exploited by phishers. Any identified vulnerabilities should be promptly addressed.

    Finally, during the deployment stage, it’s vital to ensure that all software components are securely configured and that any security patches or updates are applied promptly. This can help prevent phishing attacks that exploit software vulnerabilities.

    4. Regular Phishing Simulations and Drills

    Finally, conducting regular phishing simulations and drills is an excellent way to test the effectiveness of your phishing prevention measures. These simulations involve sending mock phishing emails to employees to see how they respond. The results can provide valuable insights into areas where further training or technical measures may be needed.

    Phishing drills should be conducted regularly and should cover various phishing techniques. The results of these drills should be shared with all team members, along with feedback and recommendations for improvement.

    Phishing simulations should also be used to test the effectiveness of your anti-phishing tools. If a significant number of mock phishing emails get through your defenses, this could indicate that your tools are not effective and need to be updated or replaced.

    Conclusion

    In conclusion, the integrity of DevOps pipelines—and, by extension, the entire supply chain—is increasingly under threat from phishing attacks. This is a grave concern given the potentially devastating implications ranging from data theft, operational disruption, legal consequences and considerable compliance risks. 

    However, organizations can proactively safeguard their DevOps pipelines by fostering a security-conscious culture underpinned by regular training and awareness programs. Deployment of state-of-the-art anti-phishing technologies and the integration of security at all stages of the SDLC can further strengthen defenses. 

    Additionally, regular phishing simulations and drills can be instrumental in testing and enhancing the resilience of these measures. The war against phishing is continuous and staying one step ahead of the threat requires constant vigilance, innovative strategies and robust technical defenses.

  • JFrog swampUP: Addressing the Advent of AI

    JFrog swampUP: Addressing the Advent of AI

    At JFrog SwampUp 2023, the buzz with all about AI, especially with JFrog’s announcement of Machine Learning (ML) Model Management capabilities. These conversations around AI and ML reflected these technologies’ growing influence and importance in the DevOps world. How much of the generative AI conversation is just hype, though? And what does that mean for AI and ML as a whole? Alan Shimel, CEO of Techstrong Group, and I sat down with Stephen Chin, VP of DevRel at JFrog, to find out.

    As far as Chin is concerned, even as more companies create and leverage AI models, these models must be managed like any other software component. Chin said JFrog Artifactory acts as a staging ground to operationalize models using DevSecOps best practices. Algorithms and models will continue to grow in size and complexity, and they will require robust processes around deployment and management – just like any other software artifact. The key, Chin said, is to think of ML as just another development language and leverage tools that standardize and streamline working with it. Compared to traditional enterprise applications, though, DevOps workflows for AI/ML are still relatively immature, but Chin said JFrog’s new model management capabilities aim to provide that missing automation and governance using DevSecOps best practices for governance, security, and licensing compliance.

    Additionally, Chin noted that AI/ML have become essential for development teams to keep up with the explosive demand for code. At this point, AI has become table stakes. In the AI arms race, the winners are those who understand AI has become a vital development tool to enhance productivity. In terms of job security, the losers are the ones who can’t keep up with the volume of code. According to Chin, you are out of the running if you don’t embrace AI.

    Looking ahead, AI will not make developers obsolete, though – quite the opposite. Given the quasi-unlimited appetite for new code, Chin emphasized that developers who embrace AI will have more work than ever. One way to think of it is that AI provides a new form of “outsourcing” to boost human productivity, much like previous waves of innovation in computer science.

    When it comes to security, there are still challenges that need to be addressed; code generated by today’s AI solutions still has significant drawbacks like potential data bias, lack of explainability and simple errors. In the long term, though, Chin believes AI itself will provide the solution to secure an exponentially growing codebase, given its superior scale. Just as AI will make individual developers more productive, it can also supercharge security teams – but it can also empower bad actors. The key will be continuing to democratize the benefits to even the playing field. AI promises to be a transformative technology on the scale of the Bronze Age or Quantum computing, Chin said, but the path forward will require humans and machines working together to ensure it’s used for good.

    It’s clear that the pace of innovation in AI and ML is rapidly accelerating. As these technologies become further democratized and integrated into developer workflows, they promise to transform how software is built and secured, Chin said. Companies must take advantage of this technology innovation by providing the pipelines and governance for this software revolution, he added. Chin believes the future will likely see a more symbiotic relationship between humans and AI assistants collaborating to generate, refine and safeguard code. By proactively shaping the responsible use of these technologies, it’s possible to maximize the benefits while minimizing the risks. While this journey has only just begun, the destination is an exciting new era of human potential unlocked by AI, Chin said.

  • Daytona Unfurls Developer Environment for On-Premises IT

    Daytona Unfurls Developer Environment for On-Premises IT

    Daytona has launched a development environment for on-premises IT environments that provides all the benefits of a cloud-based alternative.

    Ivan Burazin, CEO of Daytona, said the company’s namesake alternative to platforms such as GitHub Codespaces enables DevOps teams to retain control and better secure the intellectual property created by developers while simultaneously making them more productive.

    The Daytona platform enables developers to instantly start coding by accessing a development environment based on a set of preferred tools and frameworks, he added. Platform engineering teams looking to scale DevOps workflows can then take advantage of fine-grained access controls and advanced security features across developer environments spanning hybrid IT environments, said Burazin.

    He added that the ability to centrally manage development environments is designed to appeal to organizations attempting to improve developer productivity in highly regulated industries that are now allowed to use cloud-based alternatives. This is becoming a more pressing issue because as regulations become more stringent, organizations are trying to move source code off developers’ individual laptops that are more easily breached than a server environment, noted Burazin. The challenge is striking the right balance between enabling flexibility and centralizing the management of developer environments, he added.

    Of course, many of those organizations could opt to construct their own internal developer portal (IDP) to accomplish the same goal, but in most cases, that would prove expensive to both construct and maintain, noted Burazin.

    The same core team that created Codeanywhere, a cloud-based integrated development environment (IDE), is also behind the development of Daytona. The issue that organizations previously encountered when attempting to shift development to the cloud is that, in addition to regulations that forbid it, developers could not easily customize their environments. The Daytona platform is designed to enable more flexibility without sacrificing control, said Burazin.

    Arguably, organizations are more focused on developer productivity today than at any time in recent memory. Developers typically spend as much as 70% of their time maintaining environments versus writing code, so there is a clear need to reduce the level of toil. Reducing that toil doesn’t guarantee developers will spend more time writing code, but it does create the opportunity. DevOps teams are, as a consequence, being asked to eliminate as many of the bottlenecks that conspire to reduce developer productivity as possible in furtherance of that overall effort.

    Developer productivity issues are not going to be resolved overnight. Developers will not always be immediately inclined to give up control unless it’s conclusively proven that a platform engineering team will be responsive to their needs. The cultural challenge is finding that fine line between reducing friction while still giving the developer the sense that they can add a tool to a workflow that enables them to innovate.

    One way or another, change is coming to DevOps workflows. Each organization will need to determine how quickly that change will occur, but in the absence of any meaningful change, it’s clear they won’t be able to compete against rivals capable of producing higher-quality code faster.

  • DevOps Dozen² Awards 2023 Nominations Now Open

    DevOps Dozen² Awards 2023 Nominations Now Open

    We are excited to announce that the DevOps Dozen² 2023 nominations are now open. Now in its ninth year, the DevOps Dozen once again highlights the leading innovators in the DevOps world. 

    As we’ve done the last three years, there are actually 24 award categories. Twelve are aimed at the many great commercial DevOps solution categories in the market. The other twelve are aimed at the vibrant community and open source ecosystem which powers the DevOps movement. 

    As in years past, the selection process has two tiers. Nominations for finalists are open now and will close on October 20, 2023. We will announce the finalists on November 13, 2023. After the finalists are announced, open voting will be conducted to determine the winners in each category. Public voting accounts for 40% of the weighted final count, with 60% coming from the votes of this year’s panel of judges.

    This year’s judges are myself, Alan Shimel, CEO of Techstrong Group, Mitchell Ashley, CTO of Techstrong Group, Sharon Florentine, managing editor of DevOps.com, and John Willis, DevOps luminary and author. As we’ve done in past years, we are asking the 12 companies that receive commercial category nominations to pay a nominal fee to cut down on spam submissions and help cover the cost of tracking the nominations, trophies and other administrative expenses. Community awards remain free to nominate but spamming nominations will result in disqualification.

    We have not changed the categories from last year, though we did consider adding a category for AI in DevOps. Perhaps next year. Without further ado, here are the 24 categories for the DevOps Dozen awards:

    DevOps Dozen Tools and Services Awards

    • Best End-to-End DevOps Tool/Service
      Consolidation in the DevOps space has led to a greater selection of platforms offering end-to-end solutions. This award honors the platform with the most diverse and robust solution for DevOps teams.
    • Best DevOps Repo/GitOps Tool/Service
      GitOps has become an integral part of the DevOps toolbox. This award is presented to the most outstanding code repository/Git/GitOps tool.
    • Best CI/CD Tool
      This award recognizes the best CI/CD tool that helps teams to deliver quality code faster.
    • Best Value Stream Management Tool
      Value stream management has come into its own as an important tool for measuring the business value of DevOps and application development. This award recognizes the value stream management tool that most effectively helps organizations visualize the value stream, track metrics and measure improvement.
    • Best Observability Solution
      This award recognizes the best observability solution for understanding the performance of individual components, systems and entire applications.
    • Best DevSecOps Solution
      This award honors the best DevSecOps solution, which enables security to be included earlier in and continuously throughout the SDLC.
    • Best Testing Tool/Service
      Testing is a mainstay in the DevOps toolchain. This award recognizes the best testing tool for ensuring the quality of code.
    • Best Kubernetes Platform/Service
      The adoption of containers has paved the way for an entire solutions ecosystem. Kubernetes is perhaps the most widely recognized and utilized tool for container orchestration. There are many platforms and services that offer Kubernetes-based solutions. This award honors the best Kubernetes platform/service available.
    • Best Cloud-Native Security Solution/Service
      The cloud-native movement has taken the IT world by storm due to the depth and breadth of the solutions available and the benefits of building applications and services specifically for cloud-native environments. This award recognizes the best cloud-native tools, solutions and services.
    • Best DevOps for Mainframe Solution
      Despite their age, mainframes are a mainstay in the IT world, and a variety of solutions are available for enabling DevOps practices in mainframe and hybrid environments. This award recognizes the best of them.
    • Best DevOps for DataOps/Database Solution
      Databases often have been left out of the DevOps conversation, but that is changing  as the concept of DataOps is gaining greater traction. This award recognizes the best solution in the database and DataOps space.
    • Best New DevOps Tool/Service Provider
      This award recognizes the best new service provider or tool that has brought innovation to the DevOps space in the past year (must be less than 24 months old).

    DevOps Dozen Community Awards

    • Best DevOps Industry Implementation
      This award honors organizations that have successfully adopted and implemented DevOps principles and practices.
    • Best DevOps Survey/Analysis/Research
      This award recognizes research that has significantly and positively impacted the DevOps community.
    • Best DevOps-Related Video Series (video)
      This category highlights the most entertaining, educational and inspirational DevOps video series—those with the most engaging and highest-quality content.
    • Best DevOps-Related Audio Podcast Series
      Which is your favorite DevOps podcast from this year? This category celebrates the many great DevOps podcasts driving discussions about everything from CI/CD and testing to DevSecOps, Kubernetes, containers and more.
    • Best DevOps Book/eBook of the Year
      Each year sees a plethora of well-crafted DevOps-related books and ebooks. This category highlights the most popular and  most-read of the year.
    • Top DevOps Evangelist
      This category is one of our favorites! There are so many great evangelists who spread the DevOps gospel, but we want to know who does it best. Nominees can work within the DevOps space or outside of it.
    • Best DevOps Transformation (non-vendor)
      In a year marked by continued digital transformation, we want to know which organizations stood out in their efforts to transform their business through DevOps.
    • Best DevOps Event of the Year
      Virtual events became the norm throughout  the pandemic. But as more people crave face-to-face interaction, in-person events have slowly made a comeback. Which DevOps event stands out from the crowd?
    • Most Innovative DevOps Open Source Project
      Open source projects are meant to promote innovation and increase efficiency, velocity and quality. While there are many great open source projects out there, the best ones are those that create a community of people who work on and with the technology and support each other.
    • DevOps Executive of the Year (non-vendor)
      This category celebrates excellence among DevOps leaders at the C-suite (or equivalent) level.
    • Best DevOps Presentation of the Year
      This category recognizes the best DevOps-themed presentation from the many (virtual) conferences that took place this year.
    • Best DevOps.com Article of the Year
      This category celebrates excellence among DevOps leaders at the C-suite (or equivalent) level.

    Good luck to everyone nominating for the DevOps Dozen² Awards 2023. We will update finalists on November 13.

  • Splunk: Creating a Resilient and Dynamic Organization

    Splunk: Creating a Resilient and Dynamic Organization

    Mitch Ashley: I have the pleasure of being joined by Ryan Kovar, Ryan is a distinguished tech security technologist and leader of SURGe with Splunk, and Cory Minton who’s field CTO for The Americas with Splunk. Welcome, guys.

    Ryan Kovar: Thank you.

    Mitch Ashley: Good to be chatting with you both. A topic that’s very top of mind for CISOs or IT leaders, of course, is how do we create a resilient and dynamic organization that can keep pace with the business and the change that we see in the technology landscape, whether it’s our own infrastructure, own application portfolio, cloud, et cetera, all of those things, but also, of course, the tax services that are evolving and changing as what the bad guys, the threat actors, are doing. We’ve also not only got to have a great and resilient technology stack, but organizationally, process-wise, all of those things have to fit together into a cohesive strategy.

    We’re here to talk about that, thinking about it as an IT leader, whether you’re in security or IT or combination of both, and discussing your all expertise experience, but also you talk to a lot of customers as leaders in Splunk and the kind of things that you do. Cory, it’d be great to have you kick things off. Maybe we should start with if you want to start with the people domain or maybe you want to set it up a little bit differently. I’d love to hear your initial thoughts on this.

    Cory Minton: Yeah. No people’s perfect. I think the people process technology lens on tackling any sort of problem for leaders today is an appropriate framework. I’m happy to talk about the people portion of building great cybersecurity and IT organizations that, like you said, deliver resilience.

    Ryan Kovar: For me, at least when we think about this, you said IT leaders, and I think one of the big changes I’ve seen across for cyber for CISOs is it’s no longer IT or security. It is business leaders. When we started talking about people, that to me starts resonating because that is a cognitive change in how CISOs think of themselves and they think of their value, which we often get stuck in the technology part of people process and technology because a lot of us are technologists at heart, but I think the big change that Cory and I have seen when we start talking to CISOs and CIOs is this convergence of skills and the need to support the business differently.

    Cory Minton: Yeah, and it’s an interesting sort of people market, too. I think leaders have to think about the fact that, yeah, in technology, there has been some turmoil in some of the big tech companies, but there’s still a lot of really great talent out there. I think that there’s choice. With unemployment so low, and there’s, what, like 16 million jobs unfilled currently in the US and a lot of those being in the tech sector, people have choice and they have a place where they can go.

    I think, leaders, if they’re going to build a talent pipeline and build a great organization, they have to find folks where they are and bring them in and connect them to a value, a mission and a vision that they get excited about. Certainly, securing digital services and protecting against bad actors is exciting in and of itself for many folks, but actually connecting those people to how it affects the outcomes that matter to the business and that whole value creation process, I think, is a real critical skill for folks to understand today as they, again, are out communicating and interacting in the network, building networks of talent pools. I think that clear vision of why it matters and why the work that somebody would do with you matters is critically important.

    Ryan Kovar: That reminds me of the cliche, of course, that people don’t quit jobs, they quit leaders. When you flip that around, one of my mentors, Susan St. Ledger, told me, you can evaluate the success of a leader by how many people follow them to another company. I know right now, in this market, what I’m finding is people are staying longer because they like the people they work for or they like their leadership team, they like the culture. The money, even if they’re not making as much as they were last year because of inflation, the people I know who stuck around longest are the people happy with the people that they work for and work with.

    Mitch Ashley: It’s interesting when we talk about people. I’ve worked in part of my career where the sort of higher the purple unicorn that doesn’t exist, that got eight skills, that needs 20 years of experience for things that have only been around for three. We set our subs up to hire these phenomenal people of which there may only be a few in the world that are like that, but it’s about growing our people, but also growing our organizations, developing them. It’s not just hiring people. It’s building a team. It’s building an organization that’s got the right talent at the right time at the right place to match with the business needs, and that changes. That means you can’t always hire for it. You’re partnering for it. You’re working with companies like Splunk to bring in skills and expertise maybe you need for the moment, you need for a project, you need for strategy, all of those things. Love to hear your perspective about broaden what we think about people and how we incorporate that part of it into our strategy.

    Cory Minton: Yeah. I’ll say, top to bottom, start at the executive level down to practitioners. Everybody has a partner. Everybody has a consulting partner typically that they bring in to, like you said, fill those gaps where it’s a new skill, it’s a new technology, it’s a new capability that they’re trying to deploy for the reasons that they’ve chosen to do so. They’re valuable, but you may not have those skills internally. Training takes time and, oftentimes, just finding that partner that can help you solve that particular problem is incredibly important. I think, as the security landscape and, frankly, the tools being used to deliver and develop the latest digital services continues to get more complex, expecting to hire that skillset completely, probably not realistic especially as digital transformation objectives have an ebb and a flow of momentum and the amount of work being done on particular projects.

    Yeah, I see it as one of the top areas where, when I’m talking to CIOs and CTOs and CISOs, when they think about, “Hey, we want to deploy this new capability within Splunk,” or, “We’re, frankly, trying to figure out how to integrate better with our observability and IT teams to derive more value from the tools we’ve already purchased,” it’s oftentimes that people conversation of, “How do I actually leverage your talent to show us what good looks like and bring the experience from other companies similar to us that have already been down this path to, call it the cheat code, how do I do the thing that you guys already know how to do? How do I buy that capability?” Oftentimes, it’s through people.

    Ryan Kovar: My experience, I’ll go the other side of the halo and horns there down to just the nitty-gritty hiring. Recently building out my team, I intentionally carved out three slots for entry level, early career folks into our cybersecurity team, which is pretty rare for Splunk and pretty rare for a lot of security research teams, but the way I did that was really rewriting the job description to be embracive of second-career people. I actually really dislike the term “there’s a problem with the pipeline for hiring”. I think that’s completely false. The problem is not in the pipeline. The problem is we have a valve on incredible talent and gate-keeping who actually gets into the pipeline. We do see a problem with the pipeline when I start looking for folks who are underrepresented in cyber especially 15, 20 years down.

    What I don’t see is a problem of, basically, the reservoir. The reservoir is healthy. The reservoir is huge, but people either don’t feel like they’re welcome or they self-select out especially of cybersecurity. For me, a lot of the people problems that we have, if we can start working on the problems of today, future us, as a security leader, will be very happy if present me can help unlock that reservoir by turning open the valve by creating job descriptions and roles that are less around years of experience and understanding technologies that I can Google or generative AI my way through today.

    I think there’s just a lot of flexibility especially for folks who have critical thinking and communication skills. I can teach you TCP/IP. I can’t teach you how to communicate. I can’t teach you how to synthesize information. I need you to walk in the door with that. Something for me around people and these hiring gaps that we see is really about being more embracive of nontraditional cybersecurity and IT roles and then also facilitating them in ways they can succeed.

    Mitch Ashley: Well, and it’s also about hiring for what we need today, but also where the ball is going to be downfield when the ball lands. Part of that is hiring people who have demonstrated their ability to learn, adapt. Yesterday, they were TCP/IP expert. Today, They’re the security threat landscape expert. How did they do that? How did they get there? They learned it. It may have been on the job, but there’s also a lot of self-motivation and just skill in that learning, and that repetitive learning skill is I think something you can harness or leverage to accelerate their career as well as what you need.

    Cory Minton: Absolutely. It’s one of those things that, always, some folks get turned off. As Ryan said, sometimes there’s a valve, and one of those valves is like, “You must have a college education.” While I don’t necessarily believe that a college education is always the right thing, I think what it proves if you have one is that, as you said Mitch, you’re able to go through a structured learning process in an organization that’s institutional, understand dynamics and things and achieve an objective which was set before you that had some measurable outcomes that you had to deliver, which I think is something true for all of us in a corporate responsibility job as we have to operate in an institutional environment. We have to consistently learn new things. We have to interact with people around us. I think college is a good measure, but, as Ryan said, oftentimes, a second career is maybe even a more powerful measure of somebody who’s done that successfully.

    Like Ryan said, the technical skills are learnable and, frankly, the fun part is some of the technical skills are actually getting obfuscated by advancements in technology. As we think about things like, Ryan, you mentioned, generative AI, we can make the joke, but, candidly, wouldn’t you rather hire somebody who’s got incredible cybersecurity skills and understands the landscape more so than somebody who’s just really good at crafting queries because then, if you have somebody that has that domain expertise as technologies like generative AI and other sort of assistive technologies continue to evolve, then the domain expertise becomes the most important because then you start interacting on a natural language way and you don’t have to have those same technical skills to get there, which I think is probably one of the most interesting uses for generative AI is actually bringing out the barriers for technical ability to execute in a job like security or IT.

    Ryan Kovar: Prompt engineering will be one of the most significant requirements for entry-level and mid-level jobs by next year, in my opinion, categorically.

    Mitch Ashley: We’ve already been through one generation. It’s called search engines. Now, we’re doing it with generative AI.

    Ryan Kovar: Yeah. Yeah. No. We’re talking about this, but the reality is I’ve been doing cybersecurity and IT since 1999. Cory, there’s enough gray there. I’m sure you’re about the same generation. When I started, there was no Google and you had to read the Microsoft TechNet documentation, and then there was Google, and I put 65 CD binder in the trash and said, “Never again.”

    The fun thing that I always tell people that I’m mentoring or advocating for in cybersecurity is I’ve been doing this for 24 years and, of the 24 years, I have about four years of knowledge that’s relevant. I have 24 years of wisdom, but four years of knowledge that’s relevant. Everything else, no one cares that I can fix Exchange 5.5 driven pub EDB and I know how to defrag a Windows NT 4.0 server. That doesn’t matter. That’s one of the great things about cybersecurity and IT in the general is that you can become a subject matter expert in something very quickly and not have the bias of age.

    Mitch Ashley: Well, let’s turn our lens to the process part of it, and we’ve talked a lot about people and we could talk a lot more about it. There’s some great conversations that we’ve had already about that. We’ve got to have processes that are well-oiled to highly tuned, but can be responsive of incident management when things happen. The organization now has to operate cross-functionally, the stove-pipes, the things that we’ve lived with for so long. Now, we have to operate cross-functionally. We’re trying to tear those down, but our processes have to work across those.

    I’m interested in your all thoughts about how are organizations, what do they best do to adapt to what we need today so we can be speedier, respond more quickly and more reliably to threats or incidents or needs of the business?

    Cory Minton: Yeah. Do you want to to hit it, Ryan?

    Ryan Kovar: No. Please, Cory, go ahead.

    Cory Minton: Okay. I was actually going to say it’s actually not even an option anymore. We talk about that they need to do it. It’s not an option. If you look at some of the SEC’s recent rulings on the disclosure of material and incidents that happen for publicly traded companies in the US, you have to report that. You have to have that cross-functional view of how if a cybersecurity incident happens or an IT sort of incident outage or breach, any of these categorical things that happen, if they have a material impact on operations that would affect shareholder value, then you must disclose those things, and so now the impetus is on every publicly traded company to get this figured out really well. It’s got executive buy-in now that we’re going to tear down the walls between security and IT operations and our engineering teams developing our next digital capabilities because, anywhere across that spectrum that we have a process breakdown, whether it’s externally caused or internally caused, if it’s material, we better report it.

    These are no longer like, hey, it’s a good idea. This is like “test that theory and report the results” kind of stuff that the SEC will send somebody to jail over. We’ve already seen some convictions on previously publicly traded companies on misreporting things. It’s no longer a game. We’re serious about this. I think, from a process perspective, one of the things I’m seeing is reaching across the aisle. CISOs are talking to CIOs and CTOs more so now than I’ve seen in the last couple of years because they have to understand those impacts and they’re looking to organizations that actually already do some of them within the company that are already being trusted by different pockets of the organization. They’re looking externally for that guidance and help, whether it’s from consulting partners that are, hey, advising on security operations or advising on software development capabilities or technology partners.

    I mean, even the CISA, the Cloud Infrastructure Security Agency, their recent strategy update talked about one of the key pillars of their annual strategy for resilience included technology partners that were going to help them achieve that resilience. I think we have the impetus. The measurement is required now. There’s no longer a game, and I think executives are starting to understand it and they’re looking external to say, “Who can help me deal with this?” No matter where we land on what is material and some of the legal questions on how it’s implemented, we still have to respond and we still have to report some of those capabilities. Now is the time to start reaching across the aisle and starting to ask questions of your partnerships that you have in the org on how can you help us solve this problem.

    Ryan Kovar: I look at things like DevOps which still today has a little bit of a carve out separate. In a lot of organizations, DevOps is a slightly different place than maybe traditional security and, well, certainly than security, and then possibly even different than traditional IT engineering or infrastructure. In today’s society or today’s world of technology, a lot of the security issues that organizations are facing are in their DevOps pipeline, and so I find it fascinating what Cory said. The SEC has this term, material finding. You have to report a material finding.

    Now, I as a security professional may very well know what that is, but, frankly, the DevOps world has gone feral to a point where the security organizations are not a part of it and so the DevOps team are the only ones who understand what the security implications are. They’re the ones finding it and, because they’re in a DevOps mind frame, they’re not stopping and saying, “Let’s create an incident. Let’s walk this through.” No. They’re just fixing it and they’re moving it on. That’s how the cloud works. That’s how DevOps works.

    There is this essential need for us as an industry to really start reaching more across because we are being outpaced by DevOps home growing their own SecOps without oversight, without the wisdom of the security world, but we can’t stay in the way of the business, which is why that’s so important for us to go across and understand this because now there’s regulatory requirements and the real world is people are going to move forward whether they like it or not, so get on that train.

    Cory Minton: Yeah/ it’s funny, Ryan, you say that. I was reading the SEC reports from JPMorgan Chase. Jamie Dimon, their CEO, said in the letter to shareholders he had two sort of funny juxtaposed statements that used the same phrasing, which I found interesting. He said two things he could not overemphasize, one was, “I cannot overemphasize the need for cybersecurity in our organization. Everything must be secure,” and then later in the letter he said, “I cannot overemphasize our need to deploy innovative technology,” which is exactly what you said. If my CICD pipeline is continuing to push updates, but I’m not applying those cybersecurity principles that, like you said, was incident review and forensics and actually looking into it, then we’ve missed the boat on this. One of the key parts of this SEC filing is you not only have to report the outage, but you actually have to from an annual reporting perspective talk about your processes that you’re using to ensure cybersecurity in those areas. That’s one of those that’s like the CEO is saying this to shareholders. We better pay attention.

    Ryan Kovar: The CEO of a bank that touches 20% of every dollar in the world every day, right?

    Cory Minton: We cannot overemphasize.

    Ryan Kovar: I love that example because I look at DevOps as a perfect world where I have no fear at all that the DevOps team will identify and remediate security issues very quickly and then continue to do so because they’re not looking at the larger picture. Why did this get ingested? Why do we have this happening? Why is this CRO a threat actor you may not know anything about a threat actor. You may not understand that Tampa strawberry as per Microsoft has a really significant interest in your organization, and one of their TTPs is actually moving up the chain and actually jumping ahead for software supply chain.

    That is something that I would not expect a DevSecOps engineer to understand, but that is where the context of a larger security world, and that’s why this resiliency message, this is why it’s working across aisles. It becomes so important for the process. I honestly think the technology is usually the easiest part of this. The people and the process to implement and secure the technology, way harder, way less interesting than most people, except nerds like me, but much harder.

    Mitch Ashley: Well, and what you really bring to light is that the security specialists are going to understand the threat landscape much better. DevOps developers, et cetera, know their environment focused on the things that they are. You mentioned Jamie Dimon. Those two things, he’s expecting us, our CEOs are expecting us to figure out how we bring that together, how we make that happen so we don’t have feral organizations, so we don’t have processes that are brittle when things really fall apart or our environments get more complex. I mean, that’s an obvious thing. They are complex and they’re getting more complex, and no one understands the whole thing, so we’ve all got to pull together and say there are three dimensions to this problem, not one, and here’s the steps. We together figure out how we fix this.

    Great. Well, let’s talk about the technology perspective then. Obviously, Splunk being a great technology company, have been around for a long time. I remember finding Splunk on the showroom floor in the early days still doing black T-shirts then just like they are today, a great company that you all work for. Let’s talk about some of the technology side of what’s important and, again, thinking about not just internal people and processes, but also third parties that we’re using like the Splunks of the world in our organizations and how you can help with this challenge.

    Ryan Kovar: I’ll start really easily on this one, and then Cory will actually say something much better. People buy software to solve their problems. That’s it. That’s the easiest way when I pay people. I didn’t come from a sales background. I was operational. I was a threat hunter for the government, threat intelligence at DARPA, places like this. I didn’t think about why I bought software until I worked for a software vendor. It’s very easy. You buy software to make your life easier, to solve a problem faster or to solve it better.

    When I look at what Splunk does, a lot of our recent efforts are really around this incremental growth of just making people’s lives better, making people’s lives easier, cutting down the barriers to do their job faster because that’s what people need, whether it be security, whether it be observability, whether it be traditional IT engineering, that to me is really what we focus on here at Splunk which is why I’m still here after nine years. We make people’s lives better and we allow them to fulfill the mission that they’re actually being paid for rather than fighting the software that they’re buying and paying money for. That to me really is at the heart of what we do.

    Cory Minton: Yeah, and I think, if anybody’s seen some of the Splunk messaging in the market, we’ve really rallied behind this idea of resilience, and I think it actually sums up nicely the things that we do which I think, based on the conversation we’ve just had about the people and process, things that are challenging organizations, I think we’re in a really unique place to be a partner, to be one of those partner organizations and be the software that people buy to make their lives easier because we sit in this unique place of being one of the few organizations that actually unify, tear down the walls between security, IT operations and those feral DevOps teams that are out there building the next digital services because our corporate mission is around resilience, and we think of that as a simple statement of, “How do you keep all this digital business secure and up and running?” Simple statement, but, secure, not a simple thing to do, and up and running oftentimes equally on simple thing to do.

    Ryan Kovar: Sometimes at loggerheads.

    Cory Minton: Yeah. Exactly, a pearl of either sort of objective. There used to be this term we called a MOM. We all love MOM, monitor of monitors. It sits above. It sits at an enterprise level, at a higher level to give you visibility. I think that’s what folks have maybe struggled with that we see. Hybrid cloud is reality, right? There’s lots of clouds, not one cloud. Clearly, there’s leaders in terms of revenue, and each quarter they grow at different paces, but candidly, even Michael Dell said some years ago, “The cloud is not a place. It’s more of an operating model.” As you see SaaS being part of cloud journeys, yes, data centers are still run by organizations. They just are run now in a much more orchestration pattern similar to cloud providers, and that reality of lots of silos across different landscapes is creating challenges.

    If you use one tool that’s provided by said provider, then, yes, you can secure and keep up and running that one particular cloud real estate, but what about how it’s connected across the organization? How about the ways that applications traverse deployment centers? I think that’s what we really see as interesting is organizations are leaning on Splunk especially in the macroeconomic conditions where everybody’s focused less on growth and more on profitability, so they’re looking at like, hey, how do I reduce the number of tools?

    Ryan Kovar: More out of less.

    Cory Minton: Yeah. Exactly. Do more with less. They’re looking at it and going, “Wait a second, so I have this network environ, this router, so my data center. I’m sending the logs from that thing to nine different tools. Why am I doing that?” That’s nine times I’m paying for that bit of data to be stored and processed and analyzed by some number of tools, and organizations are looking at it and going, “Maybe that’s not smart. Maybe that same piece of data has nine questions being asked of it, and is there may be a fewer number of tools that could answer those nine questions effectively to give me the outcome without having to have all this sprawl?”

    It’s a unique conversation that we’re in today. Candidly, the tools consolidation, that rationalization of tools is one, looking at data. The data deluge hasn’t stopped. As we talk about sending data to nine different places, well, as that data becomes richer, it’s no longer just logs. Now we’re looking at metrics for real time. We’re looking at traces for spanning. How does an application actually impact multiple sort of environments? The data is getting rich and large. Is it all equal? Should we all be sending it to the same place? Should we be treating it with some valuation?

    Those are conversations that I’m having today that technology is helping solve, and Splunk is doing some really interesting work in the data rationalization and in that tools consolidation area that’s making it easier for customers to do more. As Ryan said, you bought a piece of software to make your life easier. We’re trying to go out and help folks figure out how to make more lives easier with the software they already bought.

    Ryan Kovar: I’m just required to hit all the buzzwords, so I’ll say ransomware now. It’s fascinating to me. Resilience, as in the concept of business leaders and cybersecurity, CISO leaders having to come together, I think, partly is really being driven by this threat of ransomware because the first time a cybersecurity threat has consistently and continually impacted every aspect of a business where 10 years ago, 15 years ago, oh, APT1 from China, exfil data. Well, that was the big issue. Well, by the time you heard about in the news, it’s already done. It’s actually relatively small scale. The impact is we were compromised, there was a breach, and now we have to deal with the public relations fallout, the stock drop, and also we have to make sure this doesn’t happen again.

    With ransomware, it’s actually we’ve encrypted your critical business systems and you can no longer do business, and now you have questions of do we pay the ransom? Well, now you need to involve the finance or you need to involve your chief legal officer. Oh, it’s publicly out there that we have this security issue. You have to involve the PR team, the public relations team. Oh, it’s actually knocked out our core business. Oh, well, now you’re involving the sales org. Now you’re involving the IT org. It’s coming across all these different places, so it’s builds back into what tools do you have that facilitate this and then, going back to the people part that we touched on earlier, this is where someone needs to be across all of them because they need to understand the impact and actually drive those changes that are needed and, because of ransomware, we actually have to deal with it. We can’t just pretend it doesn’t exist. CISOs can’t just be nerds in the closet playing cyber. They actually have to be able to speak eloquently to their business leaders, to the board of directors and others.

    Cory Minton: Yep. I’m just going to jump on the bandwagon of buzzword bingo. One of the things we’re seeing a ton of, too, is we talk about new digital workloads constantly being brought in, new innovation, we’re hiring consultants to bring in and actually help generative AI get real in our organizations, there’s a lot of conversation about is generative AI threat versus risk? There’s a whole kind of conversation there, but one that actually, as we’re moving past the hype and the AI washing nonsense that’s happened in the market, once we’re getting to real, a lot of CISOs have been talking about or talking to are trying to figure out how do I deploy those technologies internally? How do I go find the models that I can train against my data sets that would actually go help my teams do their jobs better?

    What I would say is is that’s another one of those areas where, as you’re innovating and bringing in new technology, treat it like another digital workload. You’re still going to have to detect when something goes wrong. You’re still going to have to investigate when something goes wrong in that generative AI hallucinates, it has an outage, whatever, and you’re still going to have to respond to those outages and take those learnings and put it back into the operational process of how do you run that system more effectively.

    I think people need help oftentimes finding those consistent patterns that we have to deploy to just keep things securely up and running regardless of what the workload is under the covers, and I candidly see that as one of the greatest areas for leaders that are looking to partner with a professional services organization or an assay, those outside consultancies. There’s a lot of value there in having them help bring in the patterns especially on proven technologies to short cycle, increase your time to value or, excuse me ,shorten your time to value when thinking about those new digital workloads and making sure that they’re secure and up and running.

    Mitch Ashley: I like where you’re taking this, which is one I wanted to wrap with, which is our organizations are expecting us to innovate, not just ourselves innovate, but also support the rest of the organization to be able to innovate. You talked about Ransomware. The newest thing is they’re not encrypting your data anymore. They’re just threatening to release it, so it’s not a matter of losing the data. It’s losing control over it. AI is the next, maybe it’s the next boon to phishing. We may not be able to tell the difference between an email from a generative AI LLM that’s been trained on how our CEO talks versus a well-scripted, predefined email.

    These new kind of threat models as well as new technologies force us to innovate and find new ways to do things. We have to help our people with tools, technologies, processes. I’d love to hear some more thoughts about what do we do to set ourselves up for success to be able to innovate in a way that’s going to help support where the organization is today and where it’s going?

    Ryan Kovar: Well, I’ll make a slightly controversial opinion here. I don’t think there’s ever been a single technology that outweighs the benefits for the offense or defense. I look at generative AI as something that is going to liberate security organizations from the shackles of mundanity. It’ll democratize what we can do in terms of analysis. It will also do the same for the adversaries. The only people who will lose are those who are not taking advantage of this technology on either side.

    I really do believe that we can all up-level as we go through with different technologies just as the adversaries are. When we invented a trench or trench warfare in World War I, they invented the tank. When we invented an airplane, they invented anti-aircraft. This is a tit-for-tat thing, and there’s always a leap at some point. People created a star fortress and that stymied invasions until someone invented the cannon. Sometimes, defense beats offense and sometimes offense beats off defense, but it goes back and forth as you go.

    I don’t necessarily see anything that’s a semantic change in how organizations will defend, but I do believe, if you’re not paying attention, if you’re not innovating, if you’re not selecting vendors, if you’re buying software, if you’re not buying vendors who aren’t innovating, you’re going to have a very difficult time in the next five years because your adversaries absolutely are.

    Cory Minton: Yeah, and I would go maybe less digital and say we as leaders cannot underestimate and cannot undervalue investments that we make in our people and our business culture today that create space for the learning and the development and the skills attainment that Ryan talked about, being critical. Leaders must create space for technologists, for practitioners, for leaders to go learn and to go study and have a mantra of spending some large amount of time on a regular basis, on a cadence basis, investigating, whether that’s going and attending conferences and networking and starting to understand what’s happening in the industry or it’s partnering with a technology organization and doing more hands-on workshops and learnings or it’s going next, when you’ve got a consultant in, spending time with those folks in the office picking their brain on what are the skills that they’re paying you for? Why are you here? What is so unique about what you’re doing, and how do I learn more of that?

    It comes from leaders making that a priority. If we’re going to have an organization that has people, process and technology that’s actually going to be resilient, that’s going to be able to survive in this next epoch of technology innovation, it’s going to have to be led by the best intelligence, which is still human intelligence, and that requires investment, and I think you got to put a stake in the ground and make that a priority today.

    Ryan Kovar: I love that part because, going back to our previous discussions about leaders who often have 10, 20, 30 years of experience, none of it is relevant. When I started my career as CISO, my only half, well, first off, CISOs didn’t exist. Second, the CTO or the person in charge of technology at that company had to pay attention to desktops, printers and servers on-prem, and then 10 years ago, 15 years ago, it was laptops, desktops, printers on-prem, printers in the home offices, a data center that they co-located to in this little cloud thing that they’re trying out, and then five or 10 years ago, it was all those plus software as a service, and now it’s, okay, well, we have a CICD pipeline and we’re driving APIs and our entire business is dependent upon this one piece of open source software that’s maintained by three guys in a former Yugoslavian Republic and, if it has an issue, we’re going to be done.

    Also, we M&Aed a company out of China. Every time they develop a vulnerability, that has to go to the Chinese government along with us. It’s a lot more complex. For those who are just resting on their laurels of “I’ve been doing this for a while” and not up-leveling their own education and hiring people who are subject matter levels as direct lieutenants, you’re going to have a bad day.

    Mitch Ashley: Bringing it back to the thinking that got us here is not the thinking that will get us to the next place, I’m paraphrasing Einstein here, but it’s also our own thinking. That’s the value of bringing in perspectives like yours with your partners, your suppliers, companies like Splunk, professional services. There are a lot of ways. I think we’re at the same junction point where we were with the cloud. Where we’re like, “Here’s the cloud. Is it just somebody else’s computer or is it a new way of doing things?” But what is that new way?

    We’re kind of in that place today with AI. Okay. Right. We know we can do generative AI stuff, but what does it really mean? How do we leverage it, to your point, Ryan? You have to be a participant on the field to really figure that out. You don’t have to go spend crazy money on the latest thing just because that’s what was in the airline magazine, but it is about figuring out and learning what you can do with it and what’s possible learning by yourself, but also with others.

    Well, thank you to both of you, to Cory Minton and Ryan Kovar, for joining us, and thanks to the Splunk team for bringing us together to have this conversation. We hope it’s been really beneficial to everybody who’s listening in. I know it’s been great for me as well. Thank you.

  • Should You Measure Developer Productivity?

    Should You Measure Developer Productivity?

    A recent McKinsey article entitled ‘Yes, you can measure software developer productivity‘ has got the software developer community talking. And the general consensus is they’re not happy.

    Most software developers are not enthusiastic about the prospect of having their individual performance metrics monitored. “Anyone who attempts to measure software in some sort of scientific way is just doing the wrong thing,” said YouTube personality ThePrimeTime. Dave Farley of Continuous Delivery even goes as far as to call the McKinsey report “harmful dribble.”

    This debate arises as developer productivity engineering (DPE) is gaining traction as a means to improve developer performances and, perhaps, in response to employment landscape shifts like the Great Resignation, which saw developers and DevOps pros leaving their roles. Companies like Spotify, LinkedIn, Netflix and others are also investing in internal developer platforms (IDPs) to increase developer productivity and satisfaction amid a limited talent supply.

    Alongside these ambitions, much has been written about how managers can measure developer productivity in terms of qualitative metrics, like developer satisfaction, as well as quantitative metrics, like number of commits or story points, deployment frequency or failure rates. But can tracking such metrics deliver tangible results? Or do they go too far?

    Below, we’ll consider whether you can actually measure software developer productivity. But more than that, we’ll evaluate whether this is a meaningful tactic with desirable outcomes.

    Ways to Measure Development Productivity: DORA

    First, let’s look at some of the standard techniques to measure development productivity. The DORA metrics, produced by the DevOps Research and Assessment (DORA) team at Google after years of research, identified four key metrics:

    1. Deployment frequency: How often you ship changes.
    2. Lead time for changes: The time between a commit and it reaching production.
    3. Change failure rate: The percent of releases that fail; downtime, worsened service or rollbacks.
    4. Time to restore service: The average time it takes to respond to an issue, also known as mean-time-to-recovery.

    These metrics could be helpful indicators of a software development team’s overall agility and health. They can signal bottlenecks in the CI/CD process or highlight where too much time is spent remediating issues, encouraging positive changes. Solving these root causes could all improve a team’s productivity.

    Going Beyond DORA: SPACE

    DORA also added reliability to the list. That’s understandable, given the interest in building reliable systems. But things start to get a little more complex when SPACE enters the equation.

    Proposed in a 2021 paper by GitHub researchers, SPACE augments DORA with the following attributes:

    • Satisfaction and well-being
    • Performance
    • Activity
    • Communication and collaboration
    • Efficiency and flow

    Metrics like satisfaction or communication and collaboration are pretty subjective. Plus, it can be challenging to gather this data and encourage engineers to speak honestly in surveys. Measuring the activity of individual members, like the number of issued closed or pull requests made, isn’t exactly tied to end value and could even produce a negative psychological effect.

    “Consequently, teams can become fixated on them, as seemingly positive changes are relatively easy to achieve,” wrote James Walker. He adds that SPACE still doesn’t tell you why measurements have value—furthermore, it lacks actionable information and might be tricky to implement, given developer pushback against performance analysis systems.

    Opportunity-Focused Metrics

    The McKinsey report adds to the mix what it calls “opportunity-focused productivity metrics,” which include inner/outer loop time spent, a Developer Velocity Index benchmark, contribution analysis and a talent capability score. (It’s not totally clear how these metrics can be computed). It then organizes all these metrics into a framework that further segments measurements into system, team and individual levels.

    McKinsey's framework for measuring developer productivity

    While some believe in the framework’s legitimacy, many commentators don’t see the practicality of implementing such a framework in practice. “Introducing the kind of framework that McKinsey is proposing is wrong-headed and certain to backfire,” wrote the PragmaticEngineer.com. “The report is so absurd and naive that it makes no sense to critique it in detail,” wrote Ken Beck.

    Things to Avoid With Developer Productivity Metrics

    Apparently, developers are very weary of being measured. And given the recent backlash, it seems like overreliance on productivity metrics could negatively impact the engineering culture. So, engineering leadership should be careful when implementing these sorts of tactics. Here are some anti-patterns to consider.

    Don’t make it about individual performances. Metrics like DORA have become standards for measuring the efficiency of software teams, but they shouldn’t be applied at the individual level. “Measuring software development in terms of individual developer productivity is a terrible idea,” said Dave Farley. Technology development is more of a collaborative process; performance metrics should reflect how the team operates.

    Don’t measure developer productivity in isolation. On that note, performance metrics shouldn’t be judged in isolation. Issues with failures, for example, often depend on contexts outside of an individual developer’s control. At the same time, it can be hard to correlate individual developer contributions to end customer satisfaction.

    Don’t create an opportunity to gamify productivity systems. Leadership shouldn’t directly tie a developer’s salary, reputation or position to whether or not they hit certain metrics. If productivity systems carry this much weight, they could become gamefied.

    Don’t put in more effort than it’s worth. In tech, where there’s hype, there’s money to be made—and a host of vendors ready to hawk monitoring tools. What’s more, to enact these productivity measurement enhancements, McKinsey noted that “most companies’ tech stacks will require potentially extensive reconfiguration.” Devoting many engineering resources to monitor engineers just seems like a big pill to swallow.

    Don’t put too much faith in metrics. Lastly, metrics only show one side of the picture. They are only data points and can’t always depict what it’s really like on the ground. Furthermore, qualitative responses also might be skewed due to reluctance to speak up.

    Remember Goodhart’s Law and Focus On Quality

    Personally, I believe that measuring things like developer productivity, efficiency, experience and satisfaction does have some merit for some businesses. Most teams are now working asynchronously, and metrics help leaders take the pulse of a team’s performance, even if they are all remote workers.

    It’s also nothing new. Leadership has been attempting to measure developer productivity for years. But, these metrics often lack context and come with many caveats to consider. Thus, they shouldn’t be given a huge seat at the table.

    I like this ‘law’ from British economist Charles Goodhart, which is colloquially summarized as “when a measure becomes a target, it ceases to be a good measure.” In other words, when you start working for the metrics, the engineering culture and overall code quality might decline. This matters, since a separate study found that increases in code quality tend to be followed by an increase in developer productivity.

    So, focus first on establishing code quality and, hopefully, productivity will flow.