devopsy.kinsta.cloud

Category: DevOps Culture

  • Real-Life BDD: Getting the Best Out of Dev Teams

    Real-Life BDD: Getting the Best Out of Dev Teams

    If we want our software to have value, we need to improve it every day, rather than wait three months for the next release. If someone finds a bug and we know how to fix it, we want to deliver the fix that day. The challenge is deploying to production many times a day.

    If the developer is coding right now, we want to see their work in production today. However, when you increase the speed of delivery, you need to be sure that the code you deploy works. To increase speed and quality at the same time, you need a safety net and a lot of automation.

    Sometimes we need to change the mindset of developers. This affects interactions between the developers and product management. It also changes the way developers code and how they approach their work.

    Break It Down

    Behavior-driven development (BDD) helps all of my teams. My mature teams use the three stages of BDD: discovery, formulation and automation. To increase speed and quality in a team that isn’t as mature (not much test automation, a lot of manual testing), one of our challenges is to develop small pieces of code they can deliver.

    When we groom an item in the backlog, the product owner (PO) explains what’s needed, but sometimes when the developers start working, they find other stories that may take two weeks each. With BDD discovery, the PO and developers can reach a shared understanding of what’s needed before coding starts. If the story is too large, example mapping helps them split it. They need to obtain stories that will take less than three days.

    More Ideas, More Examples, More Agile

    To simplify collaboration between the engineering and business units, the product team needs to understand the size of the ask for the developer. You can improve communication by showing all the different cases that exist for this feature and by using example mapping.

    Here’s an example of how this works in practice: the product team asks for a feature and explains it in about 10 minutes. The developers say this will take two to three weeks. However, after the developers completed user story mapping and example mapping and began to split the big features into smaller stories with examples, they realized it would take much longer. There were simply too many things the product team hadn’t known about.

    The product team had lots of ideas, but ideas take time to implement. Now that they had more examples, they could make better decisions about what was mandatory and what would be great but was not essential. Working in this way allows us to be more agile.

    The particular aspect of BDD that helps here is discovery with example mapping. I like to use user story mapping to see different versions of the same feature to understand the value brought by the feature’s different potential iterations.

    Whenever Possible, Push Live Code

    I’ve transformed the way my team works to help them to release more often. The key is to add a continuous integration server with the first automated test and a staging environment where the code is deployed many times per day. This allows us to automate the process from the code, so when a developer pushes the code, we have the beginning of a safety net. When the safety net says everything is okay, deployment is triggered to a staging production environment. The production can be a staging production, but it’s better when it’s live. When it’s not possible to deploy directly in production, then you should at least deploy in a staging environment.

    From the developer’s point of view, when they push the code, there’s an impact on production, even if it’s just within a staging production environment. For the next step, they need to create the first unit test each time they have to add code. In other words, every time you write code, you need to have an automated test at the same time. These tests should be run by the continuous integration server.

    Once you have that, the next question is about the size of the code you push. If you wait a few days to push or use another branch, you won’t have the value to see the increment of the code you’ll push.

    Developers Should Own Quality

    Quality has traditionally been the responsibility of the QA team, but it should be owned by the developers. As a developer, you can’t wait for someone else to read your code and tell you it’s okay and then talk to the QA team to make sure you didn’t introduce a bug. If more unit tests are needed, you should write them and you should be using test-driven development (TDD) and BDD. You also need a lot of feedback. You need to be the owner of quality, and help your team achieve it.

    To summarize, here are my top three tips:

    • Automate the process of deployment with continuous integration/continuous delivery
    • Create examples with all the stakeholders of the project together—product, development and testers
    • Create small stories that you can deliver in less than three days
  • Should You Hire a Clojure Engineer?

    Should You Hire a Clojure Engineer?

    Quite often, when developing applications, it becomes necessary to execute long-running code at the same time as performing other tasks. More recently, to run a program on the Java virtual machine (JVM) it was necessary to write the program in Java. Those days are long gone due to the wide variety of programming languages ​​for the JVM. Clojure is a general-purpose dynamic language developed by Rich Hickey for the JVM. One of the main goals of an engineer who speaks this language is to simplify the process of writing a program with multiple threads that are accessing data simultaneously. What, exactly, is the role of a Clojure software engineer and how can they improve the functionality of its products?

    What Problems can a Clojure Software Engineer Fix?

    A good programming language should help us write transparent and straightforward programs. The software systems that humans create are inherently changeable. In development, it is impossible to foresee all situations in advance, especially when the system’s complexity only grows over time. Even the slightest code change can potentially lead to significant changes in overall system behavior. As a developer, it’s necessary to study carefully, analyze the code and predict each shift’s side effects. Over time, and with greater complexity, it becomes close to impossible to thoroughly analyze all the possible consequences of our changes.

    The Clojure language was created to overcome these challenges. Clojure has a more convenient and modern syntax that orients the user to the functional paradigm as much as possible. Despite the dynamic typing of the language, Clojure programs are compiled into JVM bytecode and integrate seamlessly with existing Java programs. Clojure has now been implemented for several platforms, the most notable of which are the JVM, .Net, and JavaScript. Because it runs on the JVM, Clojure code is portable, stable, reliable and secure. Each operation in Clojure is implemented in three forms: function, macro or special form.

    Typically, complexities in computer programs cause side effects. This language helps to localize those and makes writing functions cleaner. This makes testing easier for the developer, too, since they don’t have to try different queues and find the correct state of the inputs.

    What to Look for When Hiring

    Large companies in particular value senior full-stack Clojure software engineers who can work across a number of tasks from analytics to coding, testing and DevOps.

    Among the problems that a Clojure software engineer can solve for an enterprise are:

    • Development of front end and back end systems
    • Creation of algorithms for load distribution by competencies/user workload
    • Design of modules for forecasting the load users create on systems
    • Machine learning in Clojure.

    Here is the example of a Clojure engineer’s résumé:

    Clojure is also an expression of a developer’s ideas about how simple a programmer’s tool should be. Clojure typically uses only four data structures: list, vector, display and set. The main API is built around these four structures. The main abstraction in Clojure is a pure function that has no side effects and is therefore much more reusable.

    To date, many libraries have been written for Clojure, including those for developing simple and modular web applications. A Clojure program is an open architecture constructor, which means developers can choose for themselves which libraries to use regardless of which others they may have chosen previously.

    Clojure Engineer Salary

    Salaries for Clojure software engineers in the U.S. average $129,887, while a Clojure software engineer’s salary in the UK is $69,910. In Canada, the typical Clojure engineer’s salary is $61,780 and in Germany, $69,089. Clojure software engineers in Ukraine earn an average salary of $38,000.

    Value to the Enterprise

    Enterprise software performance requirements dictate the need to use parallel computing to interact with each other and work at the same time. Existing languages ​​are poorly suited for parallelizing tasks. They contain many object and state dependencies, which translate to systems with unpredictable behavior in parallel interactions. Solutions based on locks and traffic lights do not greatly simplify the implementation of these tasks; the fundamental problem of the system’s state’s dependency on the sequences of object interaction remains.

    By using Clojure in web development, a programmer can develop using other minor, task-focused languages. ClojureScript, for example, compiles quickly to JavaScript and runs in the browser. All of this provides an entirely new approach to building client applications.

    Since Clojure seamlessly integrates with any Java code and can be deployed on standard Java application servers, it is well-suited to web application development, too. The flexibility of the language and its incredible expressiveness allows developers to create fully functional web applications very quickly.

    All a developer needs to work with Clojure is the JDK and the Clojure library which are contained in a single JAR file. There are two standard ways to develop and run Clojure programs: The most common way is to use a simple interactive REPL, a read-compute-print loop.

    Since Clojure is a general-purpose language, it is not difficult for a programmer to find a use for it. Since it is based on the JVM, there is already a massive selection of libraries available, which significantly lowers the entry threshold for developers skilled in other languages and removes the need for developers to write code from scratch.

    Clojure can be used wherever modern, efficient, stable data structures are needed. Of course, there is no one-size-fits-all tool that can handle all types of development tasks. However, the fact that a Clojure developer can use the language for client-server communication is yet another advantage.

  • Where Do SREs Go From Here?

    Where Do SREs Go From Here?

    Charlene O’Hanlon talks with Leo Vasiliou, director of product marketing at Catchpoint, about the results of a study the company fielded with VMWare Tanzu and DevOps Institute of nearly 300 site reliability engineers (SREs). This year’s report underscores the challenges of multi-cloud, calls out the underutilization of AIOps and shows a systemic shift in core baselining data. The video is below, followed by a transcript of the conversation.

    Announcer: This is Digital Anarchist.

    Charlene O’Hanlon: Hey, everybody. Welcome back to TechStrong TV. I’m Charlene O’Hanlon, and I’m here now with Leo Vasiliou who is the Director of Product Marketing over at Catchpoint. Leo, thanks so much for joining me today. Really appreciate it.

    Leo Vasiliou: Oh, of course, Charlene. Thank you very much for allowing us to have the conversation today.

    O’Hanlon: Absolutely, absolutely. I want to talk to you about a recent survey study that you guys put out in conjunction with VMware Tanzu and DevOps Institute. What were you guys looking at?

    Vasiliou: Well, from an existential perspective, we hope that the survey that we do leads into the annual report and we’d like for it to be the most data-backed set of insights in its community. There are sort of three underlying constructional essences of the report. One is to provide a core baselining data for the key SRE tenants. So DevOps, who had an amount of time, would call in reported levels of toil. The other essence is to look at what’s trending. And then the third one will generally be around the IT to business relationship. And then with each one of those constructional essences, the context and line items within each section will change from year to year.

    O’Hanlon: Gotcha. Alright. So this one focuses on site reliability engineering and what they’re seeing and what they’re dealing with on kind of a daily basis. Is that right?

    Vasiliou: Yes, yes.

    O’Hanlon: Okay. Alright, great. So tell me a little bit about some of the findings in the study. I know you guys do these, as you just said, reports yearly. So I’m interested in knowing what’s changed from year to year and what really surprised you in this year’s report.

    Vasiliou: Well, I like to say SREs are double struck with challenges because the SRE is essentially two jobs in one, right? They have to increase the efficiency of their operational activities while mitigating the risk of their transformational activities. So one of the key findings this year, which by the way this may be the most provocative dataset we’ve published just because some of the findings conflict with some popular theories in the market, but take the first one just as an example of that, is as part of the baselining set of data there was a self-reported drop in toil, which is the word they use to define repeated, mundane, busywork that doesn’t have any inherent value, of 15 percent. And one of the reasons this is such an insightful nugget to lead us into our various conspiracy theories is it was across the entire distribution of data, across all of their percentiles, as well as across all of the geographies that we looked at.

    O’Hanlon: Wow.

    Vasiliou: And so for me, when I had the blessed opportunity to do the research and sit down and co-write the report, having been stuck in the house for, let’s just call it a year and a half at this point, there was just something different about working on a piece like this and it just felt a bit more meaningful. And the conspiracy theory there or the hypothesis is that if people return to work, or return to the office I should say or some type of hybrid environment, will those reported levels of toil increase again next year as people adjust to the hustle and bustle of commutes just adding to frustration in how they think about things?

    O’Hanlon: That is a really, really fascinating thought process there. We could do an entire conversation about just that and whether what happens this year is going to be indicative of what happens in the future or if this year is going to be like a reset year. So from an SRE perspective though, the fact that the level of toil decreased 15 percent, that seems pretty significant. If you look at the entire spectrum of what they have to deal with on a daily basis, 15 percent, good lord, that’s huge in the SRE space.

    Vasiliou: Yeah. It is huge. I agree. It’s also based on the response of hundreds of survey participants year over year. And what’s interesting, Charlene, is one of the unique tactics we took in this year’s report is before we publish the reports and post survey pre report, we actually showed the results to various industry, let’s just call them, leaders. One of them we shared this particular section to had an alternative theory on why levels of toil dropped, and it had to do essentially with the fact that if people are not in the office, maybe does the very act of them not being observed compel them to do less busywork.

    So there was an alternative theory. But bringing us back to and leading into one of the other essences of the report, which is kind of the trends piece, it had to do with the amount of monitoring data sources, like the volume, variety, velocity of data, and the number of multi same-service platforms, e.g. multi CDN, multi DNS, et cetera, in use. And so will the return to work and the promise of AI ops help them keep toil levels low? Which leads us into the next sort of surprising finding, which is that SREs and their implementation or shift to AI ops is quite slow to maybe what we might be reading about in the industry, which was probably the second most provocative finding in the report.

    O’Hanlon: Yeah. So why do you think it is so slow then? Were there any trends that kind of came out of that? Any reasons why organizations are kind of slow to adopt AI ops?

    Vasiliou: Well, there is obviously a deluge of information and rhetoric in the market on the internet today. We need to seriously consider this idea of an information diet, which is a concept I first heard about a couple years ago. When we think about the idea of the hype cycle versus the challenges SREs face on a day-to-day basis, sometimes the market and the promise of what these types of solutions will offer might outpace actual implementation capabilities from, hey, we’re an SRE, let’s go implement this. I think part of the challenge here is probably the concept of AI ops is too broad and too large for consumption. It’s the idea of just saying, “Hey, let’s go solve world hunger or, hey, let’s go solve or implement AI ops,” right?

    O’Hanlon: Right, right.

    Vasiliou: You’ve got to break it down into smaller components and incrementally develop capabilities from there. Otherwise, how the heck are you going to baseline your progress for when you’re done? I think it’s too big, too broad, and some of the vendor promises may be of self-healing, auto remediation, maybe they were a little more on the hype side, where some of the other AI ops of components like root cause analysis or even correlation, maybe those are truthfully more on the hope and promise side.

    O’Hanlon: Okay. Do you think that AI ops – to your point, it is broad – do you think it just means different things to different people and so somebody actually may be implementing some type of AI ops and actually not realizing it?

    Vasiliou: The short answer is yes. So when we asked the question what monitoring tools or practices are used today, they were presented with seven options, ranging from application, infrastructure, experience monitoring, AI ops, and it was the fifth lowest, meaning that almost 40 percent of respondents said they’d never used it in the first place. The second question we asked is very simply what is the received value from AI ops, where one was the low value, nine was the high, and the range evenly spanned the value scale that was presented in the survey. What was interesting is there were a subset of people who said they never used AI ops but that they still received value, which was an additional dimension we didn’t think about when writing the survey.

    But it kind of brings me back to the essence of your question, which is are you user versus a consumer? And then go back to the previous comments about how big and broad this quote/unquote AI ops idea is, and there’s absolutely a propensity that people hear and think completely different things, unfortunately.

    O’Hanlon: Interesting. Interesting. So one of the things that we’ve been talking about in the SRE space over the last couple years is really how the role is kind of morphing and changing along with the market in general and to address a lot of organizational needs as well. Are there any findings from this year’s study that kind of bear out how an SRE role is changing and even if it is changing?

    Vasiliou: That will go back to what we were saying just a moment ago about mitigating the risk of their transformational activities. When you were asking the question, I’ll just cite the stat from our very first SRE report. We said that 85 percent of respondents said that in order to be successful in this role you have to commute into an office. So that got topsy-turvy turned upside down on its head.

    O’Hanlon: Right. Yeah. [Laughs]

    Vasiliou: But to get back to the essence of the question here, when we looked at the quantity of data sources and the prolific rise of the use of multiplatform providers, we sort of looked at this correlation between how SRE orgs are structured. Are you decentralized, are you centralized, or is there some type of hybrid in between? And then the decentralized piece was broken down even further by stack component, by platform component, or by business product or service. The vast majority of SREs said they are currently decentralized by business product or service, but I think the transformational piece here, which is something we’ll look at next year when we write the report, is because of the prolific rise of multiple third party and multiple same-service third party providers, there is a huge, gigantic spotlight to shift the primary dimension of org structure from product and service to a platform basis.

    And we talk about it like this. If you’ve got 10 different teams assigned to 10 different products or services and one member from each of those SRE teams developing capabilities that happen to be the exact same capabilities in development by those other nine teams, that’s the scalability and efficiency _____ that we start to lead in and talk about in this year’s report. So I think the transformational piece will be the assignment of SRE development engineering resources from product and service as a primary dimension to a platform – for example, cloud, CDN, DNS, API – as the primary dimension where those capabilities will be normalized and developed. Even though underlying platform structures will be different, those capabilities will then be offered as part of, let’s call it, an internal marketplace for use by multiple teams to kind of get away from that concept of silos and being segmented by product or service. So a bit of a long answer but all contextual to the key part there, which is the trend and the transformational piece that we wrote about this year.

     O’Hanlon: Right. Yeah, well it really is food for thought there if you consider how the SRE role was established and how it’s kind of – if you go down that path, how it seems to be changing and evolving as the more technology is adopted and more things are moved out to the edge and the need for SREs at basically all points on the network if you think about it. So great stuff there. Well, Leo, I know we could talk about all of the findings and have a much larger conversation, but unfortunately that’s all the time that we have for this one. But I do want to thank you very much for going through some of the survey findings. If folks are interested in taking a look at the study itself, is it readily available?

    Vasiliou: It is, it is. So it’s available at Catchpoint.com. Just head over to our learning and research section and enjoy the read. Charlene, one of the sentiments we wanted to leave readers with this year was this concept of baselining against Google’s definition of SRE, but we hereby declare by the power vested in us by absolutely nobody at all that you are hereby free to be an SRE.

    O’Hanlon: Oh, wow. Just one more thing to add to my plate. [Laughs] Well Leo, thanks again. I really do appreciate your time, and it’s a pleasure talking to you. I’m sure that we’ll talk again in the future about future studies that you guys have going on. So, thanks again. Really appreciate it.

    Vasiliou: You are quite welcome. Thank you for your time, Charlene.

    O’Hanlon: Alright, everybody. Please stick around. We’ve got lost more TechStrong TV coming up, so stay tuned.

    [End of Audio]

  • What It Will Take To Bring Cloud Gaming to Fruition?

    What It Will Take To Bring Cloud Gaming to Fruition?

    The gaming industry is booming. There are 211 million people gaming in the U.S., and it’s one of the fastest-growing forms of entertainment on the planet. In the age of Twitch streaming, gaming has even upstaged traditional sports viewership. On average, young gamers between the ages of 18 and 25 spend a full hour more watching online gaming than watching traditional sports.

    Gaming has also become a way of connecting with others—65% of players say they play with others online or in person. Some software development teams even encourage gaming in the workplace to increase productivity. For this reason, there has been a massive shift toward online cooperative games, RPGs and sandboxes like Minecraft. Also, games are becoming more cloud-based, with purchases, downloads and game saves occurring in the cloud. Still, many people lack the hardware necessary to run complex games locally. All these factors are harbingers of a coming shift away from PC and console-based processing.

    I recently met with Noam Salinger, product manager, Granulate, to learn about the future of gaming and how software teams in the gaming industry can prepare. To Salinger, “cloud gaming is the next step.” The move away from consoles and toward cloud-based, remote processing will have huge ripple effects. Lowering the barrier to gaming could introduce millions of more players into the ecosystem and further balloon the industry. However, cloud gaming presents significant hurdles. The transition will require a substantial reduction in latency, 5G networking and highly efficient computing optimizations.

    Roadblocks to Cloud Gaming

    “Cloud is an integral part of any gaming company,” says Salinger. More and more functionality associated with gaming is being pushed online. There are already online game marketplaces, cloud-based account management and remote game saves. This is especially prevalent with mobile gaming. But, what about the actual processing for computationally-intensive games—could these workloads also move to the cloud soon?

    This kind of cloud-based gaming could soon come to fruition. Yet, Salinger foresees some major roadblocks to cloud gaming:

    1. The Software Components

    A complete jump to cloud gaming will require having the proper software infrastructure in place and getting vendors on board to support it. This is perhaps the biggest obstacle to cloud gaming in the software industry, said Salinger, “It’s still not mature enough.”

    2. The Business Components

    Furthermore, gaming platform companies might refrain from cloud gaming due to fears of it cannibalizing their existing hardware sales channels. On the other hand, cloud gaming presents a big opportunity to take on more of a subscription-based model, such as allowing a user to play any game they want from an extensive suite of games. Akin to streaming services like Netflix, Hulu or Peacock, we could see similar platform styles emerging in gaming. “They will have no choice but to participate at some point,” said Salinger.

    3. The Network Components

    As most gamers can attest, online games can easily suffer from network issues that negatively affect gameplay. Salinger predicts that the ubiquity of 5G networks will be necessary to propel the cloud gaming market forward. 5G could be a universal method to enable real-time connections for a broader base of end-user gamers.

    Even if software developers optimize game processing speeds, a poor network could add enough latency to data throughput to cancel out those gains. 5G could expedite this process, but it’s taking a lot longer than expected. Some estimates place 5G finally arriving at full throttle sometime in 2023, after more antennas are constructed and pre-existing physical infrastructure is upgraded.

    4. The Latency Components

    Another significant roadblock to cloud gaming is figuring out real-time remote computing on the game vendor side. “This is where the latency aspect of things really comes into play,” said Salinger. Since game clients will be making many remote calls, optimizing cloud CPU utilization will be vital to reducing the response time to requests.

    Cloud gaming will require parallel running processes that anticipate user action. Similar to the way modern web applications simultaneously trigger multiple backend requests for authentication, geolocation, user databases and so on, real-time cloud games will require many different processes running in parallel. Servers could allocate resources before requests come in, using predictive analysis of usage patterns to trigger components in advance of user behaviors.

    To enable intelligent resource allocation, Salinger describes how an OS-agnostic kernel module could be in charge of resource allocation, continuously adapting to changes in the application workload. Low-level resource allocation at the CPU level, the allocation memory, I/O, and the sequence of execution must be optimal for each specific application, he described.

    This sort of process is already being applied in production by some ad tech companies to enable real-time ad bidding. For example, by optimizing cloud CPU utilization, Perion is able to perform faster, process more responses and use fewer machines to perform the same task—thereby reducing the roundtrip time for each request. Similar principles could be used to advance cloud gaming.

    Rendering sophisticated virtual environments is highly resource-intensive, and high-performance graphics cards are expensive. Video games are set to only increase in complexity with new AR/VR modes. Cloud gaming could help respond to rising computational needs and replace the need for local computing hardware. In the process, this move could bring in more gamers, disrupt the console wars, and nullify age-old PC-versus-Mac compatibility issues.

    Moving to a cloud-based approach is also arguably more aligned with today’s fast-paced software release cadences. Pushing more frequent patches can help resolve bugs faster. But it also could aid in retaining engagement by introducing new levels and objectives continually.

    The rise of the gaming ecosystem can’t be ignored. But as it stands now, there are still significant challenges to wider penetration of cloud gaming. Cloud computing optimizations to reduce latency, mainstream adoption of 5G and significant re-platforming of software infrastructure are just some of the many issues that will need to be tackled before truly cloud-based gaming experiences advance to the next level.

  • EP 14: What Have We Learned?

    EP 14: What Have We Learned?

    The past year-plus has brought disruption and opportunity to businesses and software teams all over the world. Change is inevitable, but 2020 was a year of massive change: businesses pivoted, supply chains were disrupted, customer expectations shifted to contactless services, digital transformation projects accelerated, new digital experiences were created, the move to the cloud jumped into high gear and our ability to create and deliver software was elevated to an essential capability for business.

    It’s time to pause and ask ourselves, what did we learn from all of this disruption and change? What lessons can we carry forward? How do those experiences inform how we prepare for 2021 and beyond?

    Cornelia Davis from Weaveworks, Kurt Chase from Tricentis and Sanjeev Sharma from Truist join hosts Alan Shimel and Mitch Ashley in this episode of DevOps Unbound to discuss lessons learned, how to overcome the obstacles we’ve faced over the past year and how to improve business processes and make better decisions. The video is below, followed by a transcript of the conversation.

    DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.

    Alan Shimel: Hey, everyone, this is Alan Shimel, and you’re watching DevOps Unbound. For those of you watching for the first time, DevOps Unbound is a biweekly video series where we tackle interesting topics related to DevOps. Once every four to six weeks, we do a live roundtable with a live audience questions and live audience participation, and we invite you to our next one. You can find that scheduling at DevOpsUnbound.com.

    Today, though, is a session just with our panel, and what a panel it is. Let me introduce you to them. First off, let me introduce our friend, Sanjeev Sharma of Truist. Sanjeev, maybe you wanna introduce yourself to the audience.

    Sanjeev Sharma: Sure. Thanks, Alan. First of all, thanks. Great to see you all. Hopefully, we will be able to meet in person soon, but thanks for having me on. I am the Head of Automation and Platform Engineering at Truist Bank. Truist, for those of you who might not know, is the new bank that was created last year by the merger of BB&T and SunTrust. I have various hats I wear at Truist, but my background is mostly in the DevOps and cloudware adoption space.

    Shimel: Excellent. Thank you, Sanjeev, and welcome. Sanjeev, I should mention, is also the author of The DevOps Adoption Playbook.

    Next, we’re really, really pleased and honored to have Cornelia Davis with us. Cornelia is, of course, a pretty well-known thought leader and leader at the DevOps space, but is also at Weaveworks. Cornelia, welcome. If you wanna give a little background?

    Cornelia Davis: Sure, yeah. Thank you so much, Alan. It’s so great to be on again. Yeah, I am the CTO at Weaveworks, and you might know Weaveworks as the company that coined the term GitOps, that has been kind of driving that movement. And GitOps is just an element of a broader category, which is really developer platforms, which is what we’re talking about many, many times when we talk about DevOps.

    And so, prior to Weaveworks, I’ve been here for about a year and a half, I was at Pivotal, where I worked on Cloud Foundry and Kubernetes based developer platforms, you know, platform teams for enterprises for quite some time. And I’m also the author of a book called Cloud Native Patterns, which is a book aimed at architects and software developers, helping them learn how to build software for this highly changing, highly distributed—constantly changing, highly distributed environment that is the cloud.

    Shimel: Absolutely, thank you. It’s a pleasure to have you here. Last but not least on our panel today is our friend Kurt Chase with Tricentis. I should mention Tricentis is a sponsor of DevOps Unbound, and thank you, Tricentis, for sponsoring. But Kurt, why don’t you give them your background?

    Chase: Yeah, thanks, Alan. Great to be here today with Cornelia and Sanjeev. Nice to meet both of you. I’m currently running Release Management and Engineering Services at Tricentis, which, Engineering Services is a group that provides cross cutting services primarily for internal engineering, but effectively for the whole corporation as well.

    Prior to Tricentis, I spent years at Splunk, again, supporting the engineering function, all the CI/CD builds and releases and things like that. And then prior to that, I spent almost two decades at Autodesk, producing the AutoCAD, multiple releases of the AutoCAD software product.

    So, again, it’s great to be here today with all of you. Excited to talk about some of the things we’ve learned over the last year, year and a half—and what a year it’s been, holy cow. [Laughter]

    Shimel: Yes, it has. Then let me finally introduce my co-host here at DevOps Unbound, my good friend and partner, Mitchel Ashley. Mitch, why don’t you give a little background?

    Mitch Ashley: Thanks, Alan, and of course, I’m excited to have our panel—thank you for joining us, all of you, today.

    I’m Mitch Ashley, CEO of Accelerated Strategies, an analyst firm that focuses on DevOps and cloud native, cyber security, digital transformation. I also work with Alan as part of MediaOps, the folks that put on this event, with the CTO of that organization. So, it’s a lot of DevOps practicing and observing what others are doing, so I’m excited to hear what Sanjeev and Cornelia and Kurt have to say, too.

    Shimel: Alright. Thank you, Mitchell. And as I mentioned, DevOps Unbound is sponsored by Tricentis, so—many thanks to them.

    Today’s episode, folks, as I mentioned off camera, I don’t wanna jinx anyone or anything, but at least here in the U.S., we seem to, at least we hope that we’re seeing this whole COVID life sort of starting to recede in our rearview mirror. We are returning to normal or a new normal, some may say recklessly too quickly, or some may say not quickly enough, but certainly, the return to normal is on, and accelerating.

    But, you know, there have been many, many lessons, many, many experiences, some painful—extremely painful—from a business and a humanity point of view over this last 14, 15 months or whatever it’s been. There’s also been a tremendous amount of success, a tremendous amount of acceleration, a tremendous amount of, “Jeez, this really—this worked. We didn’t know if it would work, but it worked,” probably the Internet being the first thing, right? Where would we be with this without an Internet?

    But can we—let’s take a moment, and let’s, as I say, hope and pray that this is in our rearview mirror or starting to move to the rearview mirror. What are the lessons? What can we learn, what will we take forward from this? And, you know, I don’t wanna boil the ocean, because I think there’s a lot, but if I asked each of you to give me three things that you really, you think—not just you personally or your organization, but the tech industry has learned, the business world has learned, perhaps society in general, humanity has learned, what would you say some of those things are?

    I don’t wanna put anyone on the spot, but Kurt, you always have something to say. I introduced you last, we’ll start with you.

    Chase: Sure. I’ll jump right in, and one thing that’s amazing to me, not so much from the technology or engineering or R&D side, let’s talk about business, here—the other aspects of the business that were able to adapt to remote work, that, to me, was pretty incredible.

    When COVID really took hold, I wasn’t too worried about software engineers and folks producing software working from home, doing their coding, testing, and all that. The other functions—finance, legal, things that are required to ship your products—seeing those go remote at the pace they did, that was pretty incredible. I think they definitely took advantage of some of the learnings from their individual engineering teams. But to me, that’s one thing that sticks out right away is the speed at which businesses adapted to working remotely and the success they achieved across the functions in the business. That was incredible to me.

    Shimel: Agreed. I mean, the speed of adoption was something. Cornelia, did you wanna say something?

    Davis: Yeah, sorry, I was muted there. I did wanna chime in, because I agree, and one of the ones that really knocked my socks off was how quickly call centers went remote.

    Chase: Yeah. [Laughter]

    Davis: Because that isn’t something I would’ve expected. I would’ve expected them to still be kind of really tied into their telephony services and, you know, the phones that are on their desks.

    But I spoke with many, many people, you know, various customer service people from the banks and from, you know, the Amazon.com—well, there’s not that much of a call in center there, but from all of these different organizations where, when you chatted with these folks, they were like, “Yep.” Within a week, they were able to move everything, move everybody remote in these traditionally really physical call center based systems. Which means that there had to be some preparation. I don’t know if it was intentional, but there had to be something that allowed them to flip that switch so quickly. Is it just forwarding phones? Is it just VPNs? I don’t know.

    Shimel: No, no, it was—so, I have some info on that, if I can. I interviewed a fellow down here in Florida, I think he was out of Tampa, and he operated multiple call centers, like in Costa Rica, which is actually a call center capital, believe it or not, that’s a big business down there. It turned out that the software they used, right, people in call centers, they don’t just—Kurt just doesn’t call someone randomly. Everyone gets assigned their calls or their inbound when a call comes in and gets assigned.

    The software they used within the call center itself worked just as well, the person didn’t have to be in the desk next to you, he could’ve been anywhere, it turns out, or she could’ve been anywhere. And so, it worked just as well in a remote thing. So, they still used the software they were using that tracked how many calls they did, who they spoke to, the notes, putting it into the central database. Yes, they had to be cloud-based, right, because there was no one in the call center to kinda run that infrastructure. But the software, it turned out, worked remotely as well as it worked on everyone’s desk in the call center. It was an amazing thing.

    And so, what they found—again, and this is gonna be, I think, something that’s gonna be a recurring theme—who needs the physical call center? They don’t need that infrastructure anymore. They could have people working at home just as well.

    And so, the more advanced ones—and you’ve probably seen this yourselves—you call someone, they may be very well at home, but they’re giving you the same mediocre service that you get when you call the call center itself, right? [Laughter] But yeah, no—

    Chase: And that’s the point. I think with a business, it’s applicable across functions there, it’s amazing. And business learned that they can still be extremely successful, even if everyone’s remote. And having that pool of remote talent—that changes everything. [Cross talk]

    Sharma: I think a couple of things to be noted there, right? First of all—sorry, Cornelia, I think I spoke over you. You know, one is, the pendulum also needs to swing back from that extreme.

    Chase: That’s the challenge, yeah. [Laughter]

    Sharma: Right? Because, while we have people who are able to work from home because the technology is there, from an equity perspective, one has to look at people who cannot work from home who are being forced to.

    Chase: Yeah.

    Sharma: And we experienced that when we opened up our offices. We are just beginning to open up our offices. June 1st was the first day folks would come back. And I was surprised to see some people had jumped at the opportunity and said, “I want to go back.”

    Chase: Yeah.

    Sharma: Then when you look under the hood and you talk to them, you realize, you know, they’ve been struggling. They’ve been operating out of a tiny closet, right, because they don’t have the space or because they have toddlers or they’ve been struggling with Internet issues, and it is worse overseas than it is here in the U.S.

    So, I think the pendulum needs to swing back to a more normal position, but hopefully, it won’t swing all the way back to where they say everybody has to be on site, on premises, right? So, I think that is definitely, some balance has to happen to recognize—some people do need a workplace, where they can work comfortably.

    Chase: You’re absolutely right.

    Davis: Yeah.

    Chase: I know a story of one individual who was in a one bedroom apartment in San Francisco and after three weeks, he couldn’t handle it. It’s like—I don’t have no space.

    Davis: Yeah.

    Shimel: It happens. Cornelia?

    Davis: And, you know, honestly, I have space, and I have a wonderful office and I’m super comfortable, and I’m also desperate to get back into the office.

    Chase: Yeah.

    Davis: Because—and I love, Sanjeev, that you called out the equity thing, because that’s something that is so painfully obvious in the whole pandemic is that this has only broadened the inequities that we have across society, and we need to absolutely focus on that.

    But then the other lesson, you know, with me, I’m extremely privileged, I live this super comfortable, I never really had to worry about catching COVID, I could just choose to stay home. But humans need humans and, you know, we’re all coming to you on your two-dimensional screen. I am super fortunate that last week, I had the opportunity to have my first face to face with a couple of colleagues in over a year, and it was rejuvenating. I’ve been saying, it’s a 3D work world again instead of a 2D work world.

    Shimel: Yeah.

    Davis: And so, I mean, humans need humans. But you’re right, Sanjeev, I think that the even bigger thing is that we really, really need to look at the inequities. And in the beginning, in the early parts of the pandemic, without getting too political here, we had all of this narrative around these people who we considered essential workers who were kind of the forgotten, like, taken for granted folks, and they were put on a pedestal, and that, unfortunately, didn’t last that long.

    Chase: Yep.

    Sharma: Yeah.

    Davis: And so, we need to bring that back.

    Shimel: And they weren’t paid like they were that essential, either, but that’s [Cross talk]—yeah, it was a pretty low pedestal. But let me, you know what, you bring two things—I’m sorry, Mitch, I just wanna make two quick points.

    Number one, when we talk about the inequality, let’s be really clear. This has affected women and mothers especially, more than men, right?

    Ashley: Mm-hmm.

    Davis: Absolutely.

    Shimel: Mothers have had—because not only have they had to work from home, but they’ve had to take care of the kids who are not in school. And it’s very hard being a teacher, a home care provider, and doing your work at work. And so, how many women have had to work out of the workforce because they had to take care of their kids and their kids are home and they didn’t have an option, there was no child care for them. So, I think—

    Ashley: And the challenges of re-entering the workforce.

    Shimel: Right, and now they’ve gotta re-enter after, you know, months off, and that’s never easy. People wanna know, “Well, why the gap in your résumé?” Well, there’s a gap in my résumé because I have this beautiful child that I’m raising, you know, and how dare you? That’s number one.

    Number two, it is a case of the rich get richer, right? Companies that, not only individuals as you say, Cornelia, but companies that were able to leverage this. Man, they accelerated, but they accelerated at the expense of their competitors who couldn’t, right? Think about all the open retail shops in the malls near you and the strips, right?

    Davis: And this is a place where I’d love to pull DevOps into the conversation, because we are, of course, as a part of DevOps Unbound. One of the things that I thought was impressive—and I’ve worked with some of these companies. In, you know, my previous life at Pivotal, I worked with The Home Depot, for example. When you had The Home Depot and you had these other organizations that, in a matter of weeks, updated their software so that you could do things like curbside pickup? That right there is DevOps in action.

    Shimel: Yep.

    Davis: It is the ability to release software, get feedback, release software, and there was no six-week testing cycle that we used to do, or eight-week testing cycle.

    Shimel: Mm-hmm.

    Davis: It was like, “You know what? We are gonna be testing in production, because we didn’t have time to test this for six weeks.” So, it brings in all of these tools that Kurt, I’m sure, you have day to day experience with, which is these things that allow us to put safety nets in place to be able to release all the time. I mean, that is something where the people who were along their DevOps journey already had a leg up on those that didn’t.

    Ashley: You know, Cornelia, I had that exact experience where I couldn’t get a computer, you couldn’t get one shipped to you, they just weren’t there, so I had to build a computer for some video work, and did it solely from parts from Best Buy, by curbside pickup, two Best Buys in town.

    Davis: Yeah.

    Ashley: So, it literally was, like, a week later, all of a sudden, I could do this, and I was pretty shocked by it.

    One of the things I wanna say, though, about what we’re discussing is, the people dimension of it is extremely important. I think there’s a whole other set of lessons for the business. I think we’ve pressed reset on a lot of assumptions about how businesses have to operate, that you have to be in an office. And essentially, we’ve kinda globalized the business by just taking the location out of it now. And yes, there are privileged and underserved and people who aren’t privileged. And those inequities probably are gonna get bigger, not smaller, because I don’t think we’re gonna all slap back into the office immediately.

    But what the businesses learned is that lesson you just talked about with Home Depot and you with Best Buy is—you know what? We can decide we’re gonna pick Team or Slack in two days instead of two years. We can get a software release out in a few days if we have to. Talk about constraint based management—you know, we now have this constraint and everything else comes off.

    So, I think what it’s done overall, though, is for DevOps and for all of us is, it’s moved software from maybe second or third to number one. Our ability to get software out the door that’s effective, meets what we need it to do, but that is a strategic advantage, maybe even a survival advantage of what we’ve seen happen in this past year.

    Chase: Yeah. I think some of it is, you know, businesses like that were forced to do it or die, basically, you know? And I 100 percent agree with Cornelia that I think they look to DevOps and some of the practices that they’re all used to. You know, the collaboration we need across functions to move forward and all those things are key, some of the foundations of automation, you know? That has been key in the DevOps space, and I think they learned, they need to take advantage of automation to do these things as well.

    So, yeah, it’s been fascinating to see that. And that was a point I brought up to see how businesses transformed and our world changed. And I think it’s opened up endless opportunities for us moving forward. And to Sanjeev’s point, I hope when that pendulum swings back, corporations take a look, that this is no longer a corporate decision where people work, it has to be more joint. It has to be more personal, because every situation’s different, and if you wanna maximize all your programs in HR, diversity and all those things, you have to take that into account. I don’t think you can do a one size fits all any longer.

    I still wonder if all those big office buildings, I worked in downtown San Francisco for many, many years—are we gonna fill those up again, ever? I really wonder now if that’ll ever happen. I could see it not happening, quite honestly.

    Ashley: I’m curious, Sanjeev, you’ve been in the middle of this bringing two companies together with this new bank in the middle of all of this. What in God’s name was that like? [Laughter] What are the additional challenges you’ve had for that, or maybe what opportunities has it created?

    Sharma: Yeah, so, I think it—you know, I joined in the middle of the merger, right, so we were in the middle of it when I joined. So, what I can tell you in the 10 months I’ve been there, we have shifted a lot of the priorities because of the pandemic, right? A lot of things which we would have done later on, we did first. We were able to do things.

    But the flexibility has definitely opened up. I can tell you, between the time I joined and between now, the ability to hire people who are not local to one of our offices has gone through the roof. It used to be an exception, now it’s the norm, right? The last three people I hired, two are in the West Coast and one is in Minneapolis, right, so Midwest. It doesn’t matter anymore, right? The same way there are—the operation has opened up a lot.

    And when I look at it broadly, outside of who I’m working with, when I’m talking to other companies where I have friends or colleagues, they are seeing more offshoring happen, right? Because they’re going, “Why do I need to hire somebody in L.A. if they’re never going to see the person ever anyway, does it matter whether he’s in Bangor or Bangalore, right? Let’s put the person, let’s hire the right person at the right place.” So, I think a lot of that has opened up. It has, of course, shifted some of the salaries, also.

    So, you know, a lot of things are going on, interesting dynamics are happening, and we’ll see how that all pans out. But from a perspective of, you know, doing the transformation, it is hard when you can’t have people in the same room on a white board and things which could have just drawn on a white board and knocked out in a 1 hour or 90 minute working session, now it’s like 15, you know, Zoom views, WebEx or Teams calls, which could have been, and it’s like, “Where is that white board we made up? Was it on Miro or was it on Mural or was it on this or was it on that? You know, and who can edit it, who can’t edit it?” All those—you know, dang it, if we just had a white board and all of us in the same room, we could have knocked this out in half a day.

    So, it is difficult to do certain things remotely. I think there has to be a balance, which needs to happen. And yeah, during a merger just made it more complex, because we are going through change every day, right? People’s networks are changing, people’s—how we are doing things are changing, we are updating stuff. It’s been a very interesting few months and we’re not done yet. You know, the merger isn’t over. We’ve still got a long way to go. So, it’ll be more fun going forward, I think.

    Shimel: I think it brings up the issue, though, of how much of this sticks, right? Because, I mean, look what Sanjeev just described. We’ve all had similar experiences, I’m sure, hiring people from all over the world—what difference does it make? They don’t have to come in the office.

    Does that stick? Because if it sticks, there are profound implications for society in general. Why live in San Francisco unless you love the view of the Bay. It’s a beautiful bay, right, the bridge is nice, but pay $4,000.00 for a closet-sized one bedroom apartment? Maybe not. I mean, on the other hand, I live in South Florida. Housing market is insane because of people moving here from New York and California and spending a fortune—Colorado, right?

    So, does this—that’s one example. Does the ability to work from anywhere, it makes no difference, is that something that sticks around, or do we, as Sanjeev said in the beginning, see a pendulum start to swing back? What do you think?

    Chase: Well, for me, like I said, I think elements of it have to exist and remain. I think there needs to be a balance, as Sanjeev said. I no longer think it’ll entirely be a corporate decision where employees are located. I think for some roles and groups in organizations, there needs to be a deeper conversation on where is the best talent, what’s the best way to use them? You know, having people sit—just the commute. The one thing I don’t miss is the commute. Oh, my God, you know, those three hours a day commuting, you know, that, to me, we have to somehow retain that benefit, you know, from working remotely somehow. I’m not sure how.

    But that’s a very good question, Alan, I don’t know. It’s like, will those office buildings ever be full again? I don’t know. I’m not sure.

    Sharma: On that point, I’ve been remote since 2013. In 2013, I was at IBM. I moved to a global role, and since then, I haven’t had an office. I’ve worked out of this room since then. But I used to travel 150 to 175,000 miles a year.

    Chase: Got it, yeah.

    Sharma: Right? So, for me, the office was a place to go collaborate, whether it was my office or my company, IBM—

    Shimel: And for all of us.

    Sharma: – right, or a client’s office, it was not a place to go and have a desk, right? My desk is here, right, my books are here—you know what I mean, right?

    Chase: Yeah.

    Sharma: I think we should, I would like to see us go to that model where offices become more of a collaboration space rather than a place you have to show up and work alone in a cubicle. What’s the point of that?

    Ashley: I agree.

    Sharma: Unless somebody needs it, right? That’s different if they don’t have space at home and the need a cubicle to work in—fine, put them in a WeWork next to where they live, right, close to where they live rather than have them commute to the other, you know, 45 minutes one way.

    I think that’s the model we should head towards.

    Chase: I agree.

    Sharma: And, you know, the tools to enable that are there, we’ve proved it. We’ve proved, during the lock down, during the pandemic that the tools are there, but I hope it goes that way. I hope it doesn’t go back to, “You have to show up in this cubicle” even if you are, you know, like the guy from Office Space protecting the stapler and never talking to anybody.

    Ashley: The butts in seats model, right? [Laughter]

    Chase: We’ve seen some large organizations [Cross talk] make those calls already and say you can be remote forever now, you know, things like that. So, I think that’ll become a bit more selective, but I still hope, when that pendulum does swing back, that the employee has a say, you know, and his voice is a little bit taken more seriously.

    Shimel: Or her voice.

    Davis: Or hers.

    Chase: Or hers—or hers, [Cross talk].

    Davis: Exactly, that’s where I’d like to chime—I’d like to chime in there, because Alan teed this up earlier and then, because you had two things, Alan, we followed thread number two.

    I wanna come back to thread number one, which is that this has affected women in the workforce, and it’s not just us technologists, but it’s a broad spectrum, and it’s probably even more so in other working arrangements. And so, I love what you’re saying there, Kurt, about it being a partnership between the corporation and the employee and determining what’s best for them. And then Sanjeev, you, I think, just teed up—is that the survival answer for WeWork is that part of that negotiation between the corporation and the employee, she or he, includes having the ability to move into a workplace that is close to your office but allows you that space that you need, for example, not to be on call for child care.

    Of course, the other thing I can’t help but say—and I’m very fortunate, I have one son, he’s 26, I’m not tying shoes or going to PTSA meetings anymore—

    Shimel: I hope not. [Laughter]

    Davis: – so, I wasn’t affected in that way. Yeah, [Laughter] I wasn’t affected in that way in the last year, but I was a working mother and I worked from home that whole time. And I just wanna remind everyone, men and women, that you’re right, women take on that burden, but it doesn’t have to be that way, fathers.

    Shimel: Amen.

    Chase: Mm-hmm.

    Cornelia Davis: So, please pitch in even more than you think, just because it’s been kind of the standard operating procedure. If nothing else, I hope the last year has maybe, in some cases, resulted in a, hey, a need-based way of rebalancing responsibilities. That is something that every single individual, male or female, can participate in kind of these societal expectations and challenging those.

    Ashley: I think something happens—

    Shimel: You know what? Oh, go ahead, Mitch, I’m sorry.

    Mitch Ashley: When things open up, when opportunity happens and things make that shift, it’s different when there’s opportunity there, but when things have shifted and made a shift, they don’t snap back to the old way automatically. It transforms sort of the landscape going forward, and there’s so many things about this—and you mentioned women in the workplace, we also have people of color—and we think about the emphasis of diversity that’s happened over the last year and a half, which is fantastic. And we see so much more bringing to the forefront of women in leadership positions, people of color being involved in security more, whatever it might be. You know, I don’t think that that changes.

    So, hopefully, those people who now are getting to participate in a bigger, broader way or have more opportunity to do that, (a) I don’t think this is gonna go away, (b) that’s gonna change things for us, right? You know, maybe, Cornelia—[Laughter] here, I’m speaking about what I don’t know about—but maybe more women in leadership positions make some of those, it’s assumed that the woman at home does the child care or whatever is part of our societal transformation, that it isn’t necessarily an assumption. Hopefully we can get past that and remove that barrier for people so there is more opportunity.

    Shimel: So, Mitch, on that front, though, here’s the deal. Number one, I think the move to diversity is not necessarily COVID related, I think it’s just an idea that its time had come, it has been a long time coming, number one.

    Ashley: I totally agree.

    Shimel: Number two, I think one of the positive things that happened is, it wasn’t just the woman stuck at home taking care of the kid. She was stuck at home taking care of the kid while the husband was trying to carve out a desk space in his bedroom somewhere to try to do a meeting or something, right?

    But what often happened, at least among my friends—look, my kids, also, Cornelia, they went away to college even though their college was online, just to get out of the house from their mother and father who wouldn’t let them breathe. But most—like, people I know whose kids were younger, the mother and father were home. They had to split up, delegate duties. You did the wash, I cook the meals, you clean the table, I took the kids here, you took the kids there or entertained them now because you had work to do, I had work to do.

    In a best case scenario, it became very much a shared responsibility. Of course, not every scenario or every instance has that best case outcome. There’s a lot of people [Cross talk], you know?

    Ashley: I think it’s been a journey to share. It may not be quite as, like, “Okay,” [Cross talk].

    Davis: Yeah, yeah.

    Shimel: Well, that goes, this is—

    Davis: I mean, you’ve painted a fairly ideal picture.

    Shimel: That is an ideal picture, you’re right.

    Davis: Yeah. You know, this conversation is making me wonder, and I don’t have these stats, but I wonder if anybody else has heard the stats—you’re right, absolutely, that this shift towards more underrepresented folks in leadership positions, women, people of color, and so on, that absolutely started before COVID.

    I wonder how COVID has affected that. Have the numbers accelerated, have they decelerated? I think—I suspect the latter, and I’m pretty much an optimist, but I’m also a bit of a realist, and I wonder whether the last year has slowed down progress when it comes to diversity at the leadership level. I don’t know. It’s just a gut feeling. I’d love to see some metrics on that.

    Sharma: I can tell you, Cornelia, in IT roles, it has always been very difficult to hire people from minorities, right, underrepresented minorities.

    Davis: Yeah, of course.

    Sharma: It always has been. I personally believe it has gotten worse, because if you look at especially leadership roles, senior roles—we’re not talking fresh college grads—most of those roles, no matter what anybody says, happen through referrals and connections, through your network.

    Davis: That’s right.

    Sharma: And that network stalled. That network stalled during the pandemic, right?

    Shimel: Yeah, networking, yeah.

    Sharma: Right? It is as big as it was then or grew slightly, right, that networking opportunities were gone.

    Davis: Yep.

    Sharma: No Meetups, no conferences, no lunches, no power breakfasts, no women in technology meet ups—nothing. The network stalled—you only knew the people you knew, right?

    Davis: Yep.

    Sharma: And I think that hurt, bad. I find when I’m trying to hire, when my peers are trying to hire, we are struggling more to have—see, you can’t hire when you don’t receive the résumés. When there are no résumés coming in, how do you hire, right? And I’m struggling, and if anybody has, if either one of you could—any one of you has any suggestions on how to fix that, I’m all in. Please, I’m all ears.

    Davis: I think—

    Chase: Yeah, I was gonna say, you pointed out an anti-pattern to what I was gonna say, Sanjeev. I was gonna say, at least with COVID and all the remote work, our talent pool has opened up worldwide, but you’re saying that hasn’t helped in that regard.

    It has for me with some of the roles I’m hiring. Now granted, they’re not Director, Senior Director, VP level roles yet. But for the other roles, having the world as my talent, you know, the talent pool versus the United States, that makes it [Cross talk].

    Sharma: Well, I look at it a bit differently. I look at it a bit differently, Kurt. You know, hiring—how do I, I wanna make sure I say this properly. Hiring an Indian in India is not hiring a minority.

    Chase: Yeah, true. Yeah, true.

    Sharma: Hiring an Egyptian in Egypt is not hiring a minority. Hiring a Turkish person in Turkey is not a minority, right?

    Shimel: Right.

    Sharma: They have to be a minority in where you’re hiring them, that’s the only way it works.

    Shimel: Agreed.

    Davis: Yeah, I think, [Cross talk] I wanted to comment on your remarks, Sanjeev. I think that you’re spot on. Because there were opportunities, and some of the techniques that we have used in corporations to source diversity candidates or underrepresented candidates, those things went away. So, being able to go to a historically black university or go have your recruiter show up at Women Who Code meet ups, those went away, and those aren’t the same.

    Sure, Women Who Code still had meet ups line, but it’s not the same as walking around during the social hour and saying—or, I’ve been at many meet ups where, before they start the program, they say, “Okay, who’s hiring?” And everybody stands up and then, “Who’s looking?” Everybody stands up. “Alright, you folks, talk.” That was a very organic thing that was not just so easy to say, “How do you do that in an online setting?”

    You’re right, it’s—I hadn’t even thought of that, but that whole network is the same now as it was a year ago is remarkable, especially when it comes to leadership.

    Sharma: Exactly. That’s why hiring from a historically black university, we can still do that today, because the recruiters still know the counselors there, they still know the placement office or whoever does—you know, those networks are still there. It’s just that the network hasn’t grown. The higher you go, right, the air gets, you know, rarer, right, and those jobs are fewer and you are not screening 100 résumés to do that. You’re relying on the network more than often, no matter what anybody else says, right?

    And that network stalled. I mean, who have we met at a conference who we didn’t know before? Now, when I go to a meet up and I see somebody I know—

    Davis: Yeah, but I think what you’re pointing out—

    Sanjeev Sharma: – I am IM-ing that person, not some stranger who I’ve never met, right? Unless there’s something really, some reason to do that. Then—no, no, it is the opposite, it has really hurt. I think we need to fix this.

    Davis: Yeah.

    Sharma: We need to put this [Cross talk] the moment we come back.

    Davis: Yeah, but I wanna suggest that—yeah, but I wanna challenge that assumption, which is that I hire leadership from within my network. Maybe that right there is the thing that we need to work on.

    Sharma: Well, I’m with you. We need to work on it. What I’m saying is—that’s what I was saying. I was saying that what we tend to go, tend to—we tend to grow our network to find leadership role people, right? People in leadership.

    Davis: Yeah.

    Sharma: When you can’t grow your network, you’re reliant on what’s already in your network, right? I mean, otherwise, you’re scouring LinkedIn or some other website or pinging people in your network asking for referrals.

    Davis: Yeah.

    Sharma: But that network stalled. That network didn’t grow. I can’t point to 50 people I met in the last year who were cool and interesting who I did not know before who were underrepresented minorities have I would have met, had I gone to Reinvent or IBM Think or, you know, DevOps World or KubeCon or whatever. All the people I would have met, I didn’t meet them.

    Shimel: So, that’s a whole other thing. Virtual events, right?

    Ashley: Mm-hmm.

    Shimel: We’ve all spent significant portions of our lives at conferences. Cornelia, that’s where I usually see you is at conferences—Sanjeev, as well. Virtual events are cool. We put on one a month here at MediaOps. Gene Kim does his DevOps Enterprise Summit virtually, people even paid money to attend it.

    And all of the virtual events, they try to do a social thing to it. It’s just not very good, frankly. It just—it’s just not the same as being in the hallway or grabbing a drink at the bar or a soda or whatever, you don’t have to be drinking. But, you know, just not—it’s not that, someone said it before, face to face, person to person. I think it was Cornelia, you recently had an in person. It’s not the same, it’s not the same.

    I mean, so, different than the three of you, Mitchell and I have had a chance to meet hundreds of new people this year. I do three to four to five of these interviews a day. My day starts with talking to people in Asia or Israel and goes to Europe and through the Americas, all the way to the West Coast. And what I’ve found is that I’ve become a resource for people like you, Sanjeev, who say, “Hey, I’m looking for someone to head up cloud or DevOps or testing” or what have you. And I go through, who have I interviewed recently?

    But that’s not the way it’s supposed to be. And the question is—and we’re getting near on time—I wanna bring it back to DevOps and software, right? And, you know, what—

    Chase: I think you touched upon it from the beginning, Alan, it’s the Internet. You know, the fact that our technology infrastructure was able to shoulder the burden, that was pretty incredible to me, too. You know, I expected Zoom to go out for weeks, if you will. But I think it demonstrated that the technology infrastructure we have is pretty solid. You know, it allowed us to scale up pretty quickly, pretty massively and, you know, there were outages here and there, but I would’ve expected a lot more.

    Shimel: Minimal—minimal. Even counting all the DDoS attacks and the ransomware and all of that nonsense, it was still minimal.

    So, we had that infrastructure, but we also had this incredible capability to accelerate, to automate that DevOps kind of has been teaching us for seven, eight, nine years or more now, right? And that really helped. Curbside pickup and the—it’s not just curbside pickup, it’s the back end, obviously, right, that allows you to pull into that parking space and hit the button and say, “I’m here, and this is the car I’m in,” right? That kinda stuff, how many companies and jobs were saved by that kind of capability.

    Chase: The other thing for me internally is, I think it also opened up the ability to question everything, just because—at all levels, you know? Whether you’re designing a new feature, adding something to a product, designing a new process—is that the right thing, just because we’ve done it like that? I mean, really question it, because I think it, I mentioned this before. I think the opportunities ________ are endless now, really.

    We haven’t even started to take advantage of the things we’ve learned. I believe we’re starting to now, as Sanjeev mentioned, the pendulum coming back. Part of that is some massive learnings that we’ve also had forced upon us, if you will. But I think we’re gonna come through it even stronger and better, for sure.

    Davis: Yeah.

    Sharma: I think the—

    Davis: You know, I wanna, I’d love to chime in on—

    Shimel: Go ahead, Cornelia.

    Davis: – yeah, I’d love to chime in on that scale. You used the word scale and these systems were able to scale and handle just a tremendous surge in traffic.

    Shimel: Oh, yeah.

    Davis: And when you said that, it took me immediately back to a DevOps Enterprise Summit where Nicole Forsgren and Jez Humble—and those of you who don’t know them, you should know them.

    Chase: Yeah, we know them.

    Davis: State of the DevOps Report, State of the DevOps Report, the book Accelerate. I remember this moment where they were on stage, and Jez Humble, in his very humble way, hand in his pockets, says, “Turns out, architecture matters.”

    Shimel: Yeah. Right.

    Davis: And so, it’s having these systems that are designed to scale, and we find so many enterprises that have said, “Oh, well, you know what? Google scale, Amazon scale, that’s not a problem that I need to face.” Well, surprise—

    Chase: Yeah, it is.

    Shimel: It is.

    Davis: – maybe it is. And so, architecture matters, and so I can’t help, as a software architect, coming back to that and saying, “Yep, there ya go.” [Laughter]

    Shimel: Yep, [Cross talk].

    Sharma: And I think we’ve seen companies who were—there’s two ways to look at it. One way is to look at it from a scale perspective. The other way is to look at it from a resilience and reliability perspective.

    Davis: Right, yeah.

    Sharma: Right? Which are two aspects of the same thing. It’s all architecture. And I think companies realize very early that reliability and resilience are critical—resilience to be able to make changes rapidly, right? Today, I have, I don’t have curbside pickup, tomorrow I have. Today, I had 10 people doing this, tomorrow I’m gonna have 200 people doing this.

    Ashley: Right.

    Sharma: Today, I had a teller sitting in front of her terminal, so the user interface didn’t need to be that cool. You know, the things could hang and the teller could, you know, make sense, make small talk while the system is catching up. Well, nobody’s accepting that on the mobile app when they’re directly dealing with the website or the mobile.

    So, I think that whole shift on resilience and reliability are real, are important, and all that goes back to your point, Cornelia, to architecture. That was front and center to people, and I think [Cross talk]—

    Chase: Yeah. And I think along with that, Sanjeev, is security. I think security, this really exposed—we have to do more in security. We have to do more, you know?

    Ashley: We have to rethink security.

    Chase: I think that’s one of the chief learnings.

    Shimel: No doubt about that. Guys, I hate to do this to you, but I need to pull the plug on this. We already have people in my waiting room for our next interview.

    What a great panel this is. We’re gonna have to rerun this again and go to a part two, because there’s things I wanna talk to about this as well. I spoke to a guy today who started a new, he calls it the mobile cloud for gaming, it’s called Now.gg. Because their research is that our kids, you know, these digital natives, if they don’t get a response in half a second, they’re gone, they’re onto the next thing. And so, what does that mean for your banking apps? What does it mean for we’re doing?

    But we need to end this. Sanjeev, Cornelia, Kurt—man, thank you, guys, so much. Thank you, all, so much for making this an amazing panel. Thank all of our viewers out here for watching this; we hope you enjoyed it. Mitchell, I’m gonna let you take it home.

    Ashley: Well, I think—what a fascinating group and wonderful things to say. I think the biggest learning for me—and I really appreciate Cornelia and Sanjeev talking about the lost opportunity we had through networking to help people come together and expand their networks. So, let’s invest in that going forward. We need to rebuild those and make that happen. So, I appreciate—I really wanna thank you for helping me understand that better.

    Shimel: Absolutely. Many thanks to Tricentis for sponsoring this episode of DevOps Unbound. We’ll be back in two weeks with a fresh episode, a new panel, a new topic, but hopefully it’s just as interesting a conversation.

    This is Alan Shimel for MediaOps. Until next time, be safe and be well.

    [End of Audio]

  • Report: Lots of Room for Continuous Delivery Improvement

    Report: Lots of Room for Continuous Delivery Improvement

    A report published by the Continuous Delivery (CD) Foundation suggests that, in terms of achieving that goal, most organizations still have a long way to go. Based on analysis of the Developer Nation survey data collected by SlashData, nearly two-thirds of developers said it takes at least one week to go from code committed to code successfully running in production. Only one in 10 developers are deploying code in a production environment multiple times per day.

    On the plus side, half of developers reported that they restore service from an unplanned outage in less than a day.

    The survey identified the top programming languages being employed as shell scripting languages, Go/Golang, JavaScript, PHP, Scala, Dart, C#, Ruby, Python and Java.

    Surprisingly, the vertical industry furthest along in terms of achieving continuous delivery is the retail sector that has embraced digital business transformation, while the furthest behind is the telecommunications sector.

    Tracy Miranda, executive director for the CD Foundation, noted that even after more than 10 years of advocacy, DevOps adoption of CD best practices remains nascent. Many organizations have adopted so-called continuous integration/continuous delivery (CI/CD) platforms, but they are mainly used to drive a set of CI best processes. CD, in comparison, has proven to be a more elusive goal to achieve mainly because each platform on which an application is deployed tends to have unique attributes. Automating the delivery of applications across multiple platforms has, as a result, been difficult to achieve and maintain, she noted.

    Nevertheless, with the arrival of cloud-native container-based applications destined to be deployed on Kubernetes clusters that expose a consistent set of application programming interfaces (APIs), the prospects for more widely achieving CD are improving. Each platform that has a Kubernetes cluster running will expose the same API to a CD platform.

    In the short term, however, Miranda noted that many IT professionals are still somewhat intimated by both Kubernetes and CD best practices. Too many organizations still tend to think of CD as a platform to acquire rather than a set of processes and skills to be defined and nurtured, she added.

    One way or another, however, the CD challenge will need to be addressed as more organizations realize how dependent they are on software to deliver a differentiated customer experience. The rate at which that customer experience improves will determine the level of customer affinity that organizations will be able to attain and retain. That’s doesn’t mean every organization has to be as proficient as, for example, Netflix when it comes to implementing DevOps best practices, Miranda added. However, she said, it’s already apparent most organizations will need to improve how they deliver software to drive multiple digital business transformation initiatives. The opportunity lies in reviewing those processes now with an eye toward creating a culture squarely focused on continually improving the customer experience, she said.

    It may still be a while before most organizations truly master CD, but, as is the case with most journeys, the most important thing is to simply take that first big step.

  • Mark Your Calendars and Meet the Best in Tech at Cloud-Native Days with Kubernetes

    Mark Your Calendars and Meet the Best in Tech at Cloud-Native Days with Kubernetes

    After over a year of dreaded isolation, many of us are ready to return to the networking and social atmosphere of pre-COVID life. While physical proximity continues to be a struggle, virtual proximity through events like virtual summits and teleconference meetings, is getting increasingly more helpful amidst the transition.

    Here are the top 10 reasons why Cloud-Native Days with Kubernetes is a must-attend event:

    1. Cloud-Native Days with Kubernetes is a continuation of the virtual social interaction we’ve seen throughout the last year but offers attendees more than just a chance to talk with session leaders and speakers. 
    2. Sessions will be geared toward practitioners, managers and C-level executives, and led by industry thought leaders and doers in the cloud-native space, allowing attendees an opportunity to meet and network with experts in the cloud-native field. 
    3. At the end of this two day virtual summit, attendees will walk away with a better understanding of cloud native and its impact on the IT landscape.
    4. Cloud-Native Days is hardly short of talent with over 50 speakers holding featured, insight and spotlight sessions. 
    5. Attendees will have networking access to a variety of speakers through keynote sessions at Cloud-Native Days, but catching keynote speakers Tracy Miranda and Rosalind Radcliffe’s sessions will certainly be a highlight of the virtual summit. 
    6. Not only will attendees be able to network with speakers, by means of networking, attendees will have the opportunity to forge collaboration. Keynote speaker Tracy Miranda has some valuable words to share regarding projects born from collaboration. She says, “We had this idea that Continuous Delivery is bigger than just one project, and we should be working together to make things inter-operable and better for everybody. Then as we went around talking to people for that idea— when it resonated with other people— they really became fans, and they would be like, “How can we help? What can we do? This is how we build up on it.” So find that core set of folks who will be enthusiastic, and will get you through the honeymoon period.” 
    7. Learn from the best of the best in the cloud-native space. With cloud-native technologies becoming an integral part of IT environments, it’s incredibly valuable to learn from industry specialists changing the IT world as we know it. 
    8. Listen to the best of the best at Cloud-Native Days. It’s always important to listen. But when listening means driving ideas forward, opening your ears can be the most beneficial tool to be better than we were yesterday. 
    9. Get real-world insight into topics like: trends and best practices in the cloud-native space, addressing Kubernetes complexities, tackling cloud-native security from the onset, etc.
    10. Have fun while learning new things! Who says networking and learning can’t be fun? With so many exciting people, sessions and topics, attendees have a lot to look forward to at Cloud-Native Days with Kubernetes.

    Networking isn’t just about linking on LinkedIn anymore. Collaboration doesn’t come from a request, it comes from interaction, engagement and communication. Find all three and start collaborating at Cloud-Native Days with Kubernetes, Aug. 4 and 5 at 9 a.m. (ET). 

    Now mark your calendars and meet the best of the best in tech! Register today.

  • EP 11: Unburdening Developers

    EP 11: Unburdening Developers

    Every app development need seems to ultimately fall to developers to fulfill. Build in security. Shift testing left. Provide users cooler, more customized experiences at cloud speed and scale. And do it all faster, of course. How do we accommodate the rising expectations without burning out developers? What can other team members do to lighten the load and allow developers to focus on the creative work they require to thrive? In this episode of DevOps Unbound, hosts Alan Shimel and Mitchell Ashley are joined by panelists Adam Kalsey of Tricentis, Tracy Ragan of DeployHub and Justin Hutchings of GitHub to talk about unburdening developers. The video is below, followed by the transcript.

    Alan Shimel: Hey, everyone. Welcome to another episode of “DevOps Unbound,” sponsored by our friends at Tricentis. We have a good, nice topic I want to talk to today, and you know, I refer to it as the straw on the camel’s back, but it’s about the ever-increasing load that developers seem to be taking on with technologies and frameworks such as DevOps and Agile and so forth that we’re seeing out here. And we’ve got a great panel to explore this. Let me introduce you to them before we go any further.

    Let me first of all introduce you to my friend Justin Hutchings. Justin is with GitHub and he’s gonna tell you a little bit about himself. Justin, welcome.

    Justin Hutchings: Thanks, Alan. Yeah, I’m Justin Hutchings. I’m a director of product here at GitHub. I work on some of our security features for developers, you know, whether that’s things like static analysis, security testing, or supply chain kind of features for developers in opensource and the enterprise. Glad to be here.

    Shimel: Nice to have you here, Justin. Welcome. Secondly, she’s a frequent guest here on “DevOps Unbound” and other MediaOps videos, my friend Tracy Ragan, CEO of DeployHub. Tracy, welcome, maybe a little bit about DeployHub and you for the audience.

    Tracy Ragan: Absolutely. Thank you, Alan, and thank you, Tricentis, for having these and always welcoming me on these panels. I like doing them. So yeah, I’m Tracy Ragan, CEO of DeployHub. DeployHub is a hub of metadata for deploying and configuring microservices. So we do automated configuration management so you know what your blast radius looks like for a microservice before you deploy. I’m also on the board of the CD Foundation, and you might see me oftentimes evangelizing around microservice usage.

    Shimel: Yes, we do. And then – welcome, Tracy. And then last but not least as a panel member today is Adam Kalsey. Is that pronounced right, Adam?

    Adam Kalsey: It is, absolutely, yes.

    Shimel: And Adam’s with Tricentis, but why don’t you tell us a little bit, Adam?

    Kalsey: Yeah, so I lead developer relations at Tricentis, and we make software to help people make better software, to test it and make sure they have higher-quality software. And my role is to help make developers more awesome by helping them make better software.

    Shimel: Thank you and welcome. And then last but certainly not least is my cohost for “DevOps Unbound,” Mitch Ashley, CEO of Accelerated Strategies Group, CTO here at MediaOps. Mitchell, welcome. Great to have you on.

    Mitch Ashley: Always good to be here, and having been part of many development teams, I’m excited to talk about this great topic.

    Shimel: Great. Let’s dive into it, guys. You know, when I started DevOps.com about eight years ago, there was this constant push and pull. Was DevOps too developer-centric? It was all about the developers, the ops said. This DevOps thing is just – it’s all about the developers; we’re not getting our fair share. And you spoke to developers and it was, “You know what? DevOps is too ops-centric. It’s not about the ops. It can’t be everything ops. It’s got to be about developers.” And so there was this, you know, friction about was it too developer-centric or not.

    Over the years, though, what I’ve seen is an increasing amount of responsibility seems to have shifted left, as we say, right? And we shift left, and when we shift it left, we’re primarily – we’re shifting it to the developers. So things like testing, for instance, right, have become – developers are probably more involved in testing, or at least kicking off testing and this stuff, than they ever were. Security – certainly the whole rise of the DevSecOps movement has been about shifting security left, and in fact many of the most successful security companies in the DevSecOps space have become successful by making security tools for developers. We’re making testing tools for developers. We’re making tools to look at your blast radius for microservices for developers. Developers are deploying, right, as part of their DevOps focus.

    When is enough? When do we break the developers’ back? Not to mention the developers are the highest-paid people in the food chain, right? So by giving them everything, are we giving them nothing, right? Are we stunting their productivity? Adam, being that you’re the new guy here and you actually are in charge of developer relations at Tricentis, we’re gonna ask you to kick it off. Are we putting too much on the backs of our developers?

    Kalsey: I don’t think so. I think that the promise here is that we make things easier for the developers. I mean, sure, you have more tasks. There’s more things that you’re responsible for. But the overall effort and the amount of work that has to be done ends up shrinking. If you don’t find out about a bug in your software or a security problem or such until you’re in production and customers are using it, now all of a sudden you’ve got way more work to do. It’s harder. It’s more expensive. It’s more difficult. There’s more involved in doing that. And so by moving a lot of these things forward, we’re letting the developers be more proactive in taking care of them. We’re helping them get things fixed sooner. We’re reducing the amount of toil that developers have to do. Yes, there’s more upfront work, but that long-term toil starts to fade away and disappear.

    Shimel: Fair enough. I don’t know if you – you know, I don’t know if I’ve ever heard someone say developers are – you know, it’s making it easier or less for them. Justin, what do you think?

    Hutchings: You know, I’ve talked to a lot of companies that are the early stages of their DevOps transformation, and I think one of the most common mistakes that these companies make is they fail to recognize that adopting DevOps requires both technical changes as well as cultural changes. You can’t just get rid of the testers and say engineering owns tests and expect the same velocity. You’ve got to build out the infrastructure to support that, you have to do the training, and you also have to recognize your throughput is going to be less while you’re paying down that sort of debt on getting that infrastructure and getting the automation and getting all of those things.

    Companies that fail at the DevOps transformation, they go through sort of the motions without understanding the culture and talking about it. You know, years ago when I worked at Microsoft, the Windows team sort of famously had a one-to-one ratio between engineers and – software development engineers and test. These were folks that wrote test automation. And there was a big layoff, thousands of testers got laid off, and they said, “We don’t need them anymore. We have observability tools. This will be great.” And that was the most naïve statement I had ever heard, and it took a year for the organization to fully rebound, to figure out what the right set of responsibilities were for engineering, and how to support those folks effectively on it.

    Shimel: Fair. Tracy, in your roles, both at DeployHub and CDF, you’re dealing with both constituencies here, right, the traditional DevOps teams. But let’s face it: You’ve got dev and you’ve got ops in there, and you deal with both. What’s your take from the battlefront?

    Ragan: Oh, my goodness. I could write like a 20-page blog on this. I guess that wouldn’t be a blog. It would be a research project. [Laughter]

     Shimel: Or a book.

    [Crosstalk]

    Ragan: Yes! So I think that in my discussions with folks over the last six months, eight months, I am seeing a lot of SREs really stressed out. So I don’t know if it’s the development teams that are really taking the brunt of this, but if we think about our history and how we got here, we’ve always had somebody really important on a development team who always worked in hand with the operations side of the house to find out what happened, to help with the deployments. Even if they’re sitting there watching the operation first and hit the button, they’re the person that’s always taken this responsibility. And I think the DevOps movement was an effort to push more operations tasks to that trusted partner on the development team, and now that role has turned into an SRE.

    I would say five years ago we wouldn’t have seen this kind of stress, but because we’re going through this massive tsunami moving from monolithic to microservices, and we’re in a transition phase, we are putting a lot – what I see is we’re putting a lot more stress on the SRE role. Developers are going to the SRE, operations are going to the SRE, and the SRE is just trying to keep up with the underlying technology that’s wobbling and moving. You know, they’re trying to figure out how to stand up environments easier. They’re trying to implement Terraform or Rancher. They’re doing everything that they can right now to try to keep up, but they may be underwater for a bit longer.

    So when we think about where the stress point is right now, I think that developers have pretty much automated a lot. You know, we have tools like Tricentis to really improve the testing, but now we’re int his weird phase where we have these SREs that are trying to straddle both worlds. And they’re struggling.

    Shimel: Justin, you know, in your role there – and, guys, I don’t mean to call. If you want to say something, feel free to obviously jump on. But, Justin, I’m wondering – you know, GitHub has sort of a unique view with this.

    Hutchings: We’ve been trying to push folks towards more automation, more DevOps, you know, DevSecOps with all the security features we have. You know, again I think culture is the key. I look at anytime you have an unfunded mandate, you know, it’s gonna fail. A lot of times organizations will say we’re going to microservices or they’ll say we’re gonna adopt all these security tools, and then they don’t think about how that actually gets done and they don’t recognize the sort of pain and training and all of the things that have to happen to get folks there.

    You know, and I think there’s been a lot of advancement here. You look at the state of observability. You look at the state of CI/CD. Things are getting better, but I think, you know, we in the industry that sort of work int his space sometimes paint such a rosy picture that our customers aren’t prepared for the hardness of the journey that lies in front of them. And they’re looking at how do I get to the magical garden of DevSecOps, and the magic garden is there, but it is not a free entry, you know?

    Ashley: You know, Justin, to what you were saying, I think along with culture, part of is a role change that’s happening across the whole organization, right? A couple things are happening. We’re doing not just automation that the developers are doing, clearly SREs. Other test automation, infrastructure as code, you know, those are all development that’s happening to support those. It doesn’t say that every person is an expert developer, but there’s some development, scripting, other kinds of skills that come with it.

    So I don’t know if we’re so much loading it on the developer as much as we’re elevating our level of automation and kind of interconnected of the process, and we’re helping people evolve and growing their strengths – let’s put it that way – to also include some software development, or more closely working with people that can. I think that’s part of that cultural change, too.

    [Crosstalk]

    Kalsey: One of the things that’s actually happening there is as things become more development-centric and, oh, now it’s Toad and now it’s – the teams are making exactly the wrong lesson out of this and they’re learning the wrong thing out of it. You know, one of the things that Justin said earlier is, “Well, we got rid of all the software testers because the developers are doing the testing now,” and that’s one of the dumbest things that you can do as an organization, is say, “Well, now that my developers can do this, let’s get rid of all the experts.” And instead of saying, “What is the new role of this expert and how does their role change?” whether, okay, now they learn to code so that they can expose more of this as code, or do they become consultants to the developers and help them do this right and help them understand how to do this right? Could I remove a lot of those pieces of toil and a lot of that extra stuff from them and let them focus on the things that they’re good at – writing the code, building the code, thinking through the problems – and let me focus on the things that I’m good at – thinking about the infrastructure, designing the infrastructure, designing test plans, thinking about the non-happy path?

    And so companies – I talk to companies all the time that say, “Well, we’ve moved – all our developers are doing the testing now, so we got rid of our QA department!” And that is – that’s just frankly dumb. Don’t do that.

    Ragan: I think it was an outcome of the Agile movement, right? Because we started preaching this idea of bringing everybody into the same room together to do these tasks. So we broke up our testing departments and we spread them out across all of the teams, as well as some of the operations, which is how the SRE title came about. So, you know, first we siloed and then we tried not to silo. But ultimately it is how do you come up with a strong automation standard across the organization? And how are you gonna implement it? If there’s a testing team that has a group of people and each one is assigned to an application team, or if you have a siloed testing team and a siloed operation team, ultimately we’re still pushing the same product through the lifecycle. So how well do we do that? And maybe Agile confused things for us for a while and kind of broke things up and disrupted what we were doing. So now the task is how do we move forward with where we’re at to make it continue to be easier to get things across the pipeline?

    And I have to say that one of the biggest challenges I hear from people is that continuous integration works well; continuous delivery from dev, test, through prod doesn’t work so well.

    Hutchings: Yeah, and I think, you know, what you hit on there, Tracy, is really important because, you know, when you have an organization where the engineers are the only thing and engineers are undifferentiated and treated as some sort of fungible resource, I think things fall apart. You know, what I’ve seen in successful organizations is they have some subject matter experts, whether that’s SRE or a data team or an infra team, that are building out things that make the rest of the organization able to scale. And that’s a really good, healthy blend where you’ve got experts that you can lean on, but then engineers are empowered to do sort of the last-mile delivery there.

    Kalsey: Yeah. When a company says they’re gonna create a cross-functional team and then they eliminate most of the functions, that’s a problem. I mean, you don’t say I’m gonna have a cross-functional team, so we’re going to bring product managers and designers in and, oh, by – now we just don’t need product managers and designers. So why do we do that with operations, with testing, with all of these other things that are important for software creation? You know, eliminating those and just saying, “Well, the developers will do it,” of course that makes for a lot more work for the developer. The way that this needs to work is keep those cross-functional teams and change their role and bring them further – bring them earlier into the developer cycle. Have them involved in design. Have them involved in figuring out what the product is going to do and how it’s going to do it, not just let’s get rid of them. The options aren’t they happen at the very end of the process or we don’t have them at all. It’s let’s bring them up further along in this process.

    Shimel: You know, part of it, though – and I don’t know if anyone wants to admit it – is that there’s sort of this bias that everyone else on the IT team aspires to be a developer ’cause that is the ultimate kind of thing. So, for instance, it was this way certainly in testing, that being a tester was a waystation on the way to being a full-blown developer, right? Or maybe a developer who wasn’t that good, well, he could do testing, right, and that testing in and of itself wasn’t a “profession.” It was really just like a junior developer did testing. And, you know, security people, well, they were always the rebels, right? Half of them didn’t have formal training. But you know, operations, the SRE folks, right, how much of a developer heritage or pedigree is there in what you need to be able to do from an SRE perspective? And so when we put together these “cross-functional” teams, it’s almost with the idea that, well, you’re all gonna be developers now, right? You’re gonna reach your lifelong ambition by being in this DevOps team as part of a developer. And we lose out on the professional tester, on the professional SRE, on the professional security person, right, in this rush to all be developers. What do you think about this?

    Ashley: Well, I think we did this to ourselves in the DevOps movement. You know, I remember when I first wrote my first article on DevOps.com about what is DevOps and I was sort of grappling with it myself, the mantra was, “We don’t need operations, we don’t need test, we just need developers. Code or developers writing code will eat the world.” And it wasn’t outsiders that said that; we said that, or subsets of us said that. And I think we recognized after we got into it, no, as Adam points out, those are exceptional skills. Frankly some of the worst testers I’ve worked with were developers. They were terrible at testing their own stuff. They were great developers, right? But some of it is elevating maybe too much the role of the developer to be the rockstar beyond the platform, if you will – and there are fantastic developers. I don’t mean there aren’t. But there are amazing, amazing test people.

    And so let me put it to you this way. When I was running product development at StillSecure with you, who did I listen to? The testers. I listened to them and I asked them, “What problems are we finding?” and I could tell if we were ready to ship, not just by the numbers but by their opinion. And that’s why I really relied on, because they saw it from a different perspective than a developer does. So that’s my opinion about it. Now feel free to disagree. Anybody can take me on and say I’m full of baloney.

    Hutchings: So I was gonna say, Mitch, you know, have folks heard the story about – what is it – the IBM Black Team that they developed back in the ’80s for testing where they – basically testing was a sort of poorly-regarded function in the organization? And they ended up creating sort of a hit squad that famously would go and just destroy people’s code. And they created a culture around that, and they created sort of an ethos of that being a special place to work. And I think that’s something we need to find with our developers. Like, if we only celebrate the developers that make the shiny features, we’re not gonna be in a good state. We’ve got to celebrate the developers that can destroy other people’s code with tests or can build something that scales really well or that build great observability things. Like, it can’t all be how many features did you ship; it has to be how much you contributed to the whole.

    Ragan: Yeah, and I – you know, I think, Justin, that the cultural discussion is one that any team that’s going through this experience right now that we’re describing, that the developers do feel burdened in some way, they should be looking at how they have built that team. And kind of each role should be a specialist role, and they all have to be an equal role. And if you don’t have that, you really haven’t created an Agile team and that team’s not gonna become a high-performance computing team, because it is about equality on the team and the ability for every single person to stand up – tester, the SRE, the junior developer – and be able to communicate problems as they see them in an equitable way and not feel that they are being judged.

    And I have learned that from working in an opensource community. I’m really amazed how well the opensource communities work together, ’cause you have people with all kinds of different backgrounds, and it fascinates me to see how much they can accomplish and how well they accomplish it when they’re not all working for the same company and they’re there because they want to be. So how do you create that opensource kind of nurturing culture within a particular team if it’s built especially around Agile, where we’ve taken operations and testing and developers and, you know, other specialists and put them on the team – documentation, requirements gathering – and how do you make it an equal team? That I think is the challenge to keep the burden off of the developers’ backs.

    Shimel: You know, and that – paradoxically, that’s also the big attraction to like cloud-native communities, to these opensource communities, because I think people recognize the – and I don’t mean it in a bad way – the kumbaya kind of spirit that these opensource communities foster. GitHub, quite frankly, Justin, right, if you look at the history of GitHub, that’s probably been one of the huge drivers of 52 or whatever it is million accounts on GitHub, is that sense of community, right? How do you build that sense of community interdisciplinary across your teams within an organization? Adam, you know, though the role says DevRel, it’s really TeamRel, right? And you know, it’s got to be part of your mantra there, part of your job description, isn’t it?

    Kalsey: Yeah. One of the things that is interesting about when you look at opensource is the person who wrote the documentation is a hero, because nobody wanted to write the documentation. The person who wrote the tests is a hero because, wow, they’re writing the tests, and that was a lot of work and I didn’t want to do that amount of work. But then when we go into a company we say, “Well, we’re gonna hire a couple of QA people to do the testing because they’re cheaper than the developers.” And, well, yeah, maybe if you’re hiring bad QA people they’re cheaper than the developers, but if you’re hiring good testers, they’re as expensive or more expensive than the developers.

    And so in the opensource world we look at it and we go, “These people are peers and equals,” and then we come into a company and we say, “They’re second-class citizens, and we’re gonna hire a couple of tech writers ’cause they’re a lot cheaper than having the developers write the documentation, or we’re gonna hire some testers, or we’re gonna hire some ops people ’cause the person to watch the dashboard is way cheaper than having a developer do it.” And we’ve seen it happen with the SRE role where, when we elevate that and that becomes part of the development team, the other – SREs are expensive and hard to find. That same sort of thing I think needs to happen in other roles on a development team in order for them to stop being that second-class – and also to stop having developers be the ones that have to do all of this work. If you look at it and say, “Well, the test writer or the doc writer is a cheaper person and we’ll just hire a couple of them,” it’s more likely to fall on the developer to do the hard work than to hire the good people and have them do the hard work.

    Shimel: Fair enough. Mitch, what do you think on that?

    Ashley: You know, it’s a really good point, and I think part of what you’re saying, Adam, is it’s a specialty. You think differently to be a fantastic tester, not just testing but the whole test strategy and how to do all the different kinds of testing that’s involved, and then of course how do you automate that and do you do that yourself? Do you work with part of the dev team to do that? Same thing for SRE, right? It takes a different – someone who thinks about it differently. And it’s sort of that cognitive diversity of teams that we value from a diversity standpoint. We value that also because those roles also require specialty skills. You know, someone who’s designing – and, Tracy, you could explain this better than I could, but someone who’s designing a cloud-native architecture and thinking about the microservices and what’s gonna constitute a microservice, how we’re gonna communicate between them and how we’re gonna secure them, that’s a different kind of thinker than how do I break all that stuff. That’s really nice that you did that, but watch how fast I can break that. That’s fun, and that’s where people really enjoy working together. But I think that’s part of where you’re elevating, too, Adam, is you’ve got to value all the skills, and they may be expensive, they may be not quite as much, but they’re needed. They’re part of the process.

    Shimel: Interesting. You know, Justin, we’re talking about testers. We’re talking about SREs. We’re talking about developers. Once again, no one’s talking security. Are we just sentenced to be forever the outsiders here? What’s it gonna take?

    Hutchings: Most of the security people that I talk to take that as a challenge, you know? We obviously want developers to go and embrace security as part of, you know, their everyday job descriptions, but folks in AppSec and especially in the security research community, they’ve always felt like kind of scrappy outsiders that are going to shine a light on things that other people didn’t notice. And so I think even when we have very security-minded, security-motivated engineering teams, security is a discipline that’s got to stand apart, because we want to have that sort of trusted auditor role or that, you know, lookout for – whether it’s bug bounty reports or stack analysis findings that no one looked at. Like, someone’s got to be watching those, ’cause developers, especially in the world of microservices, they might work on a thing, finish it, and then go somewhere else. But the security world changes all the time. If you walk away from that microservice and all of a sudden we learn about a new category of security problems and no one’s watching it, well, that’s a problem now. And so security has a really special role in terms of being the stewards of all the things, even when they’re not directly accountable for the functionality of it. And I think that’s a special thing that we should celebrate.

    Shimel: Okay, we’re special. But you know, as special as we are, yet we’re still seeing security tools for developers, right? That is a bona fide trend in our industry, and at the same time we’re also seeing – when we talk about the load on developers, we’re still seeing an extreme need for more security people, right? We just can’t produce enough qualified security folks, it seems. If that’s the facts, right, there’s no getting around it. What else is there to do but put this load on the developers? How else do we deal with it, automation? I don’t know.

    Hutchings: You’re absolutely right, and I think the key for tool spenders like GitHub is to figure out how to give developers enough information that they can be successful but not overwhelm them, because security tools, you know, there’s a risk of false positives. If you give them too many, they will ignore them. And so, like, as a tool vendor, I have to be extremely careful to make sure that I don’t burn out developers on the things I think are important, ’cause they might not find them as important as I do.

    Kalsey: The other thing those vendors have to do is make less work, eliminate work for the developer, not cause more, not give them more things to pay attention to, more work to do, more administration to do, but look for ways that you can remove that work through the tools.

    Ragan: Yeah, so we have a challenge facing us when it comes to automation right now, and the challenge has to do with these imperative pipelines that we’ve been running for the last ten years. Sometimes you have a Jenkinsfile that is pretty complicated, and we’ve been running them, we copy them, and we use them. It’s imperative.

    The problem with it is because it’s imperative. We are seeing a shift now to moving away from these imperative pipelines to where it’s more declarative and event-driven, which means that it should be easier then to take these older – if you had a purely declarative pipeline based on events, think about how easy it would be to say, “Okay, we’ve decided on this set of security requirements to add to sort of the workflow steps for the testing workflow, for the development workflow, and we’re gonna plug those events in.” The reason why we don’t do that now is because it’s too hard. [Laughter] It’s just hard, because we have to take these old imperative pipelines and try to modernize them. And without having that declarative framework, these kinds of tools, even adding continuous testing, even adding continuous deployments all the way to production, it’s hard and it takes a lot of work, and you have to change the tires as you’re going down the freeway in an imperative model. The faster we can get to an event-driven process in our CD pipeline where we can declare them and we can automatically generate what we need to do in a workflow, the sooner we’ll be alleviating some of the pain that the developers might be seeing, the sooner we can bring in better testing, the sooner we can bring in the DevSecOps conversation, and in our case automated configuration management.

    This has always been a problem for any of the tools in the CD pipeline to start being plugged in, because the imperative pipelines that we are dealing with on a daily basis, nobody wants to touch them. So we’re gonna have to disrupt that first, and when that gets disrupted and people start moving to something more declarative, we are gonna see an increase in adding all of this kind of automation into the CD pipeline.

    Ashley: I’m curious, Tracy. Is it the complexity of the pipelines that – the way they’re defined, is that why people don’t watch to touch them, or –

    Ragan: It’s the number.

    Ashley: The number, okay.

    Ragan: You’ve got – literally I’ve seen people with thousands and thousands of workflows ’cause they’re running one for every single application version. And to look in that and say, “Oh, we’re gonna change all of these,” because it’s not templated so you can’t just change it in one place and it’s just gonna trickle down, because they’re all these very statically scripted processes. And for whatever reason on the distributed side of the house – when I say distributed, I’m talking about everything except the mainframe – we have relied on heavily scripted processes, and we’ve scripted ourselves into a box that we cannot – we’re not Agile. We cannot easily update these pipelines to include security in them.

    We talk about it at the CD Foundation often, trying to elevate security as a first-class citizen in the CD pipeline. But that means that everybody’s got to start updating all those CD pipelines, and they’re not templated. They’re all statically scripted so you have to address one at a time. And as I think Justin might’ve said, you’ve got to plan to pay for that. So we’re in a box, we’re in a corner, and we have to somehow get ourselves out of that corner in order to alleviate these kinds of problems. And being Agile at the CD pipeline level means that we’re gonna have to dump these imperative processes.

    Hutchings: And I think you’re exactly right there, Tracy. One of the challenges we also see with security is this desire to start slow so that you don’t overwhelm people with too much, and if you don’t have a centralized control plane that everyone pulls from, you know, as you start to introduce these security tools, you can’t dial up the difficult level as you go. You have to hit those distributed CI pipelines every single time you want to go and update the difficulty, and that’s not tenable. I mean, you know, there are services out there – I think GitHub with Actions is doing a pretty okay job with allowing some centralized management, but it’s a spot that we all have to invest in because, you know, that control plane for specialists like security to go and help inject things is just not as mature as it needs to be.

    Ragan: And the CD Foundation has started an events working group to start having this discussion. Right now they’re talking about just vocabulary, but I believe that will grow into a bigger project in the future so that we can start thinking about how you can have that event-driven process so that you can dial it as you need to, right, as oppose to having them statically scripted in all of these imperative pipeline processes.

    Shimel: So, guys, we’re running low on time. Just one other topic I wanted to bring up. You know, in looking at this problem, though, I’m reminded of when I read The Goal, right, which is kind of the book upon which Gene Kim based his Phoenix Project on. The Goal is about more manufacturing; The Phoenix Project was IT. But really it talks about bottlenecks, resource constraints. And the story of The Goal – and I forgot if it was the first rule, the second rule, whatever it was – is as soon as you fix one or get rid of one problem, the next problem down the line shows itself that maybe you didn’t see because the problem in front of it was hiding it camouflaging it. In trying to make life easier for our developers or take the strain off of that point, are we really just going through this goal process of discovering the next bottleneck? And maybe that’s what – maybe that’s the secret to life. Life is just about going from one bottleneck to the next, right, and discovering what’s downstream from there. There’s something Zen about that. But is that what we are sentenced to for sure, or is there a way out of that?

    Kalsey: I like to say that when designing large-scale distributed systems you can’t remove your scalability or your failover problems; you just move them to a different part of the stack. And we do that a lot with development teams as well, is we don’t remove the problem; we just move it to a different part of the process and move it around.

    I think one of the ways out of that is one of the tenets of Agile is let the teams make the decisions. Hand them the problem and let them decide how to organize, how to do these things, how to make stuff work. You know, if management is pushing down on development and saying, “You’re now going to be the testers,” of course we’re causing more work and more problems and more things for them, and they’re going to become the bottleneck. But if we let the teams figure out – they’re the ones close to the work, doing the work. Let them make the decisions about how we’re gonna do this and what they need to make it happen.

    Shimel: Fair enough. All right –

    Hutchings: Yeah, I was gonna say I like the term incrementally correct. You know, Alan, your thinking about bottlenecks is absolutely true. I don’t think there’s ever a world where we will get to 100 percent panacea, but we can hope to get a little bit better all the time, and that’s the journey that we’re all on, right?

    Ragan: And failing fast.

    Shimel: _____ continuous improvement, and failing fast – like get to know –

    Ragan: And allow your developers to fail and not – or your testers or your security. It’s okay to fail, because then you learn what you need to do. Failing fast is part of the learning process.

    Shimel: Fantastic. All right, guys, we are out of time. I want to thank all three of you for joining Mitchell and I on this episode of ” DevOps Unbound.” Thanks to Tricentis for sponsoring ” DevOps Unbound,” as always. We will be back in – not next week, the week after with a new, fresh episode, as well as check out – we’ll have a new ” DevOps Unbound Roundtable” open to the public for questions, and you can find out about that at DevOpsUnbound.com or on our Digital Anarchist site or DevOps.com frankly. Look under – it’ll be under webinars and you’ll find the Roundtable there.

    Until then, though, Tracy Ragan, as always, it’s a pleasure to have you on. You bring such enlightenment to the whole discussion. Thank you.

    Ragan: Thank you, Alan.

    Shimel: Thank you. Justin, again, always nice to have you on one of our panels and discussions here. Many thanks for coming. And, Adam, thank you and good luck in – you know, I know you’re relatively new in your position there at Tricentis but it looks like they made a hell of a hire. Good luck, and come back and visit us again.

    Kalsey: Would love to. Thanks a lot.

    Shimel: All right. Mitch, you want to take us home?

    Ashley: I just wanted to thank our panel as well, and I love the incremental progress idea, and it’s always getting outside perspectives sometimes, too, that’s helpful. So I think we’re here to help each other in this process and not one of us has the answers.

    Shimel: Right on. All right, this Alan Shimel for “DevOps Unbound.” We’ll see you on the next show. Bye-bye, everyone.

    [Outro music playing]

  • 5 Ways to Reduce DevOps Toil

    5 Ways to Reduce DevOps Toil

    Over the last several years, DevOps has become a bit of a buzzword. It has become simultaneously a practice, a culture, a team, a job title and a vendor product. You can hire some DevOps, buy some DevOps, adopt DevOps and sprinkle a little bit of DevOps on top for good measure.

    But, at its core, DevOps is about service ownership: having a single team with end-to-end ownership of software running in production. You should not have one team building software and another team deploying and operating it. That’s slow. DevOps is literally combining development and operations into a single consolidated team. DevOps means embracing the mantra of “you build it, you own it.”

    The primary benefit of DevOps is increased speed and agility of engineering teams. With DevOps, exactly one team is responsible for a given piece of production software. There are no handoffs or knowledge silos. Practices like continuous delivery can emerge. Features can be put in front of your customers’ eyes with a shorter turnaround time. A secondary benefit is increased reliability and security. With “you build it, you own it,” it is no longer someone else’s job to ensure that software is correct or performant or reliable or secure. These responsibilities are “shifted left” onto exactly one team.

    So, where do things go off the rails?

    Service ownership and DevOps mean giving teams the agency to make whatever changes are necessary to their software. That can and will come at the expense of product functionality at times. So, the first place things go off the rails is when there is not a healthy balance between product needs and engineering needs. If teams are constantly working on new features but do not have the capacity to fix operational issues that are affecting them, product quality will eventually suffer. Ultimately, product-engineering imbalance generally comes from misaligned incentives or poorly defined goals from management.

    The second place where things go off the rails is insufficient training. Especially with organizations early in their journey to adopting DevOps, which may have separate development and operations teams, there is a lot of training on new technologies and processes. Developers need to learn about observability, responding to incidents and debugging production applications. Operators and sysadmins need to learn about coding and the principles of software design. These are new skills for both camps and investment in training and mentorship is required.

    What are some best practices for keeping DevOps on track?

    There are lots of these best practices. It really depends on your starting point as an engineering organization, but here are five important ones:

    1. Track Service Ownership

    If DevOps is about “you build it, you own it,” you need an authoritative list of a) what’s running in production and b) which team owns it. If a piece of software goes down, you do not want to find out that the last person to work on it was the intern who left three years ago. Many folks start tracking this kind of information in spreadsheets or wikis, but keeping the data up-to-date becomes challenging. SaaS service catalogs have emerged as the easiest way to automatically keep this data accurate, complete and consistent.

    2. Adopt Continuous Integration, Continuous Delivery and Feature Flags

    Committing small changes multiple times per day is superior to release trains where one large change is committed on an infrequent basis. CI/CD allows for new features and improvements to be tested faster and put in front of customers more quickly. Feature flag tools let you easily test new functionality and product hypotheses for different segments of customers.

    3. Embrace Incident Management

    Software breaks sometimes. Having a robust process for how to triage and respond to both alerts and incidents is important. Within the context of DevOps, the team that owns a piece of software should be responsible for being on-call to respond to associated incidents. If the software breaks or behaves in some unexpected way, the team is expected to fix it. The team is also responsible for identifying what happened and investing in ways to ensure the same error does not occur again.

    4. Treat Everything as Code, Then Automate all the Things

    Applications and microservices? That’s code. Infrastructure? That’s code, too. Security policies? Also code. Deployments? You guessed it–code.

    Treating everything as code allows for processes like “code review” to emerge beyond just your applications. You can have multiple eyes on every change, which catches bugs sooner and spreads knowledge to others. As well, version control systems (like Git) are a natural audit log, allowing anyone to see what was changed, when, and by whom.

    With everything defined as code, the next step is to invest in automation. How many steps does it take for a developer to deploy new code to production? Or to spin up new infrastructure? Or to hook up monitoring or error tracking tools?

    Your goal should be “one step:” run a command. It can be a shell script, a Slack bot or a button with a nice web UI. But it should always be one step. If there are more, you’re now in “manual steps” territory, and that’s brittle. You have no guarantees that successive individuals will run whatever sequence of steps in the same order, which can lead to divergence in how various parts of production are set up.

    Another example is maintaining a list of a) what’s running in production and b) who owns it. Manually keeping that list up-to-date is impossible as the size of your production infrastructure or engineering team scales. You’re better off investing in service catalog tooling that can automatically integrate and capture this information.

    5. Invest in Reducing Toil

    Service ownership and DevOps are two-way streets between management and their engineering teams. On one hand, consolidated ownership will lead to faster delivery and more reliable software. That’s good for management. But, on the other hand, teams require actual empowerment and agency to make decisions about their software. Part of that is having capacity on their roadmaps to make improvements to issues that are causing toil.

    For example, if a particular service is paging a lot or experiencing load issues, service ownership means giving the owning team the time to make any necessary changes. That will come at the short-term expense of new production functionality.

    There’s no free lunch on this last point. Engineering management can’t just combine dev and ops responsibilities and magically expect results. Service ownership is a two-way street. Otherwise, toil will kill your teams in the long term.

  • 12 Ways to Bake Security Into a DevOps Transformation

    12 Ways to Bake Security Into a DevOps Transformation

    Security has become an integral part of any DevOps transformation. According to the Upskilling 2021: Enterprise DevOps Skills Report, DevSecOps achieved a must-have percentage vote of 56% in the automation tool category. Security not only protects the business and its customers, but it enables companies to enforce internal and external policies. 

    While considering how to bake security practices into a DevOps transformation, I asked several speakers and sponsors for the upcoming SKILup Day as well as several DevOps Institute Ambassadors to weigh in with their thoughts. Here’s what they shared: 

    Sponsor, Kendall Miller, president, Fairwinds

    “A DevOps transformation is about building great tooling so that developers can own their services all the way into production. The addition of security is the same thing; it’s building great tooling so that developers can also own the security posture of their services all the way through to production. This transformation is about enabling developers to understand security concerns at the application level. Service ownership is still service ownership—ownership of the operational and security pieces.”

    Sponsor, Guy Eisenkot, VP of product, Bridgecrew by Prisma Cloud

    “The key to a strong DevOps transformation that keeps security at the forefront is selecting and leveraging tools and processes that lend themselves to automation for your specific stack. For AppSec and cloud security alike, automation is the name of the game. But automation is only as strong as the processes it can fit into and the feedback it provides. 

    Security feedback will be futile if it’s not actionable and not given at the right time or in the right place. When undergoing a DevOps transformation, it’s important to keep that in mind when implementing tools and processes. It’s also important that the feedback surfaced is actionable and useful so that engineers can actually learn from and implement secure coding practices.”

    Sponsor, Rob Cuddy, global application security evangelist, HCL Software DevOps

    “The best place to start is with design and make sure that your epics, stories, hill statements and use cases all have security elements as part of them.  A great idea here is making ‘misuse cases’ or adversarial use cases and then testing against them.  For example–a misuse case might read, ‘As a bad actor, I can use a malformed URL to gain unauthorized access to a web server,’ or ‘As a bad actor, I can impersonate an endpoint in a transaction and both send and receive data while impersonating without the other parties’ knowledge.’

    It is also vital to have great SCA–software composition analysis. You cannot secure what you don’t know you have, and this includes open source and third-party tools. In a lot of cases, simple code sharing can inadvertently cause dozens of new dependencies to be added in, and those dependencies can add new vulnerabilities.

    It is important to have security aspects tested and validated throughout a pipeline, but particularly at any points in the pipeline where other checks are also being made. A great place to combine this is in QA and wherever functional testing is being done.  Leveraging interactive security testing (IAST) in an environment like this allows for monitoring to occur and for vulnerabilities to be found while application capabilities are being tested.  This greatly reduces the appearance of false positives and provides real-time information about the security posture of applications.”

    Sponsor, Yasser Fuentes, cloud workload security technical product manager, Bitdefender

    “In short, security should not just be considered at the CI, dev and test phases or limited to vulnerability assessments, since most of the vulnerability scanners and assessment tools DevOps teams use today to go over their CI/CD pipelines rely on either known vulnerabilities, signatures and known attack techniques. As a recommendation, we should emphasize the importance of incorporating tools or solutions which allow developers to identify potential threats and security flaws at the runtime level and which provide visibility beyond the typical anti-malware alerts to suspicious and potentially anomalous process-related behavior.” 

    Sponsor, Joni Klippert, co-founder and CEO, StackHawk 

    “To bake security practices into a DevOps transformation, teams need to reimagine how security plays in the software development life cycle. For many organizations today, security testing blocks applications from being shipped into production or tries to play catch-up once software is released. But it doesn’t have to be that way. 

    Security can (and should) live throughout the pre-production phases of software development. Modern security tools automate security testing in the IDE or CI/CD pipeline, alerting engineers if they have introduced a vulnerability or a vulnerable dependency.

    Making security just another type of test that is run in pre-production means teams can find vulnerabilities faster, remediate on the fly and, most importantly, get back to feature development. Baking security into the DevOps pipeline is what all teams should aim for!”

    Stephen Walters, sales engineer, Everbridge

    “The exact same way that any other practice has become baked in as part of DevOps—by following the CALMS principles. Implementing a culture of inclusion and acceptance; integrating automation into security testing, monitoring, builds and repository management; ensuring the flow and feedback of security into delivery and support is performed in a lean way, adding value and not waste; measuring security as a business value through the use of scorecards and as a success criterion for the business; sharing concerns, lessons learned, new practices and techniques.”

    Dheeraj Nayal, global community ambassador & region head – Asia Pacific, Middle East and Africa region, DevOps Institute

    “A proactive approach is the best method for building in the essentials of DevSecOps. This can be achieved by following some best practices.

    1. Teams should be aligned and in-sync
      The first and foremost step in implementing a DevSecOps culture is to train your teams that security is a shared duty of teams from all three systems. Once development and operations units take on the distributed responsibility of guarding code and infrastructure, DevSecOps becomes a routine part of the development cycle.  
    2. Keep the code simple
      Complicated code can carry more security vulnerabilities. And simple code is more comfortable to collaborate on. All your developers should be capable of looking at each other’s code and know what is happening.
    3. Testing
       SAST (static application security testing) is a key security tool that checks security issues regularly and earlier in the development process which enables it to address the issues more efficiently.
    4. Deployment
      To expedite product delivery and add flexibility to the development process, automated deployments are great. One can review properties across the IT infrastructure and implement secure configurations in a system employing an infrastructure-as-code tool.   
    5. Operations
      Operations teams must collude with security practitioners. They are accountable for controlling infrastructure and network arrangements. Operations and security teams unite to set up manual and automated security tests to warrant compliance with network configurations.”

    Peter Maddison, founder, Xodiac Inc.

    “Two main places: within platform engineering to enable the collection of the data you will need for creating visibility, feedback and learning. Next, it is important to engage the security and compliance teams to determine where you can integrate into existing standards or where modifications might be required.”

    Mark Peters, technical lead, Novetta

    “Security practices can be baked in through the standard DevOps approach, through people, processes or technology. Baking in practices with people means getting folks on the ground with your team who can advocate for security. This can happen through having your security experts spend time with the teams, assigning security to be responsible for teams, or even by selecting champions within those teams who regularly interact with security. The first agile principle, after all, is ‘people over process.’

    That component does not allow one to ignore the process.  Building the correct process implies using test-driven development at the start. Every delivery should be tested and those tests should include security testing. Some common security testing, such as code quality, now occurs so frequently many no longer consider it a separate security practice; merely a routine part of development.

    Finally, one can bake processes in through technology. Many security tools allow shortcutting through metrics, logs and traces relayed on the dashboard. These tools evaluate and highlight where security challenges occur. Tools allow integration through infrastructure and maintaining awareness of what functions occur and when. One of the old military strategies uses the OODA loop by John Boyd: observe, orient, decide and act. Just as with intelligence systems in the military, better awareness of your system through dashboards allows one to observe functions within the network, orient against possible problems, decide on a solution and act before those problems ever reach critical mass.”

    Supratip Banerjee, solutions architect, Principal Global Services

    “There are a few ways we can bake security into DevOps:

    1. Security as a mindset: There is a thought process that security in the development process might slow it down. But we must educate teams that the time and effort it takes to fix a security flaw early in the process is lower than doing it later.
    2. Platforms/environment: Proper configuration/sanitation of deployment platforms/environments is extremely important. Infrastructure-as-code (IaC), configuration-as-code, compliance/policy-as-code, secrets management, etc. can help with the process. Increased use of containers and cloud platforms can keep it clean as underlying infrastructures are managed by the provider.
    3. Automation: Include a lot of automation testing in the CI/CD pipeline. Also add static code analysis, third-party testing, AVA testing, code vulnerability checking, container security, intrusion detection and prevention, APM, etc. as automated processes. Make change management, auditing, etc. regular and as self-operating as possible.
    4. Secure architecture: Make sure security is enabled at the time of the software architecture/design phase. An architecture risk assessment must be a mandatory process to catch any design vulnerabilities before development starts.
    5. Culture of curiosity: Create a culture that emphasizes the importance of security, so everyone is interested and ready to discover if anything goes wrong with security in their system.”

    Amiran Alavidze, director of security and risk management, Tasktop

    “DevSecOps is about culture and enablement and making software delivery teams autonomous and efficient. Training and awareness is a core component of that enablement. This also includes focusing on tools that are developer-friendly and automated so that critical security controls are an inseparable part of how things are done, and allows development teams to take on all the responsibility, without a cognitive overhead of expert-level understanding of the security domain.”

    Sharath Dodda, IT development manager, TD

    “By engaging security early in DevOps with as much automation as possible. All the scans that are possible and relevant should be done well in advance in the pipeline, so that the artifact is cleared of all the findings.”

    Learn more about DevSecOps and similar topics, by registering for an upcoming SKILup Day. Or, start your upskilling journey by learning more about the benefits of DevOps Institute membership.