Author: Alan Shimel

  • DevOps Dozen² Awards 2021 Nominations Now Open

    DevOps Dozen² Awards 2021 Nominations Now Open

    The DevOps Dozen² Awards are back and open for nominations. This past year has been quite a journey and we are very proud of the DevOps community in their ability to pivot, innovate, adapt and think ahead in these challenging times. Some outstanding leaders and vendors have worked nonstop to reach their goals and achieve the highest level of success. We are thrilled to celebrate their hard work and achievements.

    The DevOps Dozen² Awards honor the most dedicated and inspiring thought leaders and innovators who have made significant contributions to the community. For the second year, we have expanded our awards program to include two different DevOps Dozen sections: DevOps Dozen Tools and Services Awards and the DevOps Dozen Community Awards.

    There are 12 awards for each section:

    • DevOps Dozen Tools and Services Awards — Recognizing the companies that develop and deliver outstanding solutions to empower developers, DevOps and IT operations teams.
    • DevOps Dozen Community Awards — Honoring the leaders who have made a positive impact in the DevOps community.

    Nominations are made by the audience. Anyone can nominate a tool or service, although there is a $250 entry fee for the Tools and Services Awards nominations. We implemented this fee last year to cut down on spam and superfluous nominations. However, there is no fee for Community Awards nominations. Please note that commercial tools and services are not eligible for the Community Awards and will not make it through to the finals.

    Finalists will be selected and winners will be chosen from among the finalists. The finalists will be determined by general public voting (weighted for 40% of the final total) and our judges’ selections (weighted for 60% of the final total).

    The last day to submit your nominations is October 22. To vote, follow the steps outlined on the DevOps Dozen² website.

    Who Are the DevOps Dozen² Judges?

    The following DevOps experts make up the DevOps Dozen² Awards 2021 panel of judges, and will review the submissions made by the public and choose the winners:

    • Alan Shimel, founder, CEO and editor-in-chief at MediaOps
    • Jayne Groll, co-founder and CEO at the DevOps Institute
    • Mitch Ashley, founder and CEO at Accelerated Strategies Group
    • John Willis, senior director of the Global Transformation Office at Red Hat
    • Helen Beal, chief ambassador at DevOps Institute and ambassador at Continuous Delivery Foundation

    The 2021 DevOps Dozen² Categories

    DevOps Dozen Tools and Services Awards

    • Best End-to-End DevOps Tool/Service
      Consolidation in the DevOps space has led to a greater selection of platforms offering end-to-end solutions. This award honors the platform with the most diverse and robust solution for DevOps teams.
    • Best DevOps Repo/GitOps Tool/Service
      Git has led to GitOps and it has become an integral part of the DevOps toolbox. This award is presented to the most outstanding Repo/Git/GitOps tool.
    • Best CI/CD Tool
      This award recognizes the best CI/CD tool that helps teams to deliver quality code better.
    • Best Value Stream Management Tool
      Value stream management has come into its own as an important tool for measuring the business value of DevOps and application development. This award recognizes the value stream management tool that most effectively helps visualize the value stream, track metrics and measure improvement.
    • Best Observability Solution
      This award recognizes the best observability solution for observing and monitoring to understand the performance of individual components and systems and entire applications.
    • Best DevSecOps Solution
      This award honors the best DevSecOps solution, which enables security to be included earlier and continuously throughout the SDLC.
    • Best Testing Service/Tool
      Testing is a mainstay in the DevOps toolchain. This award recognizes the best testing tool for ensuring the quality of code.
    • Best Kubernetes Platform/Service
      The adoption of containers has paved the way for an entire solutions ecosystem supporting and enhancing the container technology. Kubernetes is perhaps the most widely recognized and utilized tool in the space. There are many platforms and services that offer Kubernetes-based solutions. This award honors the best Kubernetes platform/service available.
    • Best Cloud-Native Security Solution/Service
      The cloud-native movement has taken the IT world by storm due to the depth and breadth of the solutions available and the benefits of building applications and services specifically for cloud-native environments. This award recognizes the best tools, solutions and services considered cloud-native.
    • Best DevOps for Mainframe Solution
      Mainframes are far from being the dinosaur of the IT world, and a variety of solutions are available for enabling DevOps practices in mainframe environments and hybrid environments. This award recognizes the best of them.
    • Best DevOps for DataOps/Database Solution
      Databases often have been left out of the DevOps conversation, but that is changing. The whole concept of DataOps is gaining greater traction. This award recognizes the best solution in the database for DevOps and DataOps categories. 
    • Best New DevOps Tool/Service Provider
      This award recognizes the best new service provider or tool that has brought innovation to the DevOps space in the past year (must be less than 24 months old).

    DevOps Dozen Community Awards

    • Best DevOps Industry Implementation
      This award honors organizations that have successfully adopted and implemented DevOps principles and practices.
    • Best DevOps Survey/Analysis/Research
      This award recognizes research that has significantly and positively impacted the DevOps community.
    • Best DevOps-Related Video Series (video)
      This category highlights the most entertaining, educational and inspirational DevOps video series—those with the most engaging and highest-quality content.
    • Best DevOps-Related Audio Podcast Series
      Which is your favorite DevOps podcast from this year? This category celebrates the many great DevOps podcasts sparking some interesting discussions on a multitude of topics from CI/CD and testing to DevSecOps, Kubernetes, containers and more.
    • Best DevOps Book/eBook of the Year
      Each year sees a plethora of well-crafted DevOps-related books and ebooks. This category highlights the most popular and best read of the year.
    • Top DevOps Evangelist
      This category is one of our favorites. There are so many great evangelists who spread the DevOps gospel and we want to know who does it best. Nominees can work within the DevOps space or outside of it.
    • Best DevOps Transformation (non-vendor)
      In a year in which digital transformation became a must-have for survival, we want to know which organizations stood out in their efforts to transform their business through DevOps.
    • Best DevOps Virtual Event of the Year
      COVID-19 has changed the game when it comes to hosting events. In 2021, virtual events remain the norm, reaching a global audience while everyone stays safe at home. Which DevOps virtual event stands out from the others?
    • Most Innovative DevOps Open Source Project
      Open source projects are meant to promote innovation and increase efficiency, velocity and quality. While there are many great open source projects out there, the best ones are those that create a community of people who work on and with the technology and support each other.
    • DevOps Executive of the Year (non-vendor)
      This category celebrates excellence among DevOps leaders at the C-suite (or equivalent) levels.
    • Best DevOps Presentation of the Year
      This category recognizes the best DevOps-themed presentation from the many (virtual) conferences during the year. 
    • Best devopsy.kinsta.cloud Article of the Year
      devopsy.kinsta.cloud publishes multiple new articles every day. Which was the best article from 2021? Let us know so we can recognize the author.

    All the solutions nominated to the Tools and Services Awards will automatically be registered into the DevOps Tools Guide authored by Techstrong Group. This initiative is our effort to create a comprehensive listing of tools and tools vendors to help end users better understand the DevOps market landscape.

    Choosing a winner is difficult when there are so many talented and inspiring leaders doing an incredible job to empower and support our community. Although not everyone can win an award, being nominated says a lot about each and every one of you. We greatly appreciate your dedication and service to the DevOps community.

    For more information, please visit the DevOps Dozen² website. If you have any questions regarding entry process, awards, timing or any technical issues regarding the website, please contact us at DD@devops.com.

  • MediaOps, Boca Code Award ‘Engineer the Change’ Scholarship

    MediaOps, Boca Code Award ‘Engineer the Change’ Scholarship

    I am really proud to announce that, in partnership with Boca Code, MediaOps has awarded a scholarship for the Fall 2021 term to the Software Engineering Career Course, a 10-week rigorous and fast-paced coding boot camp from Boca Code. Alina Maliuk, a first-generation immigrant from Ukraine, is the recipient.

    The Engineer the Change scholarship was initiated by MediaOps to help foster diversity in the local tech industry and empower individuals from underserved communities to develop the skills needed to succeed in IT. Boca Code offers comprehensive training in software development to enable course attendees to develop these skills. 

    Need for Diversity in STEM fields

    Research shows a staggering lack of diversity in STEM fields, most noticeably within computer and engineering positions. The tech industry is predominantly white and male, which appalls me (yes, even though I am a white male), and I wanted to do something to encourage more diversity across our industry. Gender and race inequity are glaring workforce disparities that require both recognition and responsibility from those at the highest levels in the tech world to create a more diversified and equitable workforce.

    Two Full Scholarships Awarded Annually

    MediaOps sponsors two $10,000 scholarships each year to enable students to enroll in Boca Code’s flagship Software Engineering Career Course. The full-time, 10-week intensive course combines theory and hands-on, project-based learning to prepare students for a career in software engineering. As part of their coursework, students take on development projects for local businesses, providing the students immediate, hands-on experience with real projects.

    The Engineer the Change scholarships combine two goals that are core to the MediaOps team and me. The first is doing our part to make South Florida a world-class hub for the tech industry. The second is providing opportunities for individuals from disadvantaged communities to acquire the skills they need for a successful career in tech. My hope is that we also serve as an inspiration to other local companies to fund similar programs in support of these worthy goals.

    About Our Winner, Alina Maliuk

    Alina Maliuk scholarship
    Alina Maliuk, Engineer the Change scholarship winner

    Alina hails from Ukraine, where she earned a bachelor’s degree in management. After college, she married and had her first son. During her maternity leave in 2012, her family emigrated to New York. Being first-generation immigrants, they had to start their life here in the United States from scratch. For the first few years, Alina and her husband worked in entry-level jobs. In 2018, now a mother of two, Alina took several online classes in web development and rediscovered her love of learning—particularly software development. Now a resident of Boca Raton, Alina found Boca Code and attended a workshop in web development.

    Alina was energized by that workshop, which furthered the continuing education she had already begun. Through that initial course, the Boca Code team inspired her to go even further. She is ready to move on to her next career, which she wants to be in software engineering. Thanks to the Engineer the Change scholarship, she will now have an opportunity to accelerate her tech career. At the same time, she is demonstrating to other female immigrants in Florida that if you can dream it and work hard to achieve it, you can make it happen.

    Alan Shimel

    CEO, MediaOps

    Additional Resources

  • Accelerating Software Testing

    Accelerating Software Testing

    Yves Junqueira is the CEO of YourBase, a software testing accelerator for large codebases. He talks with Alan Shimel about the company, accelerating software testing and the impact on developer productivity. The video is below, followed by a transcript of the conversation.

    Alan Shimel: Hey, everyone. Thanks for joining us here on another segment for our TechStrong TV. My guest this segment is a new company, first-time guest on our TechStrong TV, Yves – and I’m gonna really mispronounce his last name, Junqueira. I’m supposed to have that rolling R, and I just don’t do rolling R’s. Yves, I’m sorry, man. Welcome to TechStrong TV.

    Yves Junqueira: Thank you, Alan. It’s great to be here. You did it good.

    Shimel: Not so good. You know what? In case your mom or your family’s watching, say your name correctly for ’em.

    Junqueira: Yves Junqueira.

    Shimel: Junqueira. Okay. We’re gonna call you Yves, and we go from there.

    Junqueira: Sounds perfect.

    Shimel: So, looking from your background, are you up in Seattle, Toronto? Toronto?

    Junqueira: That’s Seattle.

    Shimel: Seattle. Yes, the mountains. And you know what? Welcome. I’m gonna let you give a little introduction to your company, and you’re a cofounder of it, but go ahead.

    Junqueira: Gladly, yeah. Our company’s called YourBase. We build developer tools for very large code bases. Specifically, we are a software-testing accelerator for large code bases. I used to work at Google for almost ten years, and my cofounder, John, was a principal engineer at Amazon, at AWS. And, yeah, we’re excited to – I’m very excited to be here, Alan.

    Shimel: Absolutely. Well, I’m excited to hear about what you’re doing. So you’re a code accelerator?

    Junqueira: We are a software-test accelerator, yeah.

    Shimel: Software-testing accelerator, excuse me. What exactly is that?

    Junqueira: Yeah, so it’s very important when you’re building software that you actually know that the software is working. You usually have very large teams that’re creating code, and you don’t want one developer to be stepping on each other – on the other’s toes. And so you need a lot of testing to make sure that when you make changes to the software, it’s still working. But as you’re adding new features, you’re adding new tests, and you add new features, and you add new tests, and over time the number of tests in your code base can grow very, very quickly, and it can become a very big problem for developer productivity and developer happiness, because it can take a very long time to actually test code. If you’re making even a small change sometimes in a large code base, it can take hours to ensure that that change works, to run all the tests. So that’s what we help with.

    Shimel: So you’re automating testing. Is that fair?

    Junqueira: We automate the test selection of testing. So, yeah, we help run the tests, but most importantly we help select what are the right tests that are safe but that have to be run for each change. Without having to test the entire world, we use a process called – we use tracing of code to build a dependency graph to then select the right test to run and then just make things more productive for developers.

    Shimel: Sure. Of course, our audience is a very technical audience, familiar with QA and testing and developing. So you’re running a trace route on the code as it’s entered into Git or – because it’s pre-deployment code, right?

    Junqueira: That’s right. It’s pre-deployment code, yeah.

    Shimel: So as the code is put into Git or some other sorta repository, you select what’re the right tests to run and run it on the code. So this all takes place within the repository like a Git-ops kind of thing?

    Junqueira: Yeah. It can take place in the repository connected with a CI, with a continuous-integration tool, or it can also happen during local development. You can have multiple developers working at the same time on the same code base but perhaps making slight – different code changes. And with your base, they are testing their code but also using their colleagues’ testing to – so that they don’t have to repeat the test that their colleagues already ran for the same code base. So they only have to test exact the small changes that they made on their code.

    Shimel: I love it. Sounds great. And then I don’t wanna use the buzzwords, but is it some sort of AI/ML that is picking what’s the right code – what’s the right test to run for this particular code for these particular changes?

    Junqueira: That’s an important question. It’s not AI. We use just simple data structure and algorithms that we collect when doing tracing. And this is actually important, because if we were using just AI or any sort of statistical approach, we would only get a limited confidence that we are testing the right things. So imagine that you’re a financial institution, right, or a hedge fund, where tests are very important. If you don’t catch certain bug, it can cause millions of dollars in losses, right, in lost revenue, for example. So it’s very important that you actually detect the correct regressions in your code, and if we were using just a simple AI approach, we would only, let’s say, have 90 percent confidence that we were running the right tests in your code. That is not enough. So our customers need assurance. They need full confidence, so we use an approach that is a little bit more intrusive, let’s say, because it’s detecting every single function called that is happening during the tests. So it has a little bit of a performance impact during the initial test. But on the other hand, it just provides full confidence rather than being 90 percent confidence that an AI approach would have.

    Shimel: Got it. I mean, big companies using this, small companies, anyone? Who’s the target user here?

    Junqueira:Fortunately, we can scale from small to very large code bases. So we have customers that are very small, just small startups that are getting started but that really want to have the fastest possible development process. But we also have very large code bases of very large companies, public companies, Fortune 500s, that want to keep moving fast. It’s difficult. Imagine if you have a software product that really needs to evolve quickly. Customers these days demand that software developers – that software companies evolve their products quickly. The customer needs are changing fast, and so they need to be able to make software changes very quickly and do it at – have high velocity with the most productivity possible. So what we allow those companies to have is even though they have a very large code base, the larger companies, they can be almost as productive or as productive as if they had a small code base, and this is really magical. This is something that we’re very proud of providing to these companies.

    Shimel: Absolutely. You know what? People always wanna know this, and I realize it might be coming outta left field. Interesting. You worked at Google ten years. Your cofounder’s an AWS guy. How did that come together?

    Junqueira: Yeah. It’s actually a funny story. We met at a random coffee shop. After I had left Google, I was working at random coffee shops here in the outskirts of Seattle, and I had my laptop with me with a bunch of developer stickers there like Linux and Python and GOAL like developers often have. And John had the same, and so we just looked at each other’s laptop. We were at this coffee shop here, and we – okay, we should chat, ’cause we us a lot of the same technologies. So we started talking, and over time I invited John to come to my weekly table games at my house. I host a game night every week, and John started coming, and I started talking about my project, and he was fascinated by it, and he said that software testing and developer productivity are a huge thing for him. And over time, he decided to join. I’m very lucky that he joined me at YourBase.

    Shimel: Love it. And then talk a little bit here about going to market. How’s it been so far?

    Junqueira: Yeah. We spent a long time perfecting the technology. This is a very tricky software. If you get things wrong, you may end up – your customers may end up shipping bugs. My customer’s gonna end up shipping bugs that should have been caught by the tests. But if I don’t decide to run a test or if I don’t give them right insights, they might actually introduce bugs and be less reliable than they were before. But we started just last month. We launched our first public beta now, and we’re very happy about that. So we have a number of design partners that are working with us on this initial release, and people – customers are very happy. The feedback has been fantastic. So last month we launched it to the outside world, and we made a public beta. Now people can just download it from just going to the website, yourbase.io. Anyone can download and test it. We currently support Python as – for code bases. Our technology has to be implemented on a language-by-language basis. So for now we support Python, but we’re going to support Ruby sometime soon. And we’re very excited. The reception from people has been amazing. It’s really fun to be at this stage of the company, where we actually are ready to talk about it to the outside world.

    Shimel: Excellent, man. It is. Having founded more than one company, more than four companies, five companies, it is an exciting time, and I wish you nothing but great luck and success with this. I want to get into the kinds of testing you’re doing, functional testing, unit testing. Is it selenium kinda stuff? What are the kinds of tests you’re doing there?

    Junqueira: That’s a great question. We are very good at tests that don’t cross process boundaries for now. This can be sometimes unit tests and functional tests. We don’t currently support UI tests yet, in part because there are good companies out there that’re already serving this market, and there was just nobody that was focused on the backend and in unit testing and functional testing. But this is a big space, and we’re just getting started. We have feedback from folks that we – they want support for more things, so that’s something that we’re excited to work on.

    Shimel: Excellent, man. Yves, I wanna thank you for coming on and telling folks about YourBase here on TechStrong TV. Come back. Keep us posted on your progress, and best of luck.

    Junqueira: Thank you so much, Alan. It was great to be here, and, yes, absolutely, I would love to come back.

    Shimel: My pleasure. Say your name for us one more time so I don’t – just after I mangle it, at least they got it right.

    Junqueira: Yves Junqueira.

    Shimel: Junqueira. Yves, thank you, man.

    Junqueira: Thank you, Alan.

    Shimel: Oh, the website, yourbase.com?

    Junqueira: Yourbase.io.

    Shimel: IO, yourbase.io, Y-O-U-R-B-A-S-E-dot-I-O. This is Alan. We’re gonna take a break. We’ll be right back with our next guest.

    [End of Audio]

  • MediaOps and DevOps Institute Earn Inc. 5000 Honors

    MediaOps and DevOps Institute Earn Inc. 5000 Honors

    I am really proud to announce today that both MediaOps and DevOps Institute have been honored by Inc. and earned inclusion in the Inc. 5000 list of the fastest-growing companies in America.

    For MediaOps, this is the second year in a row we have earned this honor. For DevOps Institute, it is the first year they have been honored. First of all, congratulations to all of the great people who work at and make both MediaOps and DevOps Institute the great companies they are. None of the success of these companies would be possible without their talent, skills, hard work and dedication. Thanks to all of you.

    Secondly, as the founder/co-founder of both of these organizations, I can’t tell you how proud I am that both of these companies have not only had the success they have (as evidenced by this honor) but that they have done so in the middle of a global pandemic. It really makes all of the hard work and the long days and nights worthwhile.

    For anyone who has ever started a business, you know that you put a certain amount of your blood, sweat and tears into any venture you are involved in. It is no different here! Since 2014 I have put everything I could into growing MediaOps into the company it is today. For DevOps Institute, I was lucky to have my partner Jayne Groll, DevOps Institute CEO, take the lead in making DevOps Institute the leading member organization/certification authority in the DevOps space today.

    Yes, you hope the monetary rewards of founding a company make it all worthwhile. But this kind of recognition is also very gratifying. It gives you the extra spring in your step and makes the load lighter.

    These awards also highlight something else. The ascendancy of technology. It is no coincidence that both of these companies play in the digital transformation space and have seen such success. Digital transformation is driving the world right now. Software is driving digital transformation, and DevOps, cybersecurity and cloud are driving software.

    At the end of the day, it is a great honor to have not one but two companies you started win this award at the same time. I think it speaks volumes about the future of both of these organizations. More news on that front is coming, so stay tuned for more great things coming from MediaOps and DevOps Institute!

  • EP 17: Lies, Damn Lies and Metrics

    EP 17: Lies, Damn Lies and Metrics

    In this episode of DevOps Unbound, hosts Alan Shimel and Mitch Ashley are joined by Helen Beal of the DevOps Institute, Viktoria Praschl of Tricentis, Frank Ohlhorst, principal analyst with Accelerated Strategies Group and Alex Hidalgo of Nobl9 to discuss how to use DevOps metrics effectively to improve delivery performance and help teams become high-performing with a culture of continuous improvement. The video is below and a transcript of the conversation follows.

    Alan Shimel: Hey, everyone. This is Alan Shimel and you’re watching DevOps Unbound. DevOps Unbound is sponsored by Tricentis and we’re happy always for their sponsorship and helping up put this show together. DevOps Unbound is an every-other-week session with experts from around the world talking about relevant topics to DevOps. In addition to that, about once a month or every six weeks we will do a live roundtable which is open to you, our audience, to come on, come in, ask questions live, and really kind of steer the conversation. We do have another live roundtable coming up. If you check the devopsy.kinsta.cloud website you can register for that. Today’s conversation, though, is prerecorded and unfortunately that means I get to ask all the questions, not you.

    But let me introduce you to what I think is an amazing panel we have lined up for us today. Well, actually, before I do that let me tell you what we’re going to talk about today on DevOps Unbound. So, it’s the issue of metrics and kind of the ghost in the machine, the dark side of metrics – lies, damn lies, metrics – whatever you want to call it. How do we measure success or failure? How do we measure progress in things, in areas like DevOps or digital transformation? What does success really look like? Our panel is going to talk about that today. Let me introduce you to them.

    First, I’m happy to introduce Frank Ohlhorst. Frank, if you wouldn’t mind saying a few words about yourself?

    Frank Ohlhorst: Sure. And I’ll be quick as not to bore people.

    Shimel: Okay.

    Ohlhorst: I’m Frank Ohlhorst. I’m a journalist, author, and consultant. And I currently work over at DevOps as a contributor, at Security Boulevard as a contributor, and I am also a contributing analyst. And that’s pretty much my story, other than I’ve got 30-plus years – I hate to date myself – 30-plus years in the IT field and I’ve worked both for the federal government, I’ve been a developer, and I’ve put a lot of effort into cybersecurity.

    Shimel: Excellent. Thanks, Frank. Next, let me introduce you to one of my friends from the other side of the pond. I’ve known her now for eight years or more. The one and only Helen Beal. Helen, welcome.

    Helen Beal: It’s a pleasure to be here as always and spending time with you and the others. So, thank you very much for the opportunity. I’m Helen Beal. I’m Chief Ambassador at DevOps Institute, and therefore one of the co-creators of ADOC, which is the Assessment of DevOps Capability, which I may mention once or twice during today’s session. I’m also chair of the Value Stream Management Consortium, and prior to all of that I was coaching in DevOps in a number of different companies across Europe and the Middle East for several years.

    Shimel: Thank you and welcome, Helen. Next up, let me introduce you to Viktoria Praschl. Viktoria, if I mispronounced your last name I apologize. I just realized when I looked at it that I thought I remembered it, but maybe not. Go ahead.

    Viktoria Praschl: It was totally correctly pronounced, so you did a good job.

    Shimel: Thank you.

    Praschl: Hey, everybody. Happy to be here today. I’m Viktoria Praschl. I’m with Tricentis, a software and quality assurance company based out of Austria and Vienna. And I’m responsible for our UP and sales and transformation team where we work a lot with customers on how they transform their organization when it comes to DevOps and quality assurance.

    Shimel: Excellent. Thank you, Viktoria. Our last panel member today also is somewhat of a regular; he’s been with us a number of times. And that’s author, expert Alex Hidalgo. Alex, why don’t you give them a little background and welcome?

    Alex Hidalgo: Thanks so much, Alan. Hi, my name is Alex Hidalgo. I am the Director of Site Reliability Engineering at Nobl9, which is a startup focused entirely on providing the best possible service level objectives, the best possible SLO platform. I’m also an author of Implementing Service Level Objectives, and I’ve been working in this space for – in some way or another the last two decades or so. My current area of research is really into how to bring reliability lessons from other industries into the tech world and what we can learn from others.

    Shimel: Excellent. Thank you, Alex. And then, last but not least, my cohost for DevOps Unbound, my partner in business, Mitch Ashley. Mitchell?

    Mitch Ashley: As always, good to be with you, Alan. And I love our panel. You all are fantastic; I’m excited to hear what you say. As Alan mentioned, business partner and cohost. I run Accelerated Strategies Group, an analyst firm that’s focused on DevOps, cloud native, cybersecurity. And also, I serve as CTO with MediaOps, the company that puts this event on. And both a practitioner and product creator in software, software as a service, cybersecurity for a number of years. I’m not going to name or number how many they are, so you’ll have to guess.

    Shimel: [Laughs] Okay. I’m not guessing. But anyway. All right. Let’s get into this week’s DevOps Unbound. So, I remember a VC mentor of mine getting up in front of – we had launched a new company, he stood up and said something to the effect that in this brave new world we all live in everything can be measured and we measure everything, and it is based upon those measurements we determine who wins and who loses and by how much.

    So, that was 20 years ago, maybe more, a little more than 20 years ago: 22 years ago. I wonder, is it still true? Or have numbers, like everything else, with fake news and all of that, kind of lost their meaning and lost their importance in today’s organizations? Helen, can you –?

    Beal: I’m not sure it’s ever been true. From my experience with metrics I think what I’ve always observed is just people fall into one of two camps. So, either they’re not measuring enough, or they don’t have enough metrics, or they have far too many. I think there’s a cognitive load limit on how many metrics we can handle and we probably need about a handful.

    I think it’s true that we need to have telemetry everywhere, which is of course a very popular DevOps adage. I think we need to have the facility to be able to get to the metrics that we want, but we have to be able to choose them and be able to think of them as an adaptable framework. So, for example, deployment frequency, of course a very popular metric in DevOps, is very useful if you’re moving from deploying every three months to once a week or more. But once you’re deploying several times a day it’s a much less interesting metric.

    The adage I always use is if you can’t measure it you can’t improve it, which is a Peter Drucker phrase. And I do have a problem with that as well because I think you can improve things without measuring them. But the thing is that you’ll never know. And to me that’s the most important thing about measurement: It’s about being conscious about what you’re doing.

    Shimel: Guys, any thoughts on that?

    Ohlhorst: I think it also comes down to an issue of context. People can measure whatever they want out there and they often do, and they spend so much time measuring that they don’t take into account the context of the measurement. You can measure certain processes in the DevOps process and they really don’t matter when it comes down to it when you’re looking at trying to achieve a particular goal.

    Shimel: They don’t. You know what, Alex, we’re going to let you go, and then, Viktoria, you follow up.

    Hidalgo: Yeah, I think one of the big problems that I often see is that people end up trying to measure themselves against others. They hear “Oh, at my last company” – or “I heard at Google or whatever they’re able to deploy X number of times a day and let’s aim for that.” But why? Do you need to deploy that many times a day? Is it actually applicable to you or your application or your organization or your customers? Are you actually gaining anything from trying to do things the way you know others are? Every situation is unique. Every company is unique. Every organization is unique. Every application is unique. And that means you really need to take a step back and think about “What is right for us? What do we need to improve upon?”

    Praschl: Yeah, that’s what I really wanted to say, basically similar, that you take a step back and you really need to look at what’s your baseline. Where are you and what do you want to achieve? What are the outcomes you’re looking for? And I think it’s very important: The metrics will change over time depending – as you mature your processes and as you mature in your organization you will have to constantly change and adjust your metrics which you are measuring. So, I think the key thing is to really initially take a look: “Where am I? And what are the most important initial metrics where I start with?” And then, constantly, basically changing and improving toward further goals.

     Shimel: Great. I think – I’m reminded of the adage “Just because you can doesn’t mean you should.” And I think that’s where we have – and not just in DevOps, frankly, in IT in general. That’s where we run into problems. Because it’s digital almost by definition it’s very easy to grab metrics on all of these digital things, on all these. But just because you can doesn’t mean you should. And just because you can doesn’t mean it’s necessarily indicative. I think, as Helen mentioned, when you’re going to a release a quarter to a release a week, ooh, that’s big news. That’s good metrics. But – and to Alex’s point, not everyone needs to be Google or Amazon and release hundreds of times a day or whatever it is – dozens of times a day. And I think that’s where we get ourselves kind of wrapped around the metrics axle.

    Ashley: I think it goes back to the question of why. If you don’t understand why you’re measuring something, then why do it? Because everybody is not going to be on the same page about how you want to utilize that information. I think about the VC quote that – I remember I was there for that too, Alan. And in a financial world winners and losers, how much return on investment, that’s pretty clear. But I also think there’s a lot of fear around numbers because people are afraid that it’s going to be used against them or it’s not going to be understood, what they really mean. In a continuous improvement context it’s really to – that’s an aid to help you get better and know if you’re getting better, and you have to decide if it’s the right things, which you’ve got to put that context, as Frank mentioned.

    And we have to be on the same page about what it is we really – what does it mean to put a release into production? Is that really – is that a canary release? Is it a real – across every data center that we are in the cloud? What does it mean to do a release? So, you have to get to that specificity. We’re all in agreement that’s – and that’s important to us because we either want to measure ourselves by it or improve at it.

    Beal: I think it’s been implied by a lot of the speakers already, but I just wanted to really want to say explicitly as well that I don’t think metrics shouldn’t be inflicted on people. They should belong to the team. The teams need to generate them themselves. You can set things like KPIs at a higher level and have them connect, but things like OKR should be really generated and owned by the team themselves. And that really speaks to some of the cultural elements at DevOps, which are much harder but not impossible to measure, but things like distributing of autonomy – sorry, distributing authority and having autonomy at the team level to choose and define and improve the work, and that move from leadership from being managers and leadership becoming coaches that are moving impediments out of the way.

    I think, Alex, you were inferring this as well, the importance of the team-level metric.

    Hidalgo: Yeah. One of the things I’d like to kind of bounce off there is I explicitly try to always use the word “measure” because “measure” means something different then “collect.” I think Helen mentioned earlier it’s good to have telemetry everywhere, and so you have numbers everywhere, you have metrics everywhere, you have time series everywhere. And you run into the multiple comparison problem where you don’t really know what you’re looking at anymore. You run into spurious correlations. Have you ever seen that website where it can track people drowning in pools correlates almost perfectly with the number of Nic Cage movies released each year and things like that? And that’s where you end up when you have too many numbers.

    And that’s why I always like to say you measure, because “measure” means you’re taking the important signals out, and you measure flour to bake a cake; you don’t just collect flour. Otherwise, you’re going to not end up with a cake. And it has to be an intentional act, and this kind of – like Helen just said, you can’t inflict metrics on people. You need to let them collect it correctly and measure it correctly.

    Ohlhorst: If we look at DevOps from the pipeline’s perspective it starts to become clear maybe where we should be measuring things or why we should be measuring things. And one of the first things that a lot of organizations run into when they implement a DevOps strategy is they run into bottlenecks because they did not consider certain things. So, perhaps the first place to start with measurement is to try to identify bottlenecks and then go from there. This way you’re improving your process while you’re getting some value out of the metrics, and you’re also establishing what type of metrics you need.

    Beal: I’m glad Frank brought that up because I really think that – for me, there are two classes of metrics, though, that are the most important and underpin everything. And one thing I will say about the DevOps pipeline is in my opinion it doesn’t start early enough. There’s that fuzzy front end and I think we need to expand it right back to where the idea happens. But my duality of metrics is basically if we measure flow and we measure realization of value we really have a concept of what the customer experience looks like. So, we know how fast we’re able to deliver the things that they need and we’re starting to understand whether we are actually delivering the things that they wanted.

    Hidalgo: And I just wanted to add: Frank’s point about bottlenecks is great, but we can get meta about it. We want to make sure we’re very carefully selecting what we’re measuring. And identifying bottlenecks might be a great start but we need to carefully select which bottlenecks we even want to address in the first place. Just because we can see that something slows down somewhere it doesn’t mean that that’s necessarily an area of focus.

    Beal: We could be waiting six weeks for a security check and that’s really hard to get that out of the way, but we could maybe – and Viktoria may be having an example around, I don’t know, test automation or something. Viktoria?

    Praschl: Yeah, I mean, also test automation and _____, you have tons of metrics you can actually measure just from a curious perspective, and if you don’t measure the right things there as well it will seem as if it’s at a standstill. But I think it’s also very important to – I liked the bottlenecks you just mentioned, Frank, because one is the team level, but then you also want to see how does it look from an end-to-end process? How does my overall customer journey look like? And I see quite often companies, actually, they start and they identify the right metrics on the team level, they get the teams there, but then they are failing on the overall picture to bring everything together. A lot of people struggle with that. That’s what I see.

    Ohlhorst: I think the mistake that a lot of the teams make nowadays is they either look at this and they try to relate either DevOps to a waterfall ideology where there’s a beginning and an end and you can measure certain points or they look at DevOps as this rigid structure on how you do things, where the whole idea behind DevOps is it’s flexible and it can be easily modified to meet specific needs. So, I think people run not only into an issue of where “Okay, where do I establish metrics? Why am I doing metrics?” but also how you interpret those metrics. I mean, like we were talking about before, release frequency and things like that are great for what we use as a term is “vanity metrics.” It’s “Hey, we accomplished something.” But when it really comes down to it, what did we accomplish and how did that improve the process? And I think people are losing sight of that.

    Beal: It’s been said already, I think, here but I think that focus on outcomes is really, really key. And I think there’s a couple of other things I wanted to bring up. One is that we talk a lot about being data-driven in DevOps and I’m starting to think actually about being more insights-driven. And the other conversation I’ve had recently is about doing less hypothesis-driven development and more impact-driven development, which I think is another really interesting iteration on our ways of working. And it reminds me of when Alex was talking about this idea of having telemetry everywhere, which is wonderful. And so, the idea then is that you can get any metrics you want. And you don’t want all of them but at least you have the ability to choose them. But as Alex pointed out, there is this downside to this, that we are just overwhelmed with data. And I think that’s something that AI can help us with there as well. I mean, it’s just a fact: There’s far too much data in the world for us to cope with. We need somebody or something probably that we’ve created, a machine to help us with it. So, Alex, I wonder what your thoughts were on AI in the context of measuring our journeys.

    Hidalgo: You’ve asked a very interesting person about this because I’m actually fairly dubious that a lot of these models will actually help. You need a lot of data, a lot of data, to train a model. And most people actually don’t have that much. We may be talking about “Oh, we have too much data” and “We’re swimming in data” and this and that. Very few people have enough data to properly train models. Can this work in some contexts? I think so. And I think a lot of smart people are working on the problem. But I do think there’s a little bit of an overblown focus on it today because I think it’s a lot of trying to fit a square peg in a round hole. We need to – like everything else, as we’ve been talking about, you need to be careful and you need to think about what outcomes you’re really looking for and then decide “Is AI, is machine learning the right thing here?” Because it – just because it’s a buzzword, just because lots of people want to do it doesn’t mean it’s always going to work. You need to make sure that you’re picking the right tool for the right job.

    Ohlhorst: Yeah, and the same can be said with automation when it comes down to it. If you don’t know the possibilities or the probabilities around a process how can you automate it unless you have the data to provide you with that insight?

    Hidalgo: Which, I should say, those words just remind me of what we can do with data is statistics. And actually, when you see me flinch a bit about “Oh, should we use ML?” or “Should we use AI?” I think a lot of the places where people are doing ML they’re actually just using stats. And that’s great because statistics can actually compensate for not having enough data. That’s kind of what it does best in some sense. So, I am a huge fan of using models. I’m a huge fan of using math. Just, again, let’s make sure we’re using the right math. Do we need to build an ML model if a simple regression tells us everything we need to know?

    Ashley: Yeah, I do agree with you about the AI, machine learning of it. It takes a ton of data, if that’s a measurement, in the data world. I want to go to something that you said, Helen. I really like the word “outcome.” What are the outcomes that – have we achieved an outcome when we’re developing software or creating an experience or something out of the process? Because ultimately, that’s what matters. We can improve the things within it. We can automate the processes within how we create software. But it’s all for a purpose. It’s all for – if we’re not doing a better job of delivering on an outcome, maybe those things we’re working on internally are sort of navel-gazing, if you will, looking internally but it isn’t really that valuable. Let’s work on the things that we can connect to an outcome. That’s what I would propose.

    Beal: It’s funny. The other day – it was at DevOps Enterprise Summit and she went up to speak because she used the phrase “time to market,” and I kind of in the Slack chat went “Oh, time to value,” because obviously there’s no point in getting something out the door if it’s useless, whereas with “time to value” at least we’re recognizing that it’s been – giving something useful to something. And then, John Smart corrected me with “time to learning.” And I thought, “Oh, no, actually, it has moved.” We’ve moved from time to market, time to value, and now we’re in time to learning, where we’re recognizing that actually only – something only becomes useful when we’ve seen it have value and we’ve learned from it and decided to do something else.

    But yeah, it’s the same as this kind of impact-driven development as well, this idea that we really focus on having an idea about what we’re trying to do, and then, importantly, measuring that it did what we thought it was going to do. And we’ve been traditionally really bad at doing this in the technology world because we’ve been really project-driven. And when we’re project-driven we have huge amounts of requirements which are (a) really difficult to track, the impacts of, because we deliver them all at once and we weren’t doing things like canary testing which does allow us to look at a much more granular level. And the other problem with being project-driven is we’re always under so much pressure to deliver. Our testing always gets squished because our development overruns and our launch date never moves. So, by the time things go live we’re so exhausted, and then we disband the team and put them into other places anyway, so no one ever goes and has a look and says, “Did that really deliver the four million quid that we – or $4 million that we said it would?”

    I wonder what other people think about how metrics change as we move from project to product.

    Shimel: So, I’ll quickly jump in here on that, and that is – and it’s, Helen, something you mentioned, Alex mentioned. I think what we’re guilty of – and that’s part – in the abstract we talked about the dark side, and it’s not that “Mitchell, I am your father,” but there’s a dark side of these metrics which are that in the absence of facts or in the absence of knowledge we try to fashion metrics to compensate. So, with something like DevOps where there’s a lot of cultural aspects that come into play, there’s a lot of intangibles that come into play, and they’re almost by definition hard then to measure because they’re intangible; they’re not tangible. And so, we compensate by coming up with what some may consider frankly nonsense metrics that make us feel better or make us feel like somehow we are measuring the immeasurable. And I think that is an important part of this whole thing. Measure what you can, but are there things that are just immeasurable? Anyone?

    Beal: Yeah, I – sorry, me again for a second. I don’t believe so. As a coach and a consultant for a long time I was discovering clients that felt that their biggest problem with deploying DevOps or implementing DevOps was the cultural side. So, that seems like something that you can’t measure. And it’s true that you can’t really get data out of Jenkins to tell you what’s happening in your culture but there are other ways to do measurement. And I think it was Viktoria – sorry if it wasn’t Viktoria, but somebody said the important of baselining your current state, and that is absolutely essential. And I think people don’t always do that. They kind of dive in to try and get metrics without really kind of going “Okay, this is where we’re starting from figuring it out.”

    So, from the human aspect – I mean, talking about ADOC, which I mentioned earlier, the Assessment of DevOps Capability, we have 5 different dimensions in that and each dimension has 12 subtopics, so we have 60 topics in total. And I’m starting to sound like we’ve got way too many metrics, but there’s a lot of subtlety in the way that it’s baselined. And the first dimension is human aspects. And within that we’re really getting a feel across an organization, team by team, where they feel their level of capability is around things like psychological safety, around transformational leadership, around how much autonomy and mastery and purpose they have in their daily role.

    So, you’d be surprised, I think, about how possible it is to measure cultural capability without necessarily having a knowledge of how many – or relating it to how many artifacts you’ve got in your artifact repository. There are other ways of doing this. And sorry, I interrupted you, Alex.

    Hidalgo: No. I think those are all great points, but we do have to, as with everything else, approach it with a little bit of caution because especially if these are self-reported, if these are numbers that you’re looking at like “Are you happy on your team?” “Do you think your organization is headed in the right direction?” – those normal kind of questions and then asking people yes or no, or do you put it on a sliding bar from one to five? And studies show most people are going to just take a one or five anyway. And that’s not saying this is bad. It’s just something else that’s another caveat. It’s another “Let’s make sure we’re looking at this correctly.” “Let’s understand that some percentage of our employees are just going to click through it because they don’t want to fill out this form.” If it is this self-reporting thing, you know?

    Beal: It’s such an important point you make, Alex. And it’s something that we deal with when we engage with organizations. There’s such a lot to talk about, because what you’ve actually identified there isn’t a problem with the tool; it’s a problem with the culture. And if people are not feeling that they are part of the change that the organization is trying to make and they’re not empowered to have their say and their voice and be involved in it then they will skew their answers.

    So, we absolutely have to help people understand how important their answers are, what impact they’re going to have on their future and their ability to grow themselves in their organization and make them feel part of the change. But yeah, humans can be a tricky business. But back to your point about data volumes, again, in this kind of assessment environment it is quite important to have enough data to weed out variations from people that maybe don’t care or are skewing for other reasons.

    Hidalgo: Which, again, you can do some basis statistics to ensure that you are dropping your outliers. You – to be able to get very useful data out of this stuff, again, you’ve just got to make sure you’re looking at it in the right way.

    Shimel: Viktoria?

    Praschl: Yeah, I think quite often we just force metrics on teams, so that’s why I like it a lot, what you said, Helen, that you need to also understand the culture. So, where is the team from a metrics perspective? Do they understand why we are measuring? Or are you already in a stage where they say, “Okay, I understand why we are measuring certain things,” but now we need to constantly figure out which metrics we are measuring?

    So, I think it goes back to the team and explaining the “why” and then really helping them or work with them together, once you have the outcome, what do we do with that? How can we improve? What do we do with the outcome of the metric? What do we do with the numbers? So, I like it a lot, what you just mentioned on assessing the culture.

    Beal: Yeah, and that’s a great point. We baseline but we baseline for a reason, because we – I talk about the improvement culture a lot, so that four-step process where we start by establishing the long-term vision direction, then we establish current state, then we establish our next target state, and then we PDCA or experiment between those two states. So, kind of the finishing part of that first cycle is defining the hypotheses and executing them, and then we start again. And we’re not baselining now; we’re comparing it against our baseline at a time period of three, six months and what’s happened with our experiments in between.

    Praschl: Yeah, because I think most companies, they just don’t know. They don’t know where they stand. They don’t know where the team stands. They don’t know where the culture stands. So, that’s the starting point to get everything rolling.

    Beal: And it’s a big investment, Viktoria, for people to go down this path not to know how far down the path they’ve got.

    Praschl: I think for that you need the executive buy-in as well. So, it’s like making sure that you have from the top also the support that you do that. And then, as you go into this constant learning process, I think it becomes very important there.

    Hidalgo: And trying to look at how far down the path you’ve gone, that – it’s going to, again, be different for everyone. Is it more important how far or how far you have to go? And is there even an end to the path? Or is this a process that’s ongoing forever? And just another example of having to be thoughtful about what’s being measured.

    Beal: I don’t believe there’s an end to the path, which is why I use words like “capability” over “maturity,” because “maturity” indicates that there is some kind of a horizon that you can reach. But of course, no matter how many more capabilities you’re building yourself we live in a market that’s forever inventing new ways of doing things and new tools to support the things that we’re doing. So, it’s an ever-moving target that we’re trying to respond to.

    Ashley: I think there’s a whole nother side of metrics I don’t know if we’ve touched on yet, and that is the metric itself can be valuable and inform us about some things, but I also think the analysis of what’s – why? What are the potential causes? In a quality process you do a Pareto diagram to kind of analyze what you think are potential contributing factors and what maybe is influencing why it’s happening in a certain way to try to get to – if you’re – for example, if you’re doing a process improvement what parts of this – a lot of things go into whatever created that metric; how do we know what we might work on first or doing some experimentation with?

    But the ultimate thing is there’s a lot of learning that comes out of that process. It really kind of helps you understand “Wow, I had no idea that’s actually how it was working.” And now that we know that with a little more fidelity we can make some adjustments.

    Beal: That’s powerful. I wanted to respond to Alex, actually, and then I forgot – it went out of my mind. But this idea that the team needs the right metrics – and Viktoria and I have both said those metrics should not be inflicted on the team, but the other thing is most organizations are made up of multiple teams, and different teams, like Alex said, have different capability patterns. But it’s very interesting to try and understand, for example, why one team is really good at something like diversity and inclusion or really good at AI ops or something, and not saying that the other teams are bad but trying to learn from those success stories. And I think it’s probably the Phoenix Project that put it, it was making local discoveries and global improvements.

    And I think that kind of mindset of looking for the strengths and spreading the strengths across a business is really important when we’re doing things like measuring people on their journey because the – we talked about culturally that people are kind of either scared of this stuff because they’re expecting to get beaten up about it or they’re cynical and they’re not giving true answers. But there’s another side of this, that people are also quite afraid of being assessed or even assessing themselves because somebody’s going to go “Well, you’re shocking at deployment frequency. What are you going to do about it?”

    And it’s like we have to change that whole mindset in an organization that they are a dynamic learning organization. It’s all about continuous learning and not about beating people up over the head or removing their annual bonuses or the like, which are things that I’ve seen in various places.

    Ohlhorst: Which gets us back to that key point of the “Why are we measuring?” Making sure people on the team know why we are measuring things, not only from a cultural perspective, but also from a value perspective so that they know that by them giving their answers based upon queries that there is value to that, and that is the “why” behind it. And ultimately, the goal is to improve processes or accelerate SDLC or whatever it is, but there is a goal there that is not just random data being thrown out asking “What flavor of gum do you like?”

    Praschl: Yeah, I think it’s also –

    [Crosstalk]

    Praschl: It’s important to get them to understand the “why.” It’s usually a long and not as easy of a process. But once they have this and understand it you have the buy-in and the outcome will be way more worth it if you do that way.

    Hidalgo: A thing that I’ve found that can help with this kind of stuff is people underestimate how powerful interviews can be. Instead of just asking people to self-report or just collecting random numbers out of your computer system sit down with people and talk to them. Not for five minutes. Schedule an hour. Sit down, find out what is this person thinking? Make sure you’re talking to the right people, not just in terms of an organizational structure, because you do want to make sure you’re talking to people at all levels of the company and all roles. But also make sure that the people you’re interviewing are comfortable with it in the first place. Not everyone has the personality to want to sit down for an hour.

    So, another example: Select carefully. Measure carefully. But I’ve found so much value in just – instead of trying to do this in some – a method that is based around metrics. We’re talking about metrics. Sometimes you don’t need metrics. Sometimes you just need the human interaction and you can walk away with that if you really want to then throw that in that kind of scale and attach a number to it maybe. But you can learn a lot just by talking to people.

    Beal: I think the best assessments are always a combination of quantitative and qualitative data. And I think we’re seeing this increasingly in the data world as well. About 80 percent plus of most businesses’ data is unstructured, which by its very nature is that kind of textural response data, whether it’s on social media or in your logs in Splunk. We’re seeing more and more people understanding that it’s not just about what’s in an RTMS. So, I absolutely agree with you, Alex. Yeah.

    I think there’s a big advantage, though, which I’m going to point out, around ADOC. So, some organizations are very large, so when you’re spending an hour with people and there’s 8000 people it’s quite impossible to speak with 8000 people. So, actually having an ability to empower everybody to be involved in the change by giving them a platform on which to share their opinions is quite useful. And of course, you can put open text fields in there as well and get those snippets and insights that you would miss if you were just getting a one to five or yes or nos.

    I did want to maybe slightly disagree with one thing you said, though, which is about the people that don’t want to talk. It always kind of worries me – and this is the big thing about, I think, diversity and inclusion and things like Heloxi that they are trying to give equal voices to people, is I don’t want us to fall into that trap where we think “Oh, that developer is really quiet and introverted; we won’t ask them what they think because they don’t really like talking.” It’s like they’ve probably got just as much right to their opinion as everyone else and we shouldn’t avoid hearing it because we just think maybe they don’t like chatting as much as other people.

    Hidalgo: I totally agree. But from my point of view you’ve just got to make sure that whoever is comfortable with the process – people don’t want to feel singled out. Just like everything else, depending on the person, depending on the culture, depending on the team, they might feel like “Why am I being picked? Is this a bad thing?” So, it’s just looking thoughtful. That’s all I meant. Definitely don’t want to exclude anyone. Definitely want to make sure we’re getting every possible voice. Just want to make sure people feel safe about it.

    Beal: Totally. And that psychological safety is so important. I learned this from one of my clients years ago. We didn’t use to do things like participant briefings, and now I think it’s really important to get everybody that’s involved in this type of work and talk to them about what DevOps is and the “why” that we’ve all talked about – so, organization performance and succeeding in a disruptive market and all that kind of stuff – and then give them the comfort about anything that they share is between them and the people that are collecting that information. And they may be quoted but they’ll never be associated with what they said. Their words may be used as an example but they can be completely honest. And actually, they are being asked to be involved because their opinion is so highly valued in the organization and we want to gather their insight, so they shouldn’t feel fearful or that their job is at risk. They should feel honored and hopefully excited to be involved in a process that’s intended to make their future more fun and secure.

    Hidalgo: The only other thing I’d add there is – and again, I totally agree; I think we’re actually –

    Beal: Finally agreeing.

    Hidalgo: – in agreement here. Yeah. Yeah, exactly. But the one other thing I would add is that even if we have a strong suspicion or even have the numbers to prove that our organizations or our teams generally are psychologically safe and we’re building a good culture here and this and that, you don’t know what it was like for people at previous companies. People can carry baggage for a long time. It can take people a long time to adjust, especially as your career goes longer and longer. You have these memories and not all of them are great. And so, even if you think “Ah, our company as a whole, we are scouring very high,” for individuals it may not have anything to do with the current organization at all; they could still be carrying some scars from previous ones.

    Beal: Yeah, and there’s –

    [Crosstalk]

    Shimel: Viktoria?

    Praschl: It’s about creating the safe space. So, some probably will take longer to open up, but usually if you see that others are joining and the more comfortable you feel, you also get those people to open up down the road.

    Shimel: I think we’re all humans, to borrow a phrase, and we bring our own experiences, outlooks, prejudices to whatever metrics we’re going to measure and to whatever we’re going to be in. And I think we can’t lose sight of that either.

    Beal: Right. Those scars that Alex was talking about, I just wanted to talk about one other element of this, which is that the scars are negatives. We’re never badly affected by positive experiences, if that makes sense. So, from a neuroscience perspective we basically train our minds to be fearful. We keep on avoiding – we have an avoidance response that’s trained into us. So, I think I really like to try to use neuroscience when I’m helping people through a DevOps change because there are things we can learn from it, things like neuroplasticity and these different avoidance and approach responses that we have in different scenarios. And actually looking at an organization as a whole is great, looking at different teams and how they compare, but actually understanding, as Alan has just said, that how individual humans operate and in the context of their previous experiences and just how we’re built, because we’re all built differently from _____ anyway, is really important. And that’s really where the transformational leadership comes in and having people that understand how to be coaches and how to understand people at a personal level.

    Ohlhorst: Interesting that we’re talking about the process from almost applying a “flight or fight” type of ideology to it, that here we are talking about software development and DevOps in particular and pipelines and all that, and so many people so often forget that there are people involved in the process. And that is where you come up against the most challenges.

    Shimel: Yeah. It’s about the humans. Guys, we are – we’re about out of time, though we could probably sit here and talk for the rest of the day. But it wouldn’t – I don’t know if anyone would want to watch, and we all have other things to do. What a great conversation on metrics and DevOps and people and everything else today. Maybe we’ll come back and we’ll run a part two of this or something. But for now I need to pull the plug.

    I want to thank our friends at Tricentis for making DevOps Unbound possible with their sponsorship. Alex, Helen, Viktoria, Frank, great conversation, man. Great conversation, men and women. Good stuff. Mitchell, you want to take us home?

    Ashley: What I’m struck by is the richness of just the word “metrics” and what all it can mean and the depths of it and aspects, whether it’s the human side of it, whether it’s the “why,” whether it’s the outcome. There’s several – I think there’s several good takeaways there from this, and so I hope it’s helped everyone kind of think about maybe why you’re doing what you’re doing and how you want to lead that going forward.

    Shimel: Excellent. All right. This is Alan Shimel for MediaOps, devopsy.kinsta.cloud, here on DevOps Unbound on TechStrong TV in the TechStrong TV network. We’ll see you in about two weeks with another great DevOps Unbound episode, but for now this is it. Thank you, everyone. Thank you for watching. Take care..

    [End of Audio]

  • EP 16: How DevOps has Changed Testing Forever

    EP 16: How DevOps has Changed Testing Forever

    DevOps has changed the game in almost every aspect of testing: who, what, when, where and even why. What’s working, what needs to change immediately, and what challenges—and opportunities—lie ahead?

    In this DevOps Unbound episode, hosts Alan Shimel and Mitch Ashley are joined by Abel Wang from Microsoft, Tracy Miranda from CDF, Caroline Wong from Cobalt, Kurt Chase and Ben Simo from Tricentis to share a variety of perspectives on how testing fits into the ever-evolving world of DevOps. The video is below, followed by a transcript.

    DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.

    Alan Shimel: Hey, everyone. Thanks for joining us on this special edition of DevOps Unbound. You’re watching this, probably, at the Tricentis Virtual Event and we are happy to have you join us, happy that you’re attending the event, and we’ve got what we think is gonna be a great discussion on testing and all of its various facets and what does the future – or near future – hold. We’ve got a great lineup of speakers that I’d like to introduce you to who are on our panel today. Let me first start with – she’s on the top of my screen – is Caroline Wong. Caroline, why don’t you say, “Hello” and introduce yourself?

    Caroline Wong: Hello. My name is Caroline Wong. I’m the Chief Strategy Officer at Cobalt. So, at Cobalt, what we do is pen testing as a service. We build a security software platform – like a lot of DevOps companies.

    We have data-driven product-based teams. We have – we really value automation and failing fast. I’ve spent 15 years in the information security industry. I wrote a book called Security Metrics. I host a podcast called Humans of InfoSec, and I teach the OWASP Top 10 Learning Path on LinkedIn Learning, and super excited to be here with this group today. Thanks for having me.

    Shimel: Thank you, Caroline. Next up, we have Kurt Chase.

    Kurt Chase: Hey, everyone. I’m Kurt Chase. I’m currently Head of Release Management and Engineering Services at Tricentis. Engineering Services is a group that supports the engineering function so, we do our own type of testing – testing all the processes and everything we use – to deliver our software at Tricentis. So, prior to Tricentis, I spent some years at Splunk and at AutoDesk. It’s great to be here with all of you today.

    Shimel: Fantastic.

    Chase: Be here today with all of you.

    Shimel: We got what you meant, Kurt. It’s okay. Next up, we have Ben Simo. Ben, why don’t you introduce yourself?

    Ben Simo: Hi. I’m Ben Simo. I’m a software tester of 30 years and past president of the Association for Software Testing who has recently stepped into a product management and product research role at Tricentis trying to discover and develop our testing platform of the future and bring new things to Tricentis customers and hopefully introduce new people to Tricentis with new products that enable new ways of testing.

    Shimel: Okay. Happy to have you here. Next up, Tracy Miranda. Tracy – well, I’m gonna let you introduce yourself. Go ahead.

    Tracy Miranda: Hi, everybody. I’m Tracy Miranda. I’m the executive director at the Continuous Delivery Foundation. It’s an open source organization that’s part of the Linux Foundation, and also home to projects to like Jenkins and Spinnaker, and our members include companies like Netflix and Google. And I have over 20 years of experience in open source so, I love open source and I’ve worked in it as a developer – also in QA and all different facets. So, yeah. Big fan.

    Shimel: Thank you, Tracy. Last, but not least, is my co-host on DevOps Unbound – my partner in crime here, Mitchell Ashley. Mitchell, why don’t you introduce yourself?

    Mitch Ashley: Thanks, Alan. Yes. I’m CEO of Accelerated Strategies Group, which is an analyst firm that focuses on, in addition to cyber security, digitial transformation, DevOps, and cloud-native kinds of applications, which really matches my background very well. I’ve been both a product creator CEO/CTO as well as running IT, CIO, CSO – that kind of thing. So, currently, I’m also CTO with MediaOps – I guess CSO at MediaOps, too, so, I’m trying to collect as many three or four-letter titles as I can. So far so good.

    Shimel: Alrighty. So, first of all, thank you all for coming on today. Secondly, testing – what a wild, mixed-up world we have in testing, right? Just from looking at the folks on our panel today, we have security testing – pen testing, in particular. We have the world of open source and what a factor that is in testing.

    We have product testing, right? We have sort of new product testing – taking testing to new heights. And you know, that variety is sort of a double-edged sword, right? Because it’s like jack of all trades, master of none type of thing. There’s so many aspects of testing, right?

    Even in our audience here at this virtual event, we have testers who come from, you know, what’s called old line QA testing, right, to DevOps enabled testing to visual testing – testers who, if it’s not an open source testing platform, they won’t use it, to proprietary systems. Through it all, though, a lot of times, all of this testing becomes sort of a red-headed stepchild or a whipping boy, if you will, right, to the whole application development system operations teams. It’s like, at the same time, their best friend and their worst enemy, right? They want everything tested, but they don’t want to give you the time, the budget, or the resources to do it, and then, when something doesn’t work, they said, “Well, we should have tested more, right?” So, how – first of all, do we agree that testing has been sort of the whipping boy of a lot of what we’ve seen go down in app dev and operations? Anybody want to jump on that?

    Chase: Well, I think it’s certainly gotten a bad name. I think some of that comes from prior to the digital transformation. A lot of the engineering was waterfall, and testing was always that last stop before the software went out live, you know? Nowadays, I think everyone’s come to realize the speed and scale at which we develop things, you have to do the testing up front at the very beginning. You can’t wait till the end, you know?

    You’re gonna be too slow. So, I think, from that perspective, absolutely. I think that’s why testing has really, I think, gotten a bad name, because they’re kind of the blockers to getting anything out into the public domain, whereas nowadays, you have to enable that up front because you want to get it into the public domain yesterday.

    Shimel: Yeah. And so, in many ways, to me, that sounds like, well, DevOps has been the best thing that ever happened to testing, right? In many ways, DevOps has been the best thing that’s happened to security testing, right? We’ve certainly seen that shift left, if you will. But has that shift left helped – has that rising tide lifted all boats?

    Miranda: I still see –

    Shimel: Oh, okay, Tracy. Go ahead.

    Miranda: I’m gonna jump in and say one thing that strikes me as funny. Like, we have, in our industry, everybody has embraced the DevOps terms, and then, you start to see all these spinoffs to put emphasis on things like, for security, we have DevSecOps, and you’ve got every variant of something with ops. But when it comes to testing, you know, there’s no TestOps. There’s no new buzzwords and it kind of speaks to how much it’s being neglected. Like, we had a conference on continuous delivery, and we asked for talks of everything in the life cycle, and we got lots and lots of submissions except for testing.

    So, again, I think it still is really under-appreciated and just not given the due credit for what it stands for.

    Shimel: Very –

    Simo: Yeah. And I think some of this comes from, as Alan mentioned, that testing is a very broad and diverse subject, and I’ve often watched people argue about testing that each interpret it to their piece of what they do with testing. And things like CI and some of the DevOps practices have helped us improve with the checking that our software can work – you know, demonstrating that it can work – and there’s a lot of people – if you primarily wear a developer hat, you’d look at that’s what testing is. And we’ve automated that.

    There’s been huge improvements in tooling and technology to do that, but sometimes, along with that is we throw out that we need someone involved that’s looking at things from a critical perspective that’s concerned about the safety and efficacy of what we’re delivering that is the, I guess, the one questioning and searching for information that they raise issues that we need to deal with that are reasons to not push things out to production now – so that we can make informed decisions. So, often, I think just that critical side gets thrown out as we improve the other side of testing, which is being able to demonstrate that things can work.

    Shimel: Anyone else?

    Ashley: You know, Alan, one of the ways you can tell who drew a diagram is to look at what’s in the center. And so, if we think about CICD, we kind of surround testing, but we don’t talk about testing. So, in some ways, it’s elevated, and some ways, it still sort of abstracts it out of DevOps. But my experience has been not really, because most teams know that testing has to get automated as part of that process, ’cause you can’t have this automated checking and automated deployment, then let’s kind of let a bunch of manual things happen. And not that testing hasn’t been automated – it has been – but I think it has elevated, at least in part, I think a good deal.

    My experience has been. I haven’t worked in every environment, maybe, where test still isn’t thought of very highly. By the way, the thing that the – it isn’t the last thing in the chain. Documentation is the last thing.

    Chase: That’s true. That’s true.

    Ashley: Let’s give documentation –

    Chase: And release notes. It’s true. I have tech doc’s team, too. You’re right. [Laughs]

    Shimel: And that hasn’t really changed much either, though testing has. No. No. But you know what? Caroline, I’d like you to focus in specifically.

    I mean, I think about – you know, Mitchell and I co-founded a security company, and we came out with a product in 2003 of just a vulnerability assessment and manager – “VAM”, we called it – and at that point, look, we used to beg people to scan their infrastructure at least once a year, right, if not quarterly. We were really pushing for quarterly. And you know, the idea of scanning and testing applications before you deployed them – unheard of. It was unheard of, right? We didn’t have the security tools or people.

    We did that after the fact. Now, that’s not normal anymore, right? Today, at least 85 percent of applications are tested prior to deployment. I don’t know who those other 15 percent are, but that’s a whole nother story. But talk to us.

    You’ve lived through this, Caroline. What do you think?

    Wong: Sure. So, you know, I think there needs to be an acknowledgment. There’s sort of these like, first order priorities and these second order priorities. When I think about security – and I think some of these concepts apply to testing more broadly – there’s a saying that, “You don’t take a $200.00 fence to protect a $5.00 asset.” I think security is about you have something of value and so, you want to test it – rather, so, you want to secure it.

    Testing, I think, is like you made something new and so, you want to test it. You’re not gonna test something unless you have something new. You’re not gonna secure something unless you have something of value. So, it’s okay that security and testing are not sort of the number one thing, you know? Inherently, by their definition, it’s sort of a second order attribute, if you will.

    Now, I think, you know, each of us has been through a version of annual organizational budgeting. You know, each organization says, “Okay. Here are the goals for the years. We’ve got limited resources and budget to accomplish those goals. How are we gonna allocated those resources?”

    And there are some things that, for a lot of organizations, are very clear. You know, even if I take a step way back and I say, “Okay. Most organizations might think about budget allocations in three buckets. Number one – general and accounting; number two – sales and marketing; number three – research and development.” You know, nowhere in those words is either security or testing.

    And that’s okay, you know? I think, Alan, when you share your story about VAM, you know, Vulnerability Assessment and Management, I agree that over the last, I would say, decade-two decades – so, when I started my security career 15 years ago, we were almost trying to convince folks that breaches really happened – that this was really a thing to be worried about. You know, today, everyone knows because it’s on the media headlines every day. This is real. And so, that has been a shift in the perception, I think, in our industry.

    There’s also been a big compliance driver. So, the – in the area that I’m very familiar with, security pen testing, a lot of that pen testing is actually driven by external compliance requirements – things that say, “Because of compliance framework A, B, C, you must actually conduct manual pentest on your application, for example, once a year.” You know, doing it more often than that is often because of perhaps what a lot of this group might think of as a really good thing – trying to actually secure your software, trying to actually test your software to ensure it does what you think and what you intend for it to do, but that’s not necessarily the first thing that comes to an executive’s mind when they’re thinking about budgeting, and there’s a variety of reasons for that, right? There’s an external perception thing. I think there’s also a lot of learning, certainly, that the security industry can do to be more approachable and to really understand better what is the software development methodology that we’re engaging with.

    What is the specific methodology and behaviors of these individuals? And how can we actually collaborate with them most effectively?

    Shimel: Fair enough. I mean, look, I don’t want to be the pessimist in the group or the bluebird of happiness here, but we have had success over the last 5-10 years, Caroline. You’re right. Especially in security, but not just security. We’ve had success in testing across the board.

    Tracy, to your point, the fact that we don’t have DevTestOps, I think, is, you know, testament to the fact that we’ve recognized that testing is so critical to the CICD process that it’s part of it. I mean, you can’t have DevOps, you can’t have CICD without a level of continuous automated testing in there – in my mind, anyway. I mean, prove me wrong. But, you know, I think that has been probably one of the biggest successes that the DevOps movement has brought – has been responsible for.

    Miranda: Yeah. And I really like Caroline’s analogy with security because, you know, that’s something else we think of as fundamental to DevOps, but it’s sort of elevated itself and, in a relatively short space of time, went from being ignored to being, you know, headline news. And I think there’s a lot of lessons there. And I – like, I will say, I do think testing is going through that journey and we’re probably on the cusp of it kind of really exploding and watching a few players in the industry and companies starting to do really interesting work in testing. I think that’s gonna turn around very shortly.

    So, yeah, there’s an exciting space that I think its moment is coming – just like securities has.

    Chase: Yeah. I think it’s all about the testing now, quite honestly. I mean, when you look at the future with low-code/no-code, it all becomes question-question mark ops, you know? Any function in a company could actually put something together and run it, and they’re gonna have to test it, you know, and it becomes less about all the engineering and more about the test patterns and things like that that are needed as you get to these future states. So – and speed and scale is gonna continue, I think.

    That’s driven everything with the digital transformation. It’s speed and scale. Everything needs to come faster, and we want more of it, you know? And so, I think that really pushes the envelope, and that’s what DevOps has responded to and, I think, addressed very, very well, but it’s not over. We’re gonna continue to scale up and go even faster.

    So, we have to be prepared for that, and I think it becomes more about testing in the future, and less about, you know, we’re gonna have these pre-built components – APIs you can piece together however you want – then, testing becomes, I think, almost first order, to Caroline’s point. And security, especially. I think security, we’re seeing that change now. If you’re a SAS company especially, security has to be top in line, ’cause you just can’t put something out there and not have it tested from a security perspective. And, furthermore, with compliance, it’s not just ISO and GovCloud and those programs, but large corporations now are saying, “Hey, if you want software in my ecosystem, these are my rules and I want to see your pentest reports and I want to see this.”

    So, I think more and more people are realizing that if the software’s not tested, it’s not worth it. And so…

    Simo: And, in my experience, part of adopting a DevOps culture and not just tool set is putting testing at the forefront. Some of the most productive teams that I’ve worked with that deliver some of the highest quality software I’ve seen – testing was center to absolutely everything that everyone on those development teams did.

    Ashley: Hm-hmm.

    Chase: Yep.

    Shimel: Yeah.

    Miranda: Yeah. We think – go ahead.

    Ashley: I’m just gonna jump in and say – there’s another factor that actually works to our benefit and that is not only to speed through automation and DevOps process, but contemporary software architectures, containerized micro-services, service mesh – we’re releasing smaller bits of code in encapsulated forms, which means, you know, we’re not testing monolithic applications all the time anymore so, we’re not having to run two weeks’ worth of tests just to get one pass through.

    Chase: Yeah.

    Simo: Right. _____ container solved the “It works on my machine” problem because we can ship the developer’s machine.

    Chase: Yep.

    Shimel: Yeah. That was always a big problem, right? Tracy, you were gonna say something though.

    Miranda: Yeah. No. I think the direction we see it heading is we’ll start breaking the testing monolith to talk about what is unit test, what is system test, and appreciate that is a specialization in itself. And the other things I’m starting to see from some of the end user teams we talk to is this real focus on culture – you know, a continuous quality culture – and seeing that hand in hand as part of the testing movement and again, _____ back to security – you know, the security first culture went hand in hand with that whole emphasis on security so, similarly, the continuous quality culture is starting to be something that – like, the really high performing teas we see pay a lot of attention to.

    Shimel: Absolutely. You know, Kurt, you said something there about low-code/no-code and how that could be a game changer. You know, some people refer to that as the rise of the citizen developer, and I’m wondering if maybe what we need is the rise of the citizen tester, right? Is testing too hard?

    Chase: Yes. I waste gonna say – the one thing we haven’t touched upon – I think Caroline did – is testing is hard. Security testing is super hard. And when you look at today’s modern environment and a lot of the ISVs – at least the ones I’ve seen and been associated with – you have a legacy of monolithic applications that you have to support that are bringing in a ton of revenue, and, at the same time, you’re transitioning – you know, going through your own digital transformation. So, that adds complexity after complexity in testing, and I do think, in the future, testing – you know, a lot of us, like Ben said, to his point, we have to push testing up – you know, shift left, move it far left as we can.

    And that’s gonna continue. I think that has to be the norm. To Caroline’s point, it has to become first order. And I think it’s starting to right now. I really do. Especially the – all the SAS solutions that are out there – you know, shipping code daily, hourly. You need excellent testing to do that and do it with competence.

    Shimel: Yeah. I think you do. But we do have to – and it’s funny that we’re talking about making testing easier because – not security testing, right? Security testing, pentesters – they were always sort of the hired guns of the security space, right? There were SAS or – there weren’t companies like Cobalt, right?

    The pentesters that I knew growing up – growing up – you know, that I knew coming up in the security space were gun slingers. These were, you know, white hat/gray hat folks who would come in and they’d do a little social engineering and pen testing using some open source tools, and you know, they give you a report. But you know, these were folks who traveled around the world doing that kind of thing. You know, we didn’t have a company like Cobalt that did this as a service, so to speak. So, it’s very different.

    But here’s the underlying thing with like, QA kind of testing – and, you know, I don’t mean any disrespect to anyone in our audience today, but for too long, a lot of folks in testing were viewed as failed developers or people who couldn’t be really good developers did testing or maybe testing was a weigh station on the way to becoming a real developer – like a Pinocchio stop or something, right, where you’re not a real boy yet. But that’s not true. There’s also – there’s a whole cadre of professional testers. And you know, in their capable hands, you could do great things. But I guess the question is – does testing have to be so hard that we need professional testing – professional testers to do testing?

    Or can we make testing automated enough that you – I’m not saying it has to be the dunces who don’t become developers. That’s ridiculous. But can we broaden the base of people who do testing? And that includes security testing? I’ll throw it out. Ben, you were a big-time guy in the testing industry. Tell us – what do you think?

    Simo: So, I think we can expand who’s involved in testing and that more people than those currently doing testing can learn to do testing. I don’t know _____ developers learning to become better testers and test their stuff. But I will say that ultimately, the part of the work that is the testing cannot be automated. Tasks that we do in testing can be automated. So, whether – not matter what hat someone primarily wears, if it’s a developer, you know, writing tests to demonstrate their code works or someone’s using TDD and they’re building some tests and then, building some code that matches it, the testing is the critical evaluation that’s going on in their mind of solving the problem and demonstrating that it’s solved – or evaluating how something might go wrong and coming up with some approach to demonstrate that it may go wrong and check for that.

    So, those things, whether you’re doing it automated or the execution is automated or done manually, the actually testing work, I’d argue, is the cognitive work that people are doing. And there is some – that we can never automate away. There are some ways that I think AI can help us. There’s also some ways that may lead us astray. But ultimately, I think we need people to learn to be critical thinkers to understand the technology, to understand the domain of the users and the problems that are being solved, and applying that critical thinking is the testing.

    Of course, then, we want to use tools to help us with some of our tests so, when we come into DevOps and CI, executing those demonstrations of the software working is a key part of that. Now, whether we do it as part of our, you know, building to those tests or we create them afterwards, then they can, at the very minimum, act as changed detectors going forward. But there was some testing in people’s minds that was going on that created those.

    Shimel: Caroline, what about – okay.

    Wong: I agree with quite a lot that Ben is saying. You know, I can’t help but think to myself that – I actually have this sort of visual in my mind, and it’s actually at a circus of a trapeze artists, you know? I think about these people, and they’re sort of like, whirling in the air and they’re doing these like, really incredible feats, you know, and they just do whatever they do, you know? But there’s like, a safety net to catch them if something goes wrong. It also reminds me of a metaphor that is often used in security that says, “We put brakes on a car so that you can go fast.”

    You know, when Mitch said earlier, you know, “These days, we’re releasing smaller pieces of code, modules of code” and there’s a way in which, yes, each small piece of code – each module – can be tested and some of that can be automated. We should automate performance testing. We should automate unit testing. But, to Ben’s point, there’s a lot that can’t be automated. There’s a lot of testing that actually depends on things like human judgment and opinion and it actually requires a person’s decision, you know?

    So, I think there’s – I think there’s a couple of frameworks for us to consider, right? If the first order is about creating value and making something new and the second order is about testing to make sure it does what you wanted it to do and protecting the value created, there’s also this spectrum that says there’s smaller bite-size stuff that may be a good candidate for automated testing, but, I think it’s super important to recognize that these small pieces of code – these modules that Mitch was talking about – it’s also super important to test what happens when they come together. So, to the extent that security folks need to be able to test chained exploits and business logic flaws, you know, those are exactly the types of things that will never be automated – is what I think, anyway.

    Shimel: I don’t disagree. I think security in particular is just there’s some things you’re just not gonna automate. I don’t care what you say. But that’s okay. Anyone else?

    Chase: I do want to make a comment on something you said. This was a long time ago now, when I was a dev manager, that I used to sense that when my developers – you know, if you were on my team, if you wanted to really get me mad, treat your tester badly. Treat them, you know, a bit, like you said, like they’re not quite at the level you are. I would be incensed as a manager. You know, that was a long time ago.

    Fortunately, I don’t see that as much nowadays. I think that role has been elevated where I don’t feel that at all on my teams, and if I do, that is, again, to this day, I’ll get super mad about that because I think it takes all kinds to develop software today and ship software. Testing role is just as critical as an engineering role, just as critical as the ops’ role, the security role. So, yeah, that was one thing that would drive me nuts ’cause I did sense that and go through that, and this back in, you know, the late ’80s-’90s, if you will, before the year 2000 – I’m dating myself – but it definitely was palpable in some of the large ISVs, you know? And as a manager, that was something that would drive me nuts.

    Shimel: Agreed. Agreed. Tracy, open source – look, testing has been inexplicably wound up with open source for as long as I’ve been around, right? Whether it’s security – open source security testing and you know, things like Metasploit and Nessus and Nmap or you know, in QA testing kind of stuff, it’s a rich history of open source testing tools. How do – and part of your mission at CICD is – at the CICD Foundation – is to nurture that, I’m going to assume, correct?

    So, how do you guys nurture that? What are you – what do we do to maintain that rich, open source tradition here?

    Miranda: Yeah. No. Absolutely. And, as I say, we at the Continuous Delivery Foundation, we want to look across the entire software delivery life cycle, and I think it is safe to say, you know, that testing hasn’t risen to the top. We’re actively – like the board wants it to be a more focused-on topic, the technical committee wants to encourage folks to bring us more projects in open source and with testing, but we’re still having to kind of fight through all the noise around other projects.

    And one of the ways we’re approaching it is, for example, our technical oversight committee wants to set out a landscape of the types of projects in each kind of bucket and say, “Okay, look. We need to have something in security. We need to have something in testing that covers these different aspects.” Almost like a call to action to say, “Bring us your testing projects and –” like my favorite saying about testing is, “There’s two key problems in testing – not enough tests and too many tests.” And yeah, I think we’re starting to see some really good frameworks around test generation.

    I’d love to see more open source options for that. And Kohsuke Kawaguchi, the famous creator of Jenkins, move on to focus on testing so, that says a lot about the importance of this area getting more open source focused. So, yeah, I’m excited to see what could come out, but we do have to actively encourage folks to come join us and have those conversations and not to, you know, have it assumed it’s all focused on delivery of pipelines for delivering software.

    Chase: I think another thing you’re seeing today is a lot of people, when they start developing an app, when they start creating something new, that’s the question they ask up front. “How are we gonna test this? How are we gonna automate it? How are we gonna pentest it?” I think those things, at least, are starting to bubble up a lot stronger in the cycle – you know, early on. ‘Cause I know that’s something we recommend highly – is that if you’re developing something new or need to head down a digital transformation, think about the testing. How are you gonna test that up front, you know? And products are actually being developed with that mindset now.

    Simo: And with short iterations, it’s how are we going to test this tomorrow, next week? It can’t be left off for months later to be someone else’s problem. So, people, I think, are learning to deal with those questions about testing from the start of their discussions about what the requirements are.

    Chase: The other thing I love – and in Caroline’s world, has been very successful – I know it is specialized skill set, but trying to teach that throughout your troops – you know, throughout the engineering team. Show them what, you know, across _____ scripting errors. What does it look like in code? That’s super successful. That really, really helps make – ’cause security is difficult, you know, no matter how you slice it.

    Even when you’re looking at product development – if you want to invest in fixing security defects or add a cool new thing for your users, you know, a lot of times, the cool new thing is gonna win out. But that’s starting to change as well, too. I think people are starting to value security – especially in today’s modern world, you know? It has to be up front, really.

    Ashley: Yeah. It’s all about AppSec at the moment. That’s one of the hot topics as well. You know, someone mentioned AI earlier. I just want to offer a perspective about that because we always seem to focus on the shiny new object.

    I know I have. I often do. But I think what we’ll see with AI is select applications when it comes to testing. The sort of vertical place is where it’s really well suited. When we say, “Yeah, AI”, we really mean machine learning in most cases, which is machine learning algorithms, and there’s two types – there’s supervised/unsupervised.

    Supervised is, “That’s a car. That’s a car. That’s not a car, right? That’s a bug. That’s a bug.”

    And unsupervised takes massive amounts of data and intellectual patterns and learnings from that. So, I think that as we’re taking our test automation and our test tools, there may be some of that, and it might be really useful for AI. I wouldn’t get worried about people being replaced as testers. Not in my lifetime, that’s for sure. It is still a – there’s very much a craft to it, just like there is software design.

    There is a craft to testing and a skill and a mentality about it that’s unique, just like with security.

    Shimel: No doubt. Caroline – oh, go ahead, Ben. I’m sorry. No, no, Ben. Go.

    Simo: Yeah. I was gonna say – and with any other sorts of automation – even you know, not learning automation that’s come before AI and machine learning based things, is that it not only has potential, in some cases, to replace some of the mundane work we do, but to enable to the skilled people to focus their work and apply their brain power to the things that really matter. So, I think there’s great potential for AI to help direct people’s attention to the things that maybe should be of concern and help deal with that big question of testing out of infinite possibilities of PaaS and data variations. Which one’s gonna show me the problem? And I think there’s huge potential for AI to help us find that if we’re also aware it may guide our eyes away from something that we do want to see.

    Shimel: Caroline, how does that relate to security, though? I mean, is AI helping Cobalt with pen testing? Be honest.

    Caroline Wong: So, the type of software that we’re building today is actually workflow and collaboration software. So, what we’re trying to do, is we’re trying to bring security folks and developers and penetration testers all together sort of in one place where they can collaborate. The technology piece, for us, is not about automating defect discovery. It’s not – I mean, if we’re gonna us ML or AI in the future, honestly, what we’re gonna do is we’re gonna use it to say, “What are the characteristics of this organization that wants us to do a test on this particular type of asset? What are the attributes of that asset and how might we match that to our talent pool – something that we do totally manually today?”

    I think that’s the opportunity. I don’t think that we’re gonna get to a point where we’re using the AI and the ML for automating the defect discovery itself. Frankly, I think that even if I take a step away from pen testing and I look at the security scans that are in existence today, I think, you know, whether or not they’re referred to as using AI or ML techniques, they’re just simply imperfect, you know? And there’s a super important place for them, but there’s a lot of false positives. There’s a lot of interpretation of data.

    And I think that’s always gonna be the case. I think that no matter what, when it comes to humans and machines, the machines can add a lot of value, but I don’t know that when it comes to software development testing and security in particular that there’s every gonna be a complete substitution. I don’t think that – I don’t think that makes sense.

    Shimel: Absolutely. So, guys, we’re coming up on the end of our time. I need to wrap up. Tracy, let me say this for you, if you don’t mind – and if I’m wrong, correct me. For you testers out there, the CICD Foundation – or CD Foundation, actually; I keep – we always say “CI” like it’s attached at the hip – but the CD Foundation is looking to hear from the testing community.

    Probably some companies – including Tricentis – may want to look into becoming involved with CD Foundation because if you don’t have a seat at the table, you don’t have a right to complain about not being heard. And so, you know, when we look at foundations like Linux Foundation and the CD Foundation, CNCF, we need the testing community – vendors, professionals, folks in it, you know – to be involved and get involved. Be heard and be part of it. So, I’m gonna extend that out there to everyone. But, short of that, though, Caroline, Tracy, Kurt, Ben, and, of course, Mitch, thank you so much for joining us on this special edition of DevOps Unbound.

    We hope everyone enjoys the rest of their Tricentis Virtual Summit. This is Alan Shimel for MediaOps, devopsy.kinsta.cloud, Security Boulevard, Container Journal, Tech Strong TV. Have a great day, everyone, and thanks for joining us.

    [Musical outro]

    [End of Audio]

  • EP 15: Why Testing is so Frustrating

    EP 15: Why Testing is so Frustrating

    In this episode of DevOps Unbound, hosts Alan Shimel and Mitch Ashley are joined by Adam Kalsey of Tricentis, Brian Dawson of the Linux Foundation and Caroline Wong of Cobalt to discuss why testing is considered the most frustrating bottleneck for developers and engineers. Panelists will explain the benefits of testing early on in the SDLC and how to accelerate the feedback cycle with automated and continuous testing, as well the pros and cons of testing, the value of shift-left testing and how to effectively balance quality and speed. The video is below, followed by a transcript.

    DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.

     

    Alan Shimel: Hey, everyone. I’m Alan Shimel, and you’re watching DevOps Unbound. DevOps Unbound is sponsored by our good friends at Tricentis, and it consists of a biweekly, 40, 45 minute video panel on relevant topics within the DevOps sphere, and featuring a panel of experts that rotates every other show, and really great people, and something I personally really enjoy doing.

    And then, about once a month or once every six weeks, we actually have a live studio audience—or there is no studio audiences in COVID, actually, but we have a live virtual audience that participates in our show. Today’s show is just our panel, though, without a live audience, and I’m really looking forward to our discussion.

    Our discussion today is around testing. You know, with the pressure on developers to deliver code faster, sometimes testing is considered a road bump in the way. Testing is considered slowing down the whole process. There are a lot of developers who’d like to push testing off that plate. Of course, no one wants to go back to testing after we’ve deployed software, primarily, but there is that tension there.

    Let me introduce you to our panel today; we’re gonna discuss this in depth. First of all, making his first appearance as a member of The Linux Foundation on our Tech Strong TV network is our old friend, Brian Dawson. Brian, welcome to DevOps Unbound.

    Brian Dawson: Thank you, Alan. Thank you, Alan. And it’s fitting that the first panel, first webinar I do since joining The Linux Foundation is with you.

    Shimel: It’s our pleasure to have you on. Brian, what’s your position at Linux Foundation now?

    Dawson: So, at The Linux Foundation, I am functioning as the VP of Developer Relations and Ecosystem Development. So, just to help, we’re gonna help The Linux Foundation to the development community while driving more of a DevOps opinion into a growing ecosystem of solutions that support open source development.

    Shimel: Fantastic. Next up is my good friend, Caroline Wong. Caroline is—oh, she’s an anchor on many of our different shows here at MediaOps and Tech Strong. Caroline, it’s great to see you again. A little background, maybe, for people who aren’t familiar with you?

    Caroline Wong: Great to be here. Thank you so much. My name’s Caroline Wong, I’m the Chief Strategy Officer at Cobalt. We’re a pen test as a service company.

    I actually started my DevSecOps career 16 years ago leading information security teams at eBay and Zynga. So, in both cases, we were running online operations 24/7 with millions of simultaneous users daily. And now, at Cobalt, we build security software. And, like many DevOps companies, we’ve got data driven, product based teams. We value automation and failing fast, and super excited or today’s conversation.

    Shimel: Fantastic. Welcome. Great to have you, Caroline. Our third panel member is another returning DevOps Unbound veteran, and that’s Adam Kalsey of Tricentis. Adam, welcome.

    Adam Kalsey: Thanks. Thanks for having me. I lead Developer Relations for Tricentis. I’ve been building web applications since 1995, and have always had a soft spot in my heart for bad applications and broken things. And so, I love to help developers build higher quality software and do it in a more efficient, faster way.

    Shimel: Absolutely, and thanks for being on with us today, Adam. Last but not least, in his Florida outfit here, visiting me in Florida, is my Co-host of DevOps Unbound, Mitch Ashley. Mitchell, welcome.

    Mitch Ashley: Coming to you from the MediaOps Studio B.

    Shimel: Yes.

    Ashley: Great to be here. I’m a former product, both security and software SaaS, CTO as well as a CIO, and now I run Accelerated Strategies Group, an analyst firm that focuses on those things.

    Shimel: Absolutely. Hey, Mitch, thank you.

    So, let’s get started. Adam, you’re with Tricentis, the sponsor of DevOps Unbound, and they’re a testing company, probably the leaders, one of the leaders worldwide in automated testing, continuous testing. Have we put too many straws on the developers’ camel’s back by making them, you know, be more responsible for testing to say we need to test more of our code earlier.

    Is it a bridge too far for many developers? Is it making them resent testing? You know, what’s the current state?

    Kalsey: I think where we’re failing developers is by handing them lots of more things to do without giving them any—without giving them what tools, and I don’t mean just necessarily software tools, but without giving them the things that they need to do it effectively. If we just pile things on developers and say, “Oh, this is your job now” but we don’t give them the support, we don’t give them the training, we don’t give them the expertise, we don’t give them anything to make that better—sure, it’s just gonna be harder on ‘em.

    Shimel: To play devil’s advocate—oh, those poor little testers making a quarter of a million dollars a year each. Do we want ‘em to do testing, too? Right? Is there some of that here, Brian? Or—go ahead, Adam?

    Kalsey: No, no, Brian—please, please, jump in.

    Shimel: Well, he’s from The Linux Foundation, he now represents these folks. [Laughter]

    Dawson: Well, first, not all of them make a quarter of a million dollars a year, right? But I do think what you raise is important when you bring up the investment, right? We are hiring talent with a particular expertise in a particular domain, a particular ability to solve problems and innovate.

    And there is an argument to be said, Alan, that, “Do I want to spend this money, this quarter of a million dollars plus 200,000 in RSUs on having somebody write test harnesses?” Now, I think—you know, that can be balanced by what Adam said. Well, first of all, yes, because we want everybody thinking about quality. But you know what, if we want a return on that investment, we wanna free the cognitive load, then you know what? We need to invest in automation, we need to invest in better collaboration and reducing friction from the process.

    Shimel: Agreed, agreed. Caroline, you know, it’s been a big—this whole DevSecOps movement and shift left has been about shifting security testing left and getting developers more involved in security testing, right? Yet another camel on the—yet another straw on the camel’s back. Should we be doing that?

    Wong: So, I—oh, my Internet is awful. Can you hear me?

    Shimel: Yes, we hear you fine.

    Wong: Okay, fantastic, awesome. I’ll just—I know that there’s post editing, so it’s okay for me to say that.

    Shimel: Mm-hmm.

    Wong: So, Alan, I bring a particularly unique perspective to this group in the sense that I actually oversee people at our company, I oversee Human Resources. And I’ve been with the company since we were 10 people, and today we’re 200. And one of the super interesting things that happens is, naturally, as a company grows, what people do, they specialize and then we have to figure out how to communicate more effectively.

    And I think that if you’ve got an engineering team and you say, “Okay, now you’re responsible for Development and you’re responsible for Ops, and your responsible for security, then there’s got to be some level, depending on the maturity and the size of the organization and what the organization’s trying to accomplish of specialized roles and protocols for engagement.”

    And I think that when, either number one, those roles are not clear to, I think Adam was making a point earlier about security folks need to be providing guidance to engineers where—because engineers are not expected to be security experts, frankly. Security experts are not expected to be engineers. And then, at the point where information is being handed off where the responsibility for taking an action, those, all of those sort of procedures, they have to be clear, otherwise they’re simply not going to happen.

    Shimel: Yeah, agreed. I think they have to be clear. But even if we make them as clear as we possibly can, is it just—are we putting too much on developers? I mean, I think fundamentally, that’s the question here, right?

    Wong: There’s—

    Ashley: Alan, it’s not only—sorry, I’m just gonna jump in. It’s not only quantity of work—is it the right work?

    Shimel: Right.

    Ashley: Because I know from writing software and being part of software teams for so many years, there’s this thing called unintended consequences or unintended use. Well, you’re not supposed to use it that way, but the person who designed it intended it to be used a certain way, but one of the important functions of tests is not just to uncover bugs and elicit those things or security issues, it’s also to push the boundaries about how you might use the software and essentially stretch it to try to find where it’s gonna break where they can. And oftentimes, that’s a different mentality than or perspective, certainly, than the person that wrote the code.

    Dawson: And if I may jump in and say maybe part of the discussion can get to generalist versus specialist, right? There is this idea, Alan, that when you talk about putting a lot in the developer’s hands, this idea that there is this prolific person called the full stack developer that can separate themselves from what they develop to look for security vulnerabilities, to test it, then ultimately deploy and run it, right? And I think we need to give thought to, are we trying to build a bunch of generalists, which doesn’t potentially enable them to do any job the best, or are we enabling specialists to better work together, share information, and crowdsource quality and crowdsource security?

    And I’m just curious if any of the other panelists have an opinion on this concept of full stack developers versus, say, needing traditional, lifelong quality experts.

    Kalsey: Yeah, I like to say that we decided to create cross-functional teams and we did it by removing all of the functions, and we centralized them all in a single person—and that’s not a cross-functional team. That’s—

    Shimel: That’s a cross-functional person.

    Kalsey: Yes. Well, and you end up with the jack of all trades, master of none problem. You have to have the specialists on even a cross-functional team. You have to have, there’s a lack of testers, there’s a lack of UX people, there’s a lack of security people. They have to be part of that team. You have to have people that, dedicated folks to do the hard work. A developer may be able, and a generalist may be able to do a lot of the easy things and a lot of the, “Oh, yeah, we’ve done this before or we’ve seen this, and this is a known know, let’s go fix that.”

    It’s the hard work of figuring out—what don’t we know, and then having the expertise to be able to know what that hard work is, even, to go after it. And without that, of course the developers are going to be overloaded, they’re being asked to do everything and they don’t know what everything is.

    Shimel: Agreed, agreed. I mean, I think, again, security is the poster child for this, right? We’re asking developers to be security people or to make security a priority. Now, you ask most security people four or five years ago, they’d tell you you’re smoking your socks. Developers don’t care about security. They never did. Only we care about security.

    It’s changed. It’s changed a little bit. We do think other—no one wants to develop insecure code. But we can’t ask them to be security people—or can we, Caroline?

    Wong: I sort of think we have to. I think that, you know, the big, bad thing that no one wants to happen is a security breach, and a security breach cripples an engineering team’s ability to execute on a roadmap. You know, if something big and bad happens, then entire teams have to rush and commit to fixing this problem sometimes for months, you know, time which, of course, could be invested more profitably elsewhere.

    I have a quote that I’d love to share from a colleague. Denali Lumma is the VP of Engineering over at Project Ronin, and she also makes a really good point about tech debt. She says, at a certain point, it’s too late, there’s too much tech debt, too much complexity, the code base is too big, and it becomes impossible to do anything about it. What you see there is a business will slowly die because shipping features becomes harder and harder, critical vulnerabilities and bugs are introduced more frequently, and eventually it kills the business.

    You know, I often think of software as a living organism, right? Whether you think of it as a tree or as an animal, there is this balance between making sure you’re doing sort of your fun things, your running, but you’re also resting when you need to heal. You’re also—you know, if we’re talking about a human being, you’re also sort of going to the doctor once a year.

    Shimel: Agreed.

    Wong: Yeah.

    Shimel: Agreed. Thoughts?

    Kalsey: Quality has to be a mindset, security has to be a mindset, user experience has to be a mindset on the developer time, among everybody there. The user experience person needs to understand and have a mindset of developing and of creating a secure experience and not, “Well, we’re gonna get rid of all the security because that’s hard for people to use.” Everybody has to have that mindset, but you still need the experts and you still need the expertise to be able to do things well. Just having a mindset doesn’t make you fantastic at it. You can say, “Oh, I love to get out and I’ve got an exercise mindset, so I’m gonna go run a triathlon tomorrow.” It’s probably not going to work.

    Shimel: Probably not.

    Ashley: You know, it’s interesting—we could’ve had this conversation 20 years ago, 10 years ago.

    Shimel: Absolutely.

    Ashley: You know, why are we still frustrated with tests? What is it with tests that is, you know, it used to be sort of the orphan of the development process, you know, the only thing that happens later than test is documentation. Well, now, [Laughter] it’s still an issue. Why is that? Why are we so frustrated with tests today? I’m curious what folks think.

    Dawson: I’m gonna start with one not very complex thought—it’s actually hard.

    Shimel: It damn right is.

    Ashley: A little more respect—we need a little more respect for tests. [Laughter]

    Shimel: Well, but this was like, I remember when we first launched devopsy.kinsta.cloud, a popular theme was that, you know, DevOps would kill the testing star, right? That there would be less QA jobs because everything was gonna be automated continuously and shift left, we wouldn’t need testers—poppycock, right? We have as many testers as we ever had, if not more. It didn’t kill off testing jobs. I think it was Adam that said we’ve made testing part of the DevOps team. So, we have people on the DevOps team who are responsible for the various, you know, styles and types of testing that we do, including security.

    I think what’s happened, though, is, these DevOps teams—when I first started devopsy.kinsta.cloud, if you spoke to developers, they’d say, “Oh, this DevOps is all about the Ops.” And then you’d talk to the Ops people and they’d say, “This is DevOps, it’s all about Dev, it’s Dev-centric,” right? And that’s the sign of a good compromise when they’re both pointing the finger at that.

    But it has come—at least as far as testing goes, from what I see, it has sort of tilted where, you know, as we’ve shifted testing left, we’ve shifted responsibility left as well, which kinda winds up closer to the Dev, or certainly to the DevOps teams.

    However, though, let’s look at the alternative, and I’m gonna ask all three of you—all four of you have been around the block once or twice. My personal feeling is, is that we do a much better job of test coverage today than we did 7 or 10 years ago, and I think DevOps and continuous automated testing is a big reason for that.

    Thoughts? Anybody disagree? Show me wrong.

    Wong: I will challenge you, Alan.

    Shimel: Okay. I was hoping someone did.

    Wong: I’ll challenge you, and the reason is because—so, we’re all familiar with the OWASP Top Ten.

    Shimel: Very much.

    Wong: There have been a handful of versions of the OWASP Top Ten since the first was produced in 2003. I believe we’re gonna see another version later this year. The current version is from 2017, and if you take the 2017 version of the OWASP Top Ten and you put it right next to the 2003 version, it is sort of mind blowing how similar they are.

    Shimel: Yeah.

    Wong: And what that says to me is, we actually know exactly what the problems are. We know how to find them, we know how to fix them, and we know how to prevent them, but they’re still around, and so I think that’s—

    Shimel: They are, but are there as many of them, right?

    Wong: Absolutely. [Laughter]

    Shimel: You think there’s as many cross-site scripting or buffer overflows than there was 10 or 15 years ago?

    Wong: So, there are some categories that have been eliminated. You know, there are some categories that, for which we can define a secure pattern, and we can encourage folks to use that secure pattern, and for which we can actually eliminate some of these things.

    But there’s others or which we cannot. You know, I think about—I recently visited my sister’s home, and they have two young children, a 2-year-old and a 4-year-old. And at night, what they do is, they run a Roomba, which is brilliant, you know? If you could spend your weekend playing at the park with your kids instead of vacuuming, of course you’d wanna do that. And the Roomba works because they sort of take the time to get all the toys off the floor and make sure everything’s clear and once they use the Roomba, they clean it. There is this different work involved in effectively using a Roomba that’s different from sort of manually using a vacuum.

    And I do think there are these two distinct categories of testing, one category which you could say is known, repeatable, and automatable, and another for which manual testing, I believe, will always need to exist. There are types of security flaws like business logic flaws, race conditions, chained exploits that I don’t believe that in my lifetime, we will be able to figure out how to teach a computer to find these particular types of issues.

    Kalsey: And it’s beyond just security testing. I mean, exploratory testing, from a tester, your automated tests are not going to find, “This error message makes no sense in this context. I did this, I expected—I don’t know what this error message is trying to tell me to do.”

    Your automated tools aren’t going to find that. All they’re going to be able to find is, “It didn’t work, we sent an error message. We had the things we wanted to put in the error message there.” You can test that your code is doing the right thing, but if you didn’t design it to do it right, you’re not gonna find that with an automated test.

    Shimel: Fair.

    Dawson: I would tend to say, for the surveys that we’ve issued and I’ve read, your original statement, “Has test coverage improved significantly?”—yes, I think all indications are, we are getting better at automated testing, we are getting better at that pursuit of that thing we call test coverage.

    But I think what’s more important is, has the quality of test improved? Are we testing the right things, right, from functional testing to security testing, various shades of security testing? And I’d say (1) yes, we have improved there by sort of the qualitative and quantitative measures that I’ve seen. But I also think, as we truly start to focus more on quality, we start to achieve more coverage, we start to implement better tests, it becomes more clear, as I think Adam and Caroline are speaking to how much more work there is to do, how much other things there are to improve testing on. Pair that with a constantly changing technological landscape, right? We’ve gotta learn new stacks, we’ve gotta figure out how to test new stacks. We’ve got half of our put in in a SaaS platform, but then we deploy on prem, et cetera, et cetera, et cetera. And then you add to that that, at the same time, we’re trying to transform organizations.

    So, Adam, you talked about exploratory testing; Caroline, the human element. I think one of the things that pursuit of the full stack Dev, that pursuit of the fully automated processes failed to do in a lot of orgs I speak to is, we talk quality first, but oftentimes, the DevOps transformation doesn’t involve quality engineering first, right? And I think one of the things we need to get better at is engaging with and partnering with Quality and establishing our automated and modern processes.

    Kalsey: Yeah. We talk good in the testing industry a lot about shift left and that testing is moving earlier in in the process, and too many organizations have taken that as, “Well, the developers are the ones that are early in the process, so that means the developers do it.” No! Shift your damn testers left. Get them involved in the design phase. Get them involved in talking about the architecture. Get them involved in the entire thing, not shift the task to the people who are earlier in the process. Shift the whole process, please.

    Dawson: Or to a script, yeah.

    Ashley: I’m gonna take a whole ‘nother tack, and I thought you were gonna go here, Brian, is, I think we need to shift tests down. We need full stack testing. It’s not application testing anymore. Most of the security errors that we have are due to configuration errors. And what Caroline said was absolutely true. We still have SQL injection, cross-site scripting, and overflows may be less so. But why? It’s because it’s not just in the app layer that those things happen, it’s in that software stack. And I know not every tester can be an expert at the stack itself, but maybe that’s, instead of kinda dividing the DAST and SAST and the different kinds of testing that we do and just deciding, can we shift those in one direction is, let’s think about it from a depth perspective, because that’s where a lot of those errors are coming from. It’s not just the app.

    Kalsey: Oh, yeah, and the tooling on this is also, can be super fragmented. You need one thing to test your infrastructure code, you need another thing to test your UI, you need another thing to do the security—and they’re not, consolidating those reports and combining them and figuring out what worked and everything is hard work and there’s nothing that does that. This is a place that Tricentis could definitely improve and improve our own tooling here.

    And then, once you’ve done that testing, the results are often in a silo. Okay, I found out I had a problem—how do I fix it? How do I go see the data behind what was going on in the system to understand this and solve this? And now I’ve gotta dive into a monitoring tool or my logs or an observability tool instead of just having everything all in one place.

    Shimel: So, I think there’s a—I’m sorry, Caroline, you go.

    Wong: No worries, and thank you, Alan. So, we’ve talked about shift left, shift right. Mitch is kinda talking about shift up and down. I think there’s another axis, if you will, which is sort of what I’ll call shift in and shift out. So, you can test things in and out of band, and that’s gonna make sort of a wildly different difference in terms of how those processes run.

    You know, I think 10 years ago, security people appreciated the power that came with being a gate in the process, being an end band process. You know, today, it actually makes sense, I think, for a lot of security testing to be out of band, but that removes the sort of power and authority of security people to say something like, “You must fix this, otherwise you’re not gonna go live if it’s happening out of band.”

    Shimel: Fair. So, I’m getting a little dizzy, we’re going left, right, up, down, in, out. But—

    Ashley: It’s a circle. [Laughter]

    Shimel: Yeah, you know, shake it all about. But here, fundamentally, though, are we just kinda dancing—no pun intended—are we just dancing around the real issue, which is, we wanna do more pre-deploy, we wanna do more, we want better quality software, right? I think we could all wrap our heads around that idea and vote for that. We all want better quality software. Security people are at a premium. We can’t shift security people left or en masse. Testing people are also at a premium, though we’ve integrated them into the DevOps teams. Developers are at a huge premium, they’re probably our most expensive resource in terms of humans.

    What do we need to do to make everyone’s job easier here, to make this function better? Is it a question we’ve gotta automate more? I mean, maybe that’s what it is, if we automate more, or is it a cultural thing, right, where—

    Ashley: There’s also test-driven design, right, which is, as you’re designing it, you talk about, you put together your test strategy of, “Okay, we do it this way. Let’s talk about how we are going to test it, start to lay out the framework for how we think this can be validated.” I mean, it’s not the only thing to do, but I think—so, if you start with that in mind, now you’ve got a path to pursue as opposed to, “Here’s a chunk of software, you folks figure out how to test it, right? We built it, you test it.”

    Dawson: Yeah, and I’d add, I think there’s no—again, not only is the practice of testing harder than we often like to admit, but the, you know, the path to kind of getting to where we fully wanna be, where we don’t dread testing, complex, hard path, and I’d say it’s the DevOps trinity, you know, Alan, and we all can probably hit a few points on each.

    But you have to address people and culture, as Adam brought in earlier. There’s no getting away from, security needs to be a thought for everybody, right? I can install all the locks I want on the front of my door. Actually, over 24 years—today is my anniversary—my wife has programmed me to lock the door when I come in the house without thinking, right? She’s kind of impressed a security culture on me.

    You do have to address people and culture, process, practice, tools and technology. Automation alone will not solve it. A mindset alone will not solve it. All of that needs to come to fruition.

    Shimel: Mitch, make a note—invite Brian’s wife to our next security panel.

    Dawson: [Laughter]

    Ashley: I’ve already looked up her e-mail address—thank you, Brian, appreciate it.

    Shimel: Happy anniversary, by the way, too, Brian.

    Dawson: Thank you, thank you.

    Kalsey: This is one of the places where I think what Caroline said is so important, that this in bound, out of bound—if you’re trying to do everything in line, of course, it’s gonna be harder. In shifting that mindset to doing things continually in the background, we already do so much stuff and process automation in the background where we don’t have people check, “Here’s a step in the process.” No, this is going along in the background, dependency watching and that sort of thing and updates.

    We can be doing testing in the background as well for a lot of it, and identifying what needs to be in the foreground and what needs to be in the background. Look, I’ve seen CI/CD pipelines that take hours and hours and hours to run because the tests take six hours to execute and to build. That can’t work in any sort of a reasonable DevOps system.

    Wong: I think there’s two parts to a solution, and I’m really, frankly, just parroting what Brian said. I think there’s a collaboration piece between the people. You know, to an earlier part of our conversation, you’ve gotta have the folks with those specialized skill sets. Ideally, you’re able to get access to those folks on demand, which historically has not been something that’s been possible, but that’s beginning to happen.

    The second thing is—so, you’ve got your people collaborating, you’ve also gotta have your technology integrated. So, for whatever security tool or processors, how do those outputs actually get to the engineers? What we need is bidirectional integrations for things like Dev ticketing systems. We need things like APIs for test results. I think collaboration at the people place and integration at the technology place—both of which we could call processes—then it’s just, it’s really a repeat of what [Audio skips].

    Shimel: Mm-hmm. I mean, but wasn’t this one of the promises of DevOps? I mean, I got into DevOps because I thought it was a better way to do security, truth be told. But one of the promises of DevOps is, we’re gonna automate more, so we could go faster with more quality—two out of three ain’t bad, right? And, you know, did we lose that somewhere? Is that no longer a goal, or have we taken that as far as it goes?

    Kalsey: We keep trying to solve it with technology instead of by looking at the culture and changing how people think. And you can’t think the same way and apply new technology to it and expect the technology just to magically solve all your problems.

     Shimel: Brian, you agree, or Caroline?

    Dawson: I absolutely 100 percent agree, and I’ll agree and then I’ll just add a point. And I know I’m being a bit nebulous about it, but there’s a human element to this. At the end of the day, it’s humans developing software for humans, right? Whether it’s the code on your electronic control module in your car or Waze on your phone or the OS, at the end of the day, the final endpoint is a human. And you know what? At least now, and I know we’re improving, there’s no way that we can ensure—and I don’t mean to accept security here, but to test the quality of the human experience, right?

    And think about this—at the end of the day, knowing that the language on a screen is self-descriptive and informs me what to do. Testing the fact that, or ensuring that when I submit my credit card number it’s not vulnerable, right? Whether we’re talking API testing, whether we’re talking security testing, all of this at the end of the day boils up to customer or user experience testing.

    And I guess what I’m gonna posit is that we can never entirely automate that. And where we’ve hurt ourselves and possibly made a false promise is, we’re gonna create a bunch of full stack engineers, we’re gonna get rid of, away from operations functions, testing functions, and we’re gonna use TDD and automate everything from the start, Gherkin, yadda yadda yadda. And I believe—look, that has enabled us to go faster, but our impedance mismatch, the limit we’ve hit right now is where we’ve failed to better connect people and enable them to collaborate and share knowledge. So, not just security, not just back end, not just data, but the entire customer experience is of quality. Dev can’t do that alone.

    Shimel: No, sir.

    Dawson: I’m sorry, Alan, did I get, did I—

    Alan Shimel: No, no, that’s all—I’m saying no, sir, you’re 100 percent correct, but I know Caroline wanted to say something.

    Wong: So, I do have a comment about automation. I think that DevOps has figured out how to do automation when it comes to defect discovery. I think we’re sort of good at that. I think what we’re still trying to figure out is automation when it comes to workflow and process so that, actually, again, to Brian’s point, the collaboration can happen automatically, rather than due to a human who’s involved and making it happen. How do we get the right information to the right people in the easiest, most efficient way at the right time, and how do we make it super easy for them to do their jobs?

    I do think that there remains—I do think that the next evolution of automation and DevOps is gonna go from defect discovery automation, which I think we’ve gotten quite good at, to workflow and process automation.

    Shimel: Fair enough. Adam, thoughts?

    Kalsey: I like it. I mean, identifying what you’re automating and what’s working in that automation is important, but also identifying what’s not being automated or what in your automation is broken. And not broken from the standpoint of, it doesn’t execute or it doesn’t run to completion, but broken in, it’s not solving the problem you wanted it to solve.

    Shimel: Fair enough. Guys, we have time for maybe one more question. I’m gonna pose it and throw it out there to you. Is it possible we’ve shifted too far left or that you can shift too far left and we need a little rubber band, here, going the other way, a little pendulum swing? Or you think we just scratched the surface? Caroline, you look like you wanna say something.

    Wong: I think that’s a great question. I do think it’s possible to shift too far left, and the reason is because, while we, I think, are all familiar with the statistics and the studies that say if you fix something earlier in the cycle, it’s cheaper—there’s stuff that exists to the right that doesn’t exist yet on the left. Software development is, you know, to sort of a little analogy that we talked about earlier, I think it’s a dance. I think it’s a dance between people. I think software comes about as a result of the behaviors and the actions and the decisions that various folks take together. And security and quality are emergent properties. They’re not Band-Aids, they’re not vitamins, they’re not painkillers. They have been, as a result of this dance. So, there are some things for which, if you try to test it too early, it doesn’t actually exist in its final form yet. If I take my 6-year-old to the doctor and then I don’t when she’s 21, you know, might we have caught something super important as a 6-year-old that we were able to step in and address right away that would’ve been great? Yes, but as a 21-year-old—guess what? She’s gonna have different problems that were not apparent at age six.

    Shimel: Spoken like a true mom and parent, right? Parents, we worry about that. Anyone else? Thoughts on that? Can we go too far left, or have we gone too far left?

    Dawson: I, Alan, think it just depends. I think we’ve interpreted left different ways. I think it’s a good, catchy term. It’s an important and powerful catchy term, but as I’ve heard others discuss, we sort of overemphasized this idea of left. One, to Caroline’s point, right, we just do everything here and we’re done, good to go, right? But then this idea that we can put everything in Dev? Well, no, I think—I think, for what it’s meant to communicate, shifting left is an apt term.

    I think in terms of ensuring that we’re doing everything we can as far left or as early in the process to mitigate issues later—great. Right? But we don’t just shift security left. Right? Like Caroline said, inbound, out of bound. Actually, we need to shift security everywhere. We don’t just shift quality left, we need to shift quality everywhere.

    So, I think there are—we need to be ensuring that it’s almost zero trust software delivery, right? At every stage of the process, from concept where you should be sitting down with QA and the security team to capture the requirements to test, to deploy, to run, we need to be thinking about it, also.

    Alan, I’d say shift left is good—shift everywhere is really what we meant to be doing, right?

    Shimel: It’s better. Great.

    Kalsey: Any time we engage in sloganeering, there’s gonna be people that pick the wrong things from the slogan. I mean, “Hey, we have DevOps. That means we don’t need Ops anymore!” And that’s not what it meant, and there’s people that have taken the wrong cues from shift left, and instead of shifting the processes left, they shifted the tasks onto the people who were left already, and that’s not gonna work.

    Shimel: Absolutely. Hey, Mitchell, you wanna tie a bow on this?

    Ashley: I’ll put a bow on this. You know, there are admittedly many, many things that I don’t know, but one of the things that I do know is that, if a developer says it’s tested, it’s ready and a QA person says it’s tested and it’s ready, five out of six dentists will choose the QA person. I will, too.

    So, there’s something there. There’s a reason why we don’t have DevQA or DevTest. We don’t talk about it that way, we talk about DevOps or DevSecOps. QA is extremely valid and necessary, and we can’t live without it, and I think we need to invest in helping, elevating how we test and how we think about our software applications to test in. So, those dentists will be happy.

    Shimel: And Crest, the Crest test for QA. Thanks very much, Mitchell. Caroline, Brian, Adam—thanks for joining us on this episode of DevOps Unbound. Great conversation. For our audience out there, do check it out. We have another episode coming up in—well, not next week, the week after, and then I think we have a roundtable the two weeks after that.

    But for now, though, we’re gonna wrap. Many thanks again to Tricentis for sponsoring DevOps Unbound. Thank you for watching, thanks to our panel. This is Alan Shimel, we’re out.

    [End of Audio]

  • Nolij Consulting Brings Expertise Beyond the Public Sector

    Nolij Consulting Brings Expertise Beyond the Public Sector

    Alan Shimel talks with Ashley Mehta, CEO and president of Nolij Consulting, about the company’s deep expertise in health care IT, digital health care transformation, testing and evaluation, and the use of emerging technologies to solve issues in the government IT space. Now, the company is going commercial, and Mehta explains how their expertise can benefit IT organizations beyond the public sector. The video is below, followed by a transcript of the conversation.

    Announcer: This is Digital Anarchist.

    Alan Shimel: Hey, everyone. Welcome to another session here on TechStrong TV. We’ve got a new company, a new guest, first-time guest to introduce you to today. I want to introduce you to Ashley Mehta. Ashley is a founder, CTO of a company called Nolij. It’s not spelled the way you would think though. I’m going to let her explain that. Hey, Ashley, welcome to TechStrong TV.

    Ashley Mehta: Thank you, Alan. Thanks for having me.

    Shimel: It’s a pleasure to have you on. So I know I already said it so I guess we’ve got to address that first. How do you spell the company’s name?

    Mehta: It’s N-O-L-I-J, it’s a play off of knowledge. You know, so I had some friends in the Silicon Valley and they said you have to kind of spell a little different, so that’s what we did when I started the business, yes.

    Shimel: Excellent. Well, I guess the question is let’s share with our audience, let’s assume they don’t know anything about Nolij. So why don’t we give them a quick background there and then we can talk a little bit about your background as well.

    Mehta: Yeah. No, that sounds great. So Nolij is a woman-owned small business. We started in about 2012. I had worked for some large consulting business to learn all the ins and outs of the business and management side, and slowly just kind of started building my woman-owned small business.

    Today we have around 150 employees and we are primarily a health IT company doing work within HHS and the Defense Health Agency. We also do work within Homeland Security within the GSA. So it’s been a lot of fun and we have an outstanding company.

    Shimel: Absolutely. Very cool. People think, oh, startups, overnight success. You’ve been at this now for about eight or nine years, and as have I here at MediaOps, working on eight years now. You know, you put everything you got into it, right? You said you worked for some big consulting companies, but give us a little bit more of your background if you don’t mind.

    Mehta: Yeah. So I’m originally from Ohio, went to Ohio State University, graduated in marketing.

    Shimel: OH.

    Mehta: Yeah, yeah. The Ohio State University. We have the Buckeye fans within our family. My dad has had his own business. My oldest brother’s in healthcare in mergers and acquisitions and doing very well. I’ve got a twin brother that’s also in technology and business. And then my dad was, like I said, had his own business, and cousins and uncles.

    So we have a big background in basically finance and entrepreneurs and business and I guess it’s just kind of a little bit in my blood.

    Shimel: Good for you.

    Mehta: Yeah. So it’s been a lot of fun.

    Shimel: Yep. So recently Nolij expanded into the commercial market.

    Mehta: Yeah. So actually before COVID hit we started kind of thinking about, hey, let’s start exploring into the commercial market, see what’s out there. We do a lot of work within, like I said, health IT. We thought, hey, let’s go and see the private hospitals. We do a little bit of work within the private hospitals in regards to the contract that we have with the Department of Defense, working with them to make sure the electronic healthcare records are integrated with our system, so if any of the military personnel have to go into the private hospitals that their electronic healthcare records are there.

    So we’re really just venturing out, getting our feet wet, figuring out if it’s going to be a viable market for us, and hired experts within the commercial markets. So we’re experimenting with the private hospitals and we’re also going to start to experiment with the payers as well, but we just have a very energetic workforce, great employees, a lot of technical talent, and just really trying to figure out how else we can grow. So of course started it before COVID hit and then COVID was interesting.

    Shimel: To say the least. [Laughs]

    Mehta: Yeah, yeah. But we focused a lot on marketing and kind of getting our name out there, so thanks a lot for this opportunity, Alan. It’s a pleasure and really, truly appreciate it. [Laughs]

    Shimel: Oh, no. It’s our pleasure. I mean, part of what we like to do is bring interesting companies, interesting service products to the audience so that they understand. So you’re doing this, let’s say, going on seven years, and you say, you know what, we’re going to – when did you decide to say, okay, we need to get into that commercial market?

    Mehta: Yeah. That’s a great question. So one of our largest projects is within the Department of Defense, the Defense Health Agency. So we actually have the pleasure of actually doing the IV&V for software testing for the electronic healthcare records on the Department of Defense Side. And Alan, I’m sure you’re probably familiar with that they’re trying to commercialize the product. It’s called MHS GENESIS, which is a Cerner commercial product, and Dentrix, which makes up MHS GENESIS, which is basically the Department of Defense’s electronic healthcare records that will flow digitally into the VA.

    So once you retire from the military and you become a veteran you don’t have to literally hand over your healthcare records to the VA and have them scan it in and upload and that kind of thing, that it’s just all electronically. So it’s just better care for the people that serve our country. And so we’ve been doing all this support, making sure that there’s no defects and that kind of thing within the software. I think it was just kind of a point where, hey, it’s a large project, we do an outstanding job, we know the ins and outs of interoperability and data and defects and how to mitigate a lot of the risk with them in technology and software, which is very complex because there are so many systems that are integrated to make up these electronic healthcare cords.

    So we were just thinking that, hey, let’s venture out into the commercial side, see if we can get some business there, provide opportunities for our employees to grow, for the company to grow, and I think it’s always great to get the expertise and get the guidance, and then you’ve got the resources to do the work, but just kind of see what else is out there. So that’s what we’ve been doing. Because you always have to build upon the capabilities that you have, and so the commercial market there’s a lot of opportunities to do that.

    Shimel: Absolutely. I want to bring up something, and I hope I don’t embarrass you, but obviously Nolij is a woman-owned or woman-founded run business. In the federal space we have, you want to call them, set-asides for minority, women-owned business, economically-disadvantaged community businesses, stuff like that. We don’t have that as much in the commercial space, frankly. So I wanted to ask you – and I realize you venturing forth into the commercial market is rather new for you, but how do you find it as a woman CEO? The differences between navigating what for you is the comfortable confines of the gov space versus kind of the wild west of commercial.

    Mehta: We’re not really focused on the fact that we are – I mean, we are woman-owned because I’m a woman, but we’re really focused on the customer and the technology that we have to offer. So we have to get out there, submit excellent proposals that have to do with emerging technology to help increase efficiencies and decrease risk for clients. Quite frankly, I think on the government side I think I’ve only submitted maybe one or two proposals that are set aside for woman-owned small business. A lot of them are small business and if a woman happens to own it then the government gets those brownie points and allocates those towards woman-owned small business.

    But I don’t see tons of opportunities set aside specifically for woman-owned small business. Maybe on subcontracting plans, if it’s a large set-aside, then they’ll say, okay, you have to give 5 percent to HUD zone and 5 percent to woman-owned and 8A and that kind of thing. But I don’t think that – at least I haven’t seen within the agencies that I have been doing a lot of work in, real strong set-asides for woman-owned small business where women are actually competing to win the work. I mean, a lot of times you see these companies that quite frankly are owned by men [laughs] that get their wives to be the woman –

    Shimel: Oh, I saw plenty of it. Yeah. One of the companies I founded was about 60 percent of our business was DOD back in –

    Mehta: I’m sorry, say that again, Alan. I didn’t hear that.

    Shimel: Sixty percent of our business was DOD. This is like in the mid 2005 to 2010 timeframe. And back then it was like native-owned business as a lot of set-asides. And I saw this firsthand and it got me really down on that whole line of doing things. It was like a good idea; try to give native-owned businesses business, right?

    But then what they were doing is you would have like this small native-owned business that had one person who as a Native American as the titular head/owner of the company, and then they were just kind of acting as a front for the usual suspects, right? [Laughs] The large integrators. And I realized it was a good idea but in practice it was abused and it’s not what the intent there was. It was wrong.

    Mehta: Yeah.

    Shimel: It is. So let me ask you a question. For people out here watching this now, when should they reach – what is the fact pattern, if you will, for when they say, hey, you know what, Nolij may be something that can help us here? Nolij is a company that might be able to help us.

    Mehta: What is the what? I didn’t hear that.

    Shimel: What is the fact pattern or what is – give us a typical engagement sort of scenario. Commercial companies out there saying, you know what, I’d like to reach out to these folks. They might be able to help us with this problem.

    Mehta: Yeah. So we have lots of technologists that are in the areas of quality assurance and software testing, artificial intelligence, RPA, which is robotic process automation. We’ve just recently partnered with probably about eight different emerging technologies, different various artificial intelligence platforms that either helps out with customer service or help out with operations, like a lot of bots. But I would say that the whole integration of systems, getting them to work, making sure there’s no defects, because at the end of the day it’s like the customer, they’re using these systems and they have to be happy, right? Especially when you’re dealing with people’s sensitive information in regards to their personal health information or their personal data, cybersecurity.

    We’re also kind of partnering with companies that have various cybersecurity technologies. So I think that we’re just kind of out there. And as a small business, you have to have strong partners to grow as well. Even large businesses have to have small businesses to grow because they have expertise in certain fields. So it’s really just about just building those strong partnerships, being able to go in, do the work with the clients, and just be successful in just making the customer successful. So I think there’s a lot of trust there that kind of goes into that when the customers give you an award.

    Shimel: Absolutely. Ashley, you know what I realized? We got into the whole how do you spell Nolij, but we never gave the website for people who may be wanting to get more information. What is the website and if you could spell it.

    Mehta: Sure. It’s obviously W-W-W dot N-O-L-I-J consulting dot com.

    Shimel: Very cool. Nolij Consulting with an N. I love it. Ashley, thanks for being our guest on TechStrong TV today. Great continued success. Best wishes. Come back and keep us posted.

    Mehta: Thanks, Alan. I definitely will do that. Thanks. I appreciate it.

    Shimel: Very cool.

    Mehta: Thanks.

    Shimel: Alright. Thank you. Ashley Mehta, CEO, cofounder of Nolij Consulting. That’s N-O-L-I-J, here on TechStrong TV. We’re going to take a break. We’ll be right back.

    [End of Audio]

  • EP 14: What Have We Learned?

    EP 14: What Have We Learned?

    The past year-plus has brought disruption and opportunity to businesses and software teams all over the world. Change is inevitable, but 2020 was a year of massive change: businesses pivoted, supply chains were disrupted, customer expectations shifted to contactless services, digital transformation projects accelerated, new digital experiences were created, the move to the cloud jumped into high gear and our ability to create and deliver software was elevated to an essential capability for business.

    It’s time to pause and ask ourselves, what did we learn from all of this disruption and change? What lessons can we carry forward? How do those experiences inform how we prepare for 2021 and beyond?

    Cornelia Davis from Weaveworks, Kurt Chase from Tricentis and Sanjeev Sharma from Truist join hosts Alan Shimel and Mitch Ashley in this episode of DevOps Unbound to discuss lessons learned, how to overcome the obstacles we’ve faced over the past year and how to improve business processes and make better decisions. The video is below, followed by a transcript of the conversation.

    DevOps Unbound is sponsored by Tricentis, the world wide leader in automated testing. DevOps Unbound airs biweekly on TechStrong TV, with all episodes available on Digital Anarchist. There is also a live monthly roundtable open to audience participation.

    Alan Shimel: Hey, everyone, this is Alan Shimel, and you’re watching DevOps Unbound. For those of you watching for the first time, DevOps Unbound is a biweekly video series where we tackle interesting topics related to DevOps. Once every four to six weeks, we do a live roundtable with a live audience questions and live audience participation, and we invite you to our next one. You can find that scheduling at DevOpsUnbound.com.

    Today, though, is a session just with our panel, and what a panel it is. Let me introduce you to them. First off, let me introduce our friend, Sanjeev Sharma of Truist. Sanjeev, maybe you wanna introduce yourself to the audience.

    Sanjeev Sharma: Sure. Thanks, Alan. First of all, thanks. Great to see you all. Hopefully, we will be able to meet in person soon, but thanks for having me on. I am the Head of Automation and Platform Engineering at Truist Bank. Truist, for those of you who might not know, is the new bank that was created last year by the merger of BB&T and SunTrust. I have various hats I wear at Truist, but my background is mostly in the DevOps and cloudware adoption space.

    Shimel: Excellent. Thank you, Sanjeev, and welcome. Sanjeev, I should mention, is also the author of The DevOps Adoption Playbook.

    Next, we’re really, really pleased and honored to have Cornelia Davis with us. Cornelia is, of course, a pretty well-known thought leader and leader at the DevOps space, but is also at Weaveworks. Cornelia, welcome. If you wanna give a little background?

    Cornelia Davis: Sure, yeah. Thank you so much, Alan. It’s so great to be on again. Yeah, I am the CTO at Weaveworks, and you might know Weaveworks as the company that coined the term GitOps, that has been kind of driving that movement. And GitOps is just an element of a broader category, which is really developer platforms, which is what we’re talking about many, many times when we talk about DevOps.

    And so, prior to Weaveworks, I’ve been here for about a year and a half, I was at Pivotal, where I worked on Cloud Foundry and Kubernetes based developer platforms, you know, platform teams for enterprises for quite some time. And I’m also the author of a book called Cloud Native Patterns, which is a book aimed at architects and software developers, helping them learn how to build software for this highly changing, highly distributed—constantly changing, highly distributed environment that is the cloud.

    Shimel: Absolutely, thank you. It’s a pleasure to have you here. Last but not least on our panel today is our friend Kurt Chase with Tricentis. I should mention Tricentis is a sponsor of DevOps Unbound, and thank you, Tricentis, for sponsoring. But Kurt, why don’t you give them your background?

    Chase: Yeah, thanks, Alan. Great to be here today with Cornelia and Sanjeev. Nice to meet both of you. I’m currently running Release Management and Engineering Services at Tricentis, which, Engineering Services is a group that provides cross cutting services primarily for internal engineering, but effectively for the whole corporation as well.

    Prior to Tricentis, I spent years at Splunk, again, supporting the engineering function, all the CI/CD builds and releases and things like that. And then prior to that, I spent almost two decades at Autodesk, producing the AutoCAD, multiple releases of the AutoCAD software product.

    So, again, it’s great to be here today with all of you. Excited to talk about some of the things we’ve learned over the last year, year and a half—and what a year it’s been, holy cow. [Laughter]

    Shimel: Yes, it has. Then let me finally introduce my co-host here at DevOps Unbound, my good friend and partner, Mitchel Ashley. Mitch, why don’t you give a little background?

    Mitch Ashley: Thanks, Alan, and of course, I’m excited to have our panel—thank you for joining us, all of you, today.

    I’m Mitch Ashley, CEO of Accelerated Strategies, an analyst firm that focuses on DevOps and cloud native, cyber security, digital transformation. I also work with Alan as part of MediaOps, the folks that put on this event, with the CTO of that organization. So, it’s a lot of DevOps practicing and observing what others are doing, so I’m excited to hear what Sanjeev and Cornelia and Kurt have to say, too.

    Shimel: Alright. Thank you, Mitchell. And as I mentioned, DevOps Unbound is sponsored by Tricentis, so—many thanks to them.

    Today’s episode, folks, as I mentioned off camera, I don’t wanna jinx anyone or anything, but at least here in the U.S., we seem to, at least we hope that we’re seeing this whole COVID life sort of starting to recede in our rearview mirror. We are returning to normal or a new normal, some may say recklessly too quickly, or some may say not quickly enough, but certainly, the return to normal is on, and accelerating.

    But, you know, there have been many, many lessons, many, many experiences, some painful—extremely painful—from a business and a humanity point of view over this last 14, 15 months or whatever it’s been. There’s also been a tremendous amount of success, a tremendous amount of acceleration, a tremendous amount of, “Jeez, this really—this worked. We didn’t know if it would work, but it worked,” probably the Internet being the first thing, right? Where would we be with this without an Internet?

    But can we—let’s take a moment, and let’s, as I say, hope and pray that this is in our rearview mirror or starting to move to the rearview mirror. What are the lessons? What can we learn, what will we take forward from this? And, you know, I don’t wanna boil the ocean, because I think there’s a lot, but if I asked each of you to give me three things that you really, you think—not just you personally or your organization, but the tech industry has learned, the business world has learned, perhaps society in general, humanity has learned, what would you say some of those things are?

    I don’t wanna put anyone on the spot, but Kurt, you always have something to say. I introduced you last, we’ll start with you.

    Chase: Sure. I’ll jump right in, and one thing that’s amazing to me, not so much from the technology or engineering or R&D side, let’s talk about business, here—the other aspects of the business that were able to adapt to remote work, that, to me, was pretty incredible.

    When COVID really took hold, I wasn’t too worried about software engineers and folks producing software working from home, doing their coding, testing, and all that. The other functions—finance, legal, things that are required to ship your products—seeing those go remote at the pace they did, that was pretty incredible. I think they definitely took advantage of some of the learnings from their individual engineering teams. But to me, that’s one thing that sticks out right away is the speed at which businesses adapted to working remotely and the success they achieved across the functions in the business. That was incredible to me.

    Shimel: Agreed. I mean, the speed of adoption was something. Cornelia, did you wanna say something?

    Davis: Yeah, sorry, I was muted there. I did wanna chime in, because I agree, and one of the ones that really knocked my socks off was how quickly call centers went remote.

    Chase: Yeah. [Laughter]

    Davis: Because that isn’t something I would’ve expected. I would’ve expected them to still be kind of really tied into their telephony services and, you know, the phones that are on their desks.

    But I spoke with many, many people, you know, various customer service people from the banks and from, you know, the Amazon.com—well, there’s not that much of a call in center there, but from all of these different organizations where, when you chatted with these folks, they were like, “Yep.” Within a week, they were able to move everything, move everybody remote in these traditionally really physical call center based systems. Which means that there had to be some preparation. I don’t know if it was intentional, but there had to be something that allowed them to flip that switch so quickly. Is it just forwarding phones? Is it just VPNs? I don’t know.

    Shimel: No, no, it was—so, I have some info on that, if I can. I interviewed a fellow down here in Florida, I think he was out of Tampa, and he operated multiple call centers, like in Costa Rica, which is actually a call center capital, believe it or not, that’s a big business down there. It turned out that the software they used, right, people in call centers, they don’t just—Kurt just doesn’t call someone randomly. Everyone gets assigned their calls or their inbound when a call comes in and gets assigned.

    The software they used within the call center itself worked just as well, the person didn’t have to be in the desk next to you, he could’ve been anywhere, it turns out, or she could’ve been anywhere. And so, it worked just as well in a remote thing. So, they still used the software they were using that tracked how many calls they did, who they spoke to, the notes, putting it into the central database. Yes, they had to be cloud-based, right, because there was no one in the call center to kinda run that infrastructure. But the software, it turned out, worked remotely as well as it worked on everyone’s desk in the call center. It was an amazing thing.

    And so, what they found—again, and this is gonna be, I think, something that’s gonna be a recurring theme—who needs the physical call center? They don’t need that infrastructure anymore. They could have people working at home just as well.

    And so, the more advanced ones—and you’ve probably seen this yourselves—you call someone, they may be very well at home, but they’re giving you the same mediocre service that you get when you call the call center itself, right? [Laughter] But yeah, no—

    Chase: And that’s the point. I think with a business, it’s applicable across functions there, it’s amazing. And business learned that they can still be extremely successful, even if everyone’s remote. And having that pool of remote talent—that changes everything. [Cross talk]

    Sharma: I think a couple of things to be noted there, right? First of all—sorry, Cornelia, I think I spoke over you. You know, one is, the pendulum also needs to swing back from that extreme.

    Chase: That’s the challenge, yeah. [Laughter]

    Sharma: Right? Because, while we have people who are able to work from home because the technology is there, from an equity perspective, one has to look at people who cannot work from home who are being forced to.

    Chase: Yeah.

    Sharma: And we experienced that when we opened up our offices. We are just beginning to open up our offices. June 1st was the first day folks would come back. And I was surprised to see some people had jumped at the opportunity and said, “I want to go back.”

    Chase: Yeah.

    Sharma: Then when you look under the hood and you talk to them, you realize, you know, they’ve been struggling. They’ve been operating out of a tiny closet, right, because they don’t have the space or because they have toddlers or they’ve been struggling with Internet issues, and it is worse overseas than it is here in the U.S.

    So, I think the pendulum needs to swing back to a more normal position, but hopefully, it won’t swing all the way back to where they say everybody has to be on site, on premises, right? So, I think that is definitely, some balance has to happen to recognize—some people do need a workplace, where they can work comfortably.

    Chase: You’re absolutely right.

    Davis: Yeah.

    Chase: I know a story of one individual who was in a one bedroom apartment in San Francisco and after three weeks, he couldn’t handle it. It’s like—I don’t have no space.

    Davis: Yeah.

    Shimel: It happens. Cornelia?

    Davis: And, you know, honestly, I have space, and I have a wonderful office and I’m super comfortable, and I’m also desperate to get back into the office.

    Chase: Yeah.

    Davis: Because—and I love, Sanjeev, that you called out the equity thing, because that’s something that is so painfully obvious in the whole pandemic is that this has only broadened the inequities that we have across society, and we need to absolutely focus on that.

    But then the other lesson, you know, with me, I’m extremely privileged, I live this super comfortable, I never really had to worry about catching COVID, I could just choose to stay home. But humans need humans and, you know, we’re all coming to you on your two-dimensional screen. I am super fortunate that last week, I had the opportunity to have my first face to face with a couple of colleagues in over a year, and it was rejuvenating. I’ve been saying, it’s a 3D work world again instead of a 2D work world.

    Shimel: Yeah.

    Davis: And so, I mean, humans need humans. But you’re right, Sanjeev, I think that the even bigger thing is that we really, really need to look at the inequities. And in the beginning, in the early parts of the pandemic, without getting too political here, we had all of this narrative around these people who we considered essential workers who were kind of the forgotten, like, taken for granted folks, and they were put on a pedestal, and that, unfortunately, didn’t last that long.

    Chase: Yep.

    Sharma: Yeah.

    Davis: And so, we need to bring that back.

    Shimel: And they weren’t paid like they were that essential, either, but that’s [Cross talk]—yeah, it was a pretty low pedestal. But let me, you know what, you bring two things—I’m sorry, Mitch, I just wanna make two quick points.

    Number one, when we talk about the inequality, let’s be really clear. This has affected women and mothers especially, more than men, right?

    Ashley: Mm-hmm.

    Davis: Absolutely.

    Shimel: Mothers have had—because not only have they had to work from home, but they’ve had to take care of the kids who are not in school. And it’s very hard being a teacher, a home care provider, and doing your work at work. And so, how many women have had to work out of the workforce because they had to take care of their kids and their kids are home and they didn’t have an option, there was no child care for them. So, I think—

    Ashley: And the challenges of re-entering the workforce.

    Shimel: Right, and now they’ve gotta re-enter after, you know, months off, and that’s never easy. People wanna know, “Well, why the gap in your résumé?” Well, there’s a gap in my résumé because I have this beautiful child that I’m raising, you know, and how dare you? That’s number one.

    Number two, it is a case of the rich get richer, right? Companies that, not only individuals as you say, Cornelia, but companies that were able to leverage this. Man, they accelerated, but they accelerated at the expense of their competitors who couldn’t, right? Think about all the open retail shops in the malls near you and the strips, right?

    Davis: And this is a place where I’d love to pull DevOps into the conversation, because we are, of course, as a part of DevOps Unbound. One of the things that I thought was impressive—and I’ve worked with some of these companies. In, you know, my previous life at Pivotal, I worked with The Home Depot, for example. When you had The Home Depot and you had these other organizations that, in a matter of weeks, updated their software so that you could do things like curbside pickup? That right there is DevOps in action.

    Shimel: Yep.

    Davis: It is the ability to release software, get feedback, release software, and there was no six-week testing cycle that we used to do, or eight-week testing cycle.

    Shimel: Mm-hmm.

    Davis: It was like, “You know what? We are gonna be testing in production, because we didn’t have time to test this for six weeks.” So, it brings in all of these tools that Kurt, I’m sure, you have day to day experience with, which is these things that allow us to put safety nets in place to be able to release all the time. I mean, that is something where the people who were along their DevOps journey already had a leg up on those that didn’t.

    Ashley: You know, Cornelia, I had that exact experience where I couldn’t get a computer, you couldn’t get one shipped to you, they just weren’t there, so I had to build a computer for some video work, and did it solely from parts from Best Buy, by curbside pickup, two Best Buys in town.

    Davis: Yeah.

    Ashley: So, it literally was, like, a week later, all of a sudden, I could do this, and I was pretty shocked by it.

    One of the things I wanna say, though, about what we’re discussing is, the people dimension of it is extremely important. I think there’s a whole other set of lessons for the business. I think we’ve pressed reset on a lot of assumptions about how businesses have to operate, that you have to be in an office. And essentially, we’ve kinda globalized the business by just taking the location out of it now. And yes, there are privileged and underserved and people who aren’t privileged. And those inequities probably are gonna get bigger, not smaller, because I don’t think we’re gonna all slap back into the office immediately.

    But what the businesses learned is that lesson you just talked about with Home Depot and you with Best Buy is—you know what? We can decide we’re gonna pick Team or Slack in two days instead of two years. We can get a software release out in a few days if we have to. Talk about constraint based management—you know, we now have this constraint and everything else comes off.

    So, I think what it’s done overall, though, is for DevOps and for all of us is, it’s moved software from maybe second or third to number one. Our ability to get software out the door that’s effective, meets what we need it to do, but that is a strategic advantage, maybe even a survival advantage of what we’ve seen happen in this past year.

    Chase: Yeah. I think some of it is, you know, businesses like that were forced to do it or die, basically, you know? And I 100 percent agree with Cornelia that I think they look to DevOps and some of the practices that they’re all used to. You know, the collaboration we need across functions to move forward and all those things are key, some of the foundations of automation, you know? That has been key in the DevOps space, and I think they learned, they need to take advantage of automation to do these things as well.

    So, yeah, it’s been fascinating to see that. And that was a point I brought up to see how businesses transformed and our world changed. And I think it’s opened up endless opportunities for us moving forward. And to Sanjeev’s point, I hope when that pendulum swings back, corporations take a look, that this is no longer a corporate decision where people work, it has to be more joint. It has to be more personal, because every situation’s different, and if you wanna maximize all your programs in HR, diversity and all those things, you have to take that into account. I don’t think you can do a one size fits all any longer.

    I still wonder if all those big office buildings, I worked in downtown San Francisco for many, many years—are we gonna fill those up again, ever? I really wonder now if that’ll ever happen. I could see it not happening, quite honestly.

    Ashley: I’m curious, Sanjeev, you’ve been in the middle of this bringing two companies together with this new bank in the middle of all of this. What in God’s name was that like? [Laughter] What are the additional challenges you’ve had for that, or maybe what opportunities has it created?

    Sharma: Yeah, so, I think it—you know, I joined in the middle of the merger, right, so we were in the middle of it when I joined. So, what I can tell you in the 10 months I’ve been there, we have shifted a lot of the priorities because of the pandemic, right? A lot of things which we would have done later on, we did first. We were able to do things.

    But the flexibility has definitely opened up. I can tell you, between the time I joined and between now, the ability to hire people who are not local to one of our offices has gone through the roof. It used to be an exception, now it’s the norm, right? The last three people I hired, two are in the West Coast and one is in Minneapolis, right, so Midwest. It doesn’t matter anymore, right? The same way there are—the operation has opened up a lot.

    And when I look at it broadly, outside of who I’m working with, when I’m talking to other companies where I have friends or colleagues, they are seeing more offshoring happen, right? Because they’re going, “Why do I need to hire somebody in L.A. if they’re never going to see the person ever anyway, does it matter whether he’s in Bangor or Bangalore, right? Let’s put the person, let’s hire the right person at the right place.” So, I think a lot of that has opened up. It has, of course, shifted some of the salaries, also.

    So, you know, a lot of things are going on, interesting dynamics are happening, and we’ll see how that all pans out. But from a perspective of, you know, doing the transformation, it is hard when you can’t have people in the same room on a white board and things which could have just drawn on a white board and knocked out in a 1 hour or 90 minute working session, now it’s like 15, you know, Zoom views, WebEx or Teams calls, which could have been, and it’s like, “Where is that white board we made up? Was it on Miro or was it on Mural or was it on this or was it on that? You know, and who can edit it, who can’t edit it?” All those—you know, dang it, if we just had a white board and all of us in the same room, we could have knocked this out in half a day.

    So, it is difficult to do certain things remotely. I think there has to be a balance, which needs to happen. And yeah, during a merger just made it more complex, because we are going through change every day, right? People’s networks are changing, people’s—how we are doing things are changing, we are updating stuff. It’s been a very interesting few months and we’re not done yet. You know, the merger isn’t over. We’ve still got a long way to go. So, it’ll be more fun going forward, I think.

    Shimel: I think it brings up the issue, though, of how much of this sticks, right? Because, I mean, look what Sanjeev just described. We’ve all had similar experiences, I’m sure, hiring people from all over the world—what difference does it make? They don’t have to come in the office.

    Does that stick? Because if it sticks, there are profound implications for society in general. Why live in San Francisco unless you love the view of the Bay. It’s a beautiful bay, right, the bridge is nice, but pay $4,000.00 for a closet-sized one bedroom apartment? Maybe not. I mean, on the other hand, I live in South Florida. Housing market is insane because of people moving here from New York and California and spending a fortune—Colorado, right?

    So, does this—that’s one example. Does the ability to work from anywhere, it makes no difference, is that something that sticks around, or do we, as Sanjeev said in the beginning, see a pendulum start to swing back? What do you think?

    Chase: Well, for me, like I said, I think elements of it have to exist and remain. I think there needs to be a balance, as Sanjeev said. I no longer think it’ll entirely be a corporate decision where employees are located. I think for some roles and groups in organizations, there needs to be a deeper conversation on where is the best talent, what’s the best way to use them? You know, having people sit—just the commute. The one thing I don’t miss is the commute. Oh, my God, you know, those three hours a day commuting, you know, that, to me, we have to somehow retain that benefit, you know, from working remotely somehow. I’m not sure how.

    But that’s a very good question, Alan, I don’t know. It’s like, will those office buildings ever be full again? I don’t know. I’m not sure.

    Sharma: On that point, I’ve been remote since 2013. In 2013, I was at IBM. I moved to a global role, and since then, I haven’t had an office. I’ve worked out of this room since then. But I used to travel 150 to 175,000 miles a year.

    Chase: Got it, yeah.

    Sharma: Right? So, for me, the office was a place to go collaborate, whether it was my office or my company, IBM—

    Shimel: And for all of us.

    Sharma: – right, or a client’s office, it was not a place to go and have a desk, right? My desk is here, right, my books are here—you know what I mean, right?

    Chase: Yeah.

    Sharma: I think we should, I would like to see us go to that model where offices become more of a collaboration space rather than a place you have to show up and work alone in a cubicle. What’s the point of that?

    Ashley: I agree.

    Sharma: Unless somebody needs it, right? That’s different if they don’t have space at home and the need a cubicle to work in—fine, put them in a WeWork next to where they live, right, close to where they live rather than have them commute to the other, you know, 45 minutes one way.

    I think that’s the model we should head towards.

    Chase: I agree.

    Sharma: And, you know, the tools to enable that are there, we’ve proved it. We’ve proved, during the lock down, during the pandemic that the tools are there, but I hope it goes that way. I hope it doesn’t go back to, “You have to show up in this cubicle” even if you are, you know, like the guy from Office Space protecting the stapler and never talking to anybody.

    Ashley: The butts in seats model, right? [Laughter]

    Chase: We’ve seen some large organizations [Cross talk] make those calls already and say you can be remote forever now, you know, things like that. So, I think that’ll become a bit more selective, but I still hope, when that pendulum does swing back, that the employee has a say, you know, and his voice is a little bit taken more seriously.

    Shimel: Or her voice.

    Davis: Or hers.

    Chase: Or hers—or hers, [Cross talk].

    Davis: Exactly, that’s where I’d like to chime—I’d like to chime in there, because Alan teed this up earlier and then, because you had two things, Alan, we followed thread number two.

    I wanna come back to thread number one, which is that this has affected women in the workforce, and it’s not just us technologists, but it’s a broad spectrum, and it’s probably even more so in other working arrangements. And so, I love what you’re saying there, Kurt, about it being a partnership between the corporation and the employee and determining what’s best for them. And then Sanjeev, you, I think, just teed up—is that the survival answer for WeWork is that part of that negotiation between the corporation and the employee, she or he, includes having the ability to move into a workplace that is close to your office but allows you that space that you need, for example, not to be on call for child care.

    Of course, the other thing I can’t help but say—and I’m very fortunate, I have one son, he’s 26, I’m not tying shoes or going to PTSA meetings anymore—

    Shimel: I hope not. [Laughter]

    Davis: – so, I wasn’t affected in that way. Yeah, [Laughter] I wasn’t affected in that way in the last year, but I was a working mother and I worked from home that whole time. And I just wanna remind everyone, men and women, that you’re right, women take on that burden, but it doesn’t have to be that way, fathers.

    Shimel: Amen.

    Chase: Mm-hmm.

    Cornelia Davis: So, please pitch in even more than you think, just because it’s been kind of the standard operating procedure. If nothing else, I hope the last year has maybe, in some cases, resulted in a, hey, a need-based way of rebalancing responsibilities. That is something that every single individual, male or female, can participate in kind of these societal expectations and challenging those.

    Ashley: I think something happens—

    Shimel: You know what? Oh, go ahead, Mitch, I’m sorry.

    Mitch Ashley: When things open up, when opportunity happens and things make that shift, it’s different when there’s opportunity there, but when things have shifted and made a shift, they don’t snap back to the old way automatically. It transforms sort of the landscape going forward, and there’s so many things about this—and you mentioned women in the workplace, we also have people of color—and we think about the emphasis of diversity that’s happened over the last year and a half, which is fantastic. And we see so much more bringing to the forefront of women in leadership positions, people of color being involved in security more, whatever it might be. You know, I don’t think that that changes.

    So, hopefully, those people who now are getting to participate in a bigger, broader way or have more opportunity to do that, (a) I don’t think this is gonna go away, (b) that’s gonna change things for us, right? You know, maybe, Cornelia—[Laughter] here, I’m speaking about what I don’t know about—but maybe more women in leadership positions make some of those, it’s assumed that the woman at home does the child care or whatever is part of our societal transformation, that it isn’t necessarily an assumption. Hopefully we can get past that and remove that barrier for people so there is more opportunity.

    Shimel: So, Mitch, on that front, though, here’s the deal. Number one, I think the move to diversity is not necessarily COVID related, I think it’s just an idea that its time had come, it has been a long time coming, number one.

    Ashley: I totally agree.

    Shimel: Number two, I think one of the positive things that happened is, it wasn’t just the woman stuck at home taking care of the kid. She was stuck at home taking care of the kid while the husband was trying to carve out a desk space in his bedroom somewhere to try to do a meeting or something, right?

    But what often happened, at least among my friends—look, my kids, also, Cornelia, they went away to college even though their college was online, just to get out of the house from their mother and father who wouldn’t let them breathe. But most—like, people I know whose kids were younger, the mother and father were home. They had to split up, delegate duties. You did the wash, I cook the meals, you clean the table, I took the kids here, you took the kids there or entertained them now because you had work to do, I had work to do.

    In a best case scenario, it became very much a shared responsibility. Of course, not every scenario or every instance has that best case outcome. There’s a lot of people [Cross talk], you know?

    Ashley: I think it’s been a journey to share. It may not be quite as, like, “Okay,” [Cross talk].

    Davis: Yeah, yeah.

    Shimel: Well, that goes, this is—

    Davis: I mean, you’ve painted a fairly ideal picture.

    Shimel: That is an ideal picture, you’re right.

    Davis: Yeah. You know, this conversation is making me wonder, and I don’t have these stats, but I wonder if anybody else has heard the stats—you’re right, absolutely, that this shift towards more underrepresented folks in leadership positions, women, people of color, and so on, that absolutely started before COVID.

    I wonder how COVID has affected that. Have the numbers accelerated, have they decelerated? I think—I suspect the latter, and I’m pretty much an optimist, but I’m also a bit of a realist, and I wonder whether the last year has slowed down progress when it comes to diversity at the leadership level. I don’t know. It’s just a gut feeling. I’d love to see some metrics on that.

    Sharma: I can tell you, Cornelia, in IT roles, it has always been very difficult to hire people from minorities, right, underrepresented minorities.

    Davis: Yeah, of course.

    Sharma: It always has been. I personally believe it has gotten worse, because if you look at especially leadership roles, senior roles—we’re not talking fresh college grads—most of those roles, no matter what anybody says, happen through referrals and connections, through your network.

    Davis: That’s right.

    Sharma: And that network stalled. That network stalled during the pandemic, right?

    Shimel: Yeah, networking, yeah.

    Sharma: Right? It is as big as it was then or grew slightly, right, that networking opportunities were gone.

    Davis: Yep.

    Sharma: No Meetups, no conferences, no lunches, no power breakfasts, no women in technology meet ups—nothing. The network stalled—you only knew the people you knew, right?

    Davis: Yep.

    Sharma: And I think that hurt, bad. I find when I’m trying to hire, when my peers are trying to hire, we are struggling more to have—see, you can’t hire when you don’t receive the résumés. When there are no résumés coming in, how do you hire, right? And I’m struggling, and if anybody has, if either one of you could—any one of you has any suggestions on how to fix that, I’m all in. Please, I’m all ears.

    Davis: I think—

    Chase: Yeah, I was gonna say, you pointed out an anti-pattern to what I was gonna say, Sanjeev. I was gonna say, at least with COVID and all the remote work, our talent pool has opened up worldwide, but you’re saying that hasn’t helped in that regard.

    It has for me with some of the roles I’m hiring. Now granted, they’re not Director, Senior Director, VP level roles yet. But for the other roles, having the world as my talent, you know, the talent pool versus the United States, that makes it [Cross talk].

    Sharma: Well, I look at it a bit differently. I look at it a bit differently, Kurt. You know, hiring—how do I, I wanna make sure I say this properly. Hiring an Indian in India is not hiring a minority.

    Chase: Yeah, true. Yeah, true.

    Sharma: Hiring an Egyptian in Egypt is not hiring a minority. Hiring a Turkish person in Turkey is not a minority, right?

    Shimel: Right.

    Sharma: They have to be a minority in where you’re hiring them, that’s the only way it works.

    Shimel: Agreed.

    Davis: Yeah, I think, [Cross talk] I wanted to comment on your remarks, Sanjeev. I think that you’re spot on. Because there were opportunities, and some of the techniques that we have used in corporations to source diversity candidates or underrepresented candidates, those things went away. So, being able to go to a historically black university or go have your recruiter show up at Women Who Code meet ups, those went away, and those aren’t the same.

    Sure, Women Who Code still had meet ups line, but it’s not the same as walking around during the social hour and saying—or, I’ve been at many meet ups where, before they start the program, they say, “Okay, who’s hiring?” And everybody stands up and then, “Who’s looking?” Everybody stands up. “Alright, you folks, talk.” That was a very organic thing that was not just so easy to say, “How do you do that in an online setting?”

    You’re right, it’s—I hadn’t even thought of that, but that whole network is the same now as it was a year ago is remarkable, especially when it comes to leadership.

    Sharma: Exactly. That’s why hiring from a historically black university, we can still do that today, because the recruiters still know the counselors there, they still know the placement office or whoever does—you know, those networks are still there. It’s just that the network hasn’t grown. The higher you go, right, the air gets, you know, rarer, right, and those jobs are fewer and you are not screening 100 résumés to do that. You’re relying on the network more than often, no matter what anybody else says, right?

    And that network stalled. I mean, who have we met at a conference who we didn’t know before? Now, when I go to a meet up and I see somebody I know—

    Davis: Yeah, but I think what you’re pointing out—

    Sanjeev Sharma: – I am IM-ing that person, not some stranger who I’ve never met, right? Unless there’s something really, some reason to do that. Then—no, no, it is the opposite, it has really hurt. I think we need to fix this.

    Davis: Yeah.

    Sharma: We need to put this [Cross talk] the moment we come back.

    Davis: Yeah, but I wanna suggest that—yeah, but I wanna challenge that assumption, which is that I hire leadership from within my network. Maybe that right there is the thing that we need to work on.

    Sharma: Well, I’m with you. We need to work on it. What I’m saying is—that’s what I was saying. I was saying that what we tend to go, tend to—we tend to grow our network to find leadership role people, right? People in leadership.

    Davis: Yeah.

    Sharma: When you can’t grow your network, you’re reliant on what’s already in your network, right? I mean, otherwise, you’re scouring LinkedIn or some other website or pinging people in your network asking for referrals.

    Davis: Yeah.

    Sharma: But that network stalled. That network didn’t grow. I can’t point to 50 people I met in the last year who were cool and interesting who I did not know before who were underrepresented minorities have I would have met, had I gone to Reinvent or IBM Think or, you know, DevOps World or KubeCon or whatever. All the people I would have met, I didn’t meet them.

    Shimel: So, that’s a whole other thing. Virtual events, right?

    Ashley: Mm-hmm.

    Shimel: We’ve all spent significant portions of our lives at conferences. Cornelia, that’s where I usually see you is at conferences—Sanjeev, as well. Virtual events are cool. We put on one a month here at MediaOps. Gene Kim does his DevOps Enterprise Summit virtually, people even paid money to attend it.

    And all of the virtual events, they try to do a social thing to it. It’s just not very good, frankly. It just—it’s just not the same as being in the hallway or grabbing a drink at the bar or a soda or whatever, you don’t have to be drinking. But, you know, just not—it’s not that, someone said it before, face to face, person to person. I think it was Cornelia, you recently had an in person. It’s not the same, it’s not the same.

    I mean, so, different than the three of you, Mitchell and I have had a chance to meet hundreds of new people this year. I do three to four to five of these interviews a day. My day starts with talking to people in Asia or Israel and goes to Europe and through the Americas, all the way to the West Coast. And what I’ve found is that I’ve become a resource for people like you, Sanjeev, who say, “Hey, I’m looking for someone to head up cloud or DevOps or testing” or what have you. And I go through, who have I interviewed recently?

    But that’s not the way it’s supposed to be. And the question is—and we’re getting near on time—I wanna bring it back to DevOps and software, right? And, you know, what—

    Chase: I think you touched upon it from the beginning, Alan, it’s the Internet. You know, the fact that our technology infrastructure was able to shoulder the burden, that was pretty incredible to me, too. You know, I expected Zoom to go out for weeks, if you will. But I think it demonstrated that the technology infrastructure we have is pretty solid. You know, it allowed us to scale up pretty quickly, pretty massively and, you know, there were outages here and there, but I would’ve expected a lot more.

    Shimel: Minimal—minimal. Even counting all the DDoS attacks and the ransomware and all of that nonsense, it was still minimal.

    So, we had that infrastructure, but we also had this incredible capability to accelerate, to automate that DevOps kind of has been teaching us for seven, eight, nine years or more now, right? And that really helped. Curbside pickup and the—it’s not just curbside pickup, it’s the back end, obviously, right, that allows you to pull into that parking space and hit the button and say, “I’m here, and this is the car I’m in,” right? That kinda stuff, how many companies and jobs were saved by that kind of capability.

    Chase: The other thing for me internally is, I think it also opened up the ability to question everything, just because—at all levels, you know? Whether you’re designing a new feature, adding something to a product, designing a new process—is that the right thing, just because we’ve done it like that? I mean, really question it, because I think it, I mentioned this before. I think the opportunities ________ are endless now, really.

    We haven’t even started to take advantage of the things we’ve learned. I believe we’re starting to now, as Sanjeev mentioned, the pendulum coming back. Part of that is some massive learnings that we’ve also had forced upon us, if you will. But I think we’re gonna come through it even stronger and better, for sure.

    Davis: Yeah.

    Sharma: I think the—

    Davis: You know, I wanna, I’d love to chime in on—

    Shimel: Go ahead, Cornelia.

    Davis: – yeah, I’d love to chime in on that scale. You used the word scale and these systems were able to scale and handle just a tremendous surge in traffic.

    Shimel: Oh, yeah.

    Davis: And when you said that, it took me immediately back to a DevOps Enterprise Summit where Nicole Forsgren and Jez Humble—and those of you who don’t know them, you should know them.

    Chase: Yeah, we know them.

    Davis: State of the DevOps Report, State of the DevOps Report, the book Accelerate. I remember this moment where they were on stage, and Jez Humble, in his very humble way, hand in his pockets, says, “Turns out, architecture matters.”

    Shimel: Yeah. Right.

    Davis: And so, it’s having these systems that are designed to scale, and we find so many enterprises that have said, “Oh, well, you know what? Google scale, Amazon scale, that’s not a problem that I need to face.” Well, surprise—

    Chase: Yeah, it is.

    Shimel: It is.

    Davis: – maybe it is. And so, architecture matters, and so I can’t help, as a software architect, coming back to that and saying, “Yep, there ya go.” [Laughter]

    Shimel: Yep, [Cross talk].

    Sharma: And I think we’ve seen companies who were—there’s two ways to look at it. One way is to look at it from a scale perspective. The other way is to look at it from a resilience and reliability perspective.

    Davis: Right, yeah.

    Sharma: Right? Which are two aspects of the same thing. It’s all architecture. And I think companies realize very early that reliability and resilience are critical—resilience to be able to make changes rapidly, right? Today, I have, I don’t have curbside pickup, tomorrow I have. Today, I had 10 people doing this, tomorrow I’m gonna have 200 people doing this.

    Ashley: Right.

    Sharma: Today, I had a teller sitting in front of her terminal, so the user interface didn’t need to be that cool. You know, the things could hang and the teller could, you know, make sense, make small talk while the system is catching up. Well, nobody’s accepting that on the mobile app when they’re directly dealing with the website or the mobile.

    So, I think that whole shift on resilience and reliability are real, are important, and all that goes back to your point, Cornelia, to architecture. That was front and center to people, and I think [Cross talk]—

    Chase: Yeah. And I think along with that, Sanjeev, is security. I think security, this really exposed—we have to do more in security. We have to do more, you know?

    Ashley: We have to rethink security.

    Chase: I think that’s one of the chief learnings.

    Shimel: No doubt about that. Guys, I hate to do this to you, but I need to pull the plug on this. We already have people in my waiting room for our next interview.

    What a great panel this is. We’re gonna have to rerun this again and go to a part two, because there’s things I wanna talk to about this as well. I spoke to a guy today who started a new, he calls it the mobile cloud for gaming, it’s called Now.gg. Because their research is that our kids, you know, these digital natives, if they don’t get a response in half a second, they’re gone, they’re onto the next thing. And so, what does that mean for your banking apps? What does it mean for we’re doing?

    But we need to end this. Sanjeev, Cornelia, Kurt—man, thank you, guys, so much. Thank you, all, so much for making this an amazing panel. Thank all of our viewers out here for watching this; we hope you enjoyed it. Mitchell, I’m gonna let you take it home.

    Ashley: Well, I think—what a fascinating group and wonderful things to say. I think the biggest learning for me—and I really appreciate Cornelia and Sanjeev talking about the lost opportunity we had through networking to help people come together and expand their networks. So, let’s invest in that going forward. We need to rebuild those and make that happen. So, I appreciate—I really wanna thank you for helping me understand that better.

    Shimel: Absolutely. Many thanks to Tricentis for sponsoring this episode of DevOps Unbound. We’ll be back in two weeks with a fresh episode, a new panel, a new topic, but hopefully it’s just as interesting a conversation.

    This is Alan Shimel for MediaOps. Until next time, be safe and be well.

    [End of Audio]

  • EP 13: Observability and Performance Testing

    EP 13: Observability and Performance Testing

    In this episode of DevOps Unbound, Alan Shimel is joined by Christine Yen of Honeycomb, Paul Bruce of Tricentis and Mitch Ashley from ASG/MediaOps to discuss how to effectively test the performance of your applications and how bringing observability into testing environments helps DevOps teams identify and resolve issues early in the SDLC. The video is below, followed by a transcript of the conversation.

     

    Alan Shimel: Hey, everyone. I’m Alan Shimel, CEO of MediaOps, Editor-in-chief of devopsy.kinsta.cloud, Security Boulevard, Container Journal, and you’re watching DevOps Unbound. DevOps Unbound is a biweekly video series where we have some of the leading lights in our community talk about topics of interest around DevOps. We’re lucky enough to be sponsored, DevOps Unbound, by our friends at Tricentis, so many thanks to Tricentis for their sponsorships. But this is a media ops event.

    Let me introduce you to today’s panel and then we’re going to jump into our topic. Let me first introduce the guy with all the robots in his background there: Paul Bruce, with a 404 shirt. Paul, welcome. Why don’t you introduce yourself to our audience?

    Paul Bruce: Sure. Thanks, Alan. So, my name is Paul Bruce, as you said. I’m on the nerdy side. I’m a performance and reliability geek. That’s just a place of complexity that I love to – it’s just what – it’s just my bag; it’s just what I like to do. On the other side I’m a community organizer. I work with DevOpsDays Boston and the Boston DevOps Meetup. I am hosting an event along with Liz Fong-Jones this year called o11yfest around observability, a vendor-neutral and also – community-focused but also the open telemetry project-focused two-day event. It’s half days so that we don’t burn people out. And I work with the Tricentis team since the NeoLoad acquisition – again, performance reliability. I was with Neotys and NeoLoad for years now. I really loved that team. And I love my new one, the broader one at Tricentis. So, that’s me.

    Shimel: Very cool. Thanks, Paul. And then, joining Paul today is – we’re very lucky to have her – someone I’ve had the pleasure of interviewing more than several times now over this past year of Covid but hopefully soon to see in person, my friend Christine Yen. Christine is CEO of Honeycomb. Christine, welcome.

    Christine Yen: Hello. It’s wonderful to be here. Alan, you shared my title, but my background is as a product engineer. Honeycomb, for anyone who isn’t familiar, is a tool who – a tool for observability. And I love working on it and I love talking about this topic because even though I tend more towards the software engineering side I have done my fair share of breaking production, and so this topic is near and dear to my heart for software engineers to do that less and find out what their code really did more quickly.

    Shimel: Very cool. Thanks, Christine. It’s great to have you here. And then last but not least is my cohost of DevOps Unbound. We’ve worked together for probably longer than either of us want to admit, but it’s hard to believe we’re both only 25. But CTO here at MediaOps, CEO of the Accelerated Strategies Group, Mitch Ashley. Hey, Mitch. Welcome.

    Mitch Ashley: Thank you, Alan. But the best 25 – this year again. Twenty-five again, right?

    Shimel: Yeah, 25 again again.

    Ashley: Good to be here.

    Shimel: Anyway – yep. Thanks, Mitch. So, today’s show, really, we’re going to explore the intersection – or maybe it’s a border between performance, performance testing, and observability. What – how do they play with each other nicely? What is the interaction? What is the relationship? And for those in our audience, you may be looking at it and saying, “Well, observability is kind of almost post-deployment. Hopefully we’re doing some performance testing pre-deployment and then post-deployment.” But how – I don’t know if I see a nexus right away but there’s – in my mind anyway there’s an obvious nexus. There’s an obvious connection.

    So, let’s talk about it at a high level. How is observability and performance testing connected? And how can they – how can we get a one plus one equals five or four out of performance testing and observability? Christine, if it’s okay, I’m going to ask you to kick off with that and kind of lay out the framework, the ground rules, the – and we’ll all jump from there.

    Yen: Well, laying out the framework and ground rules is a little bit of a big ask for early in the morning, but I’ll do what I can.

    Shimel: Okay. I’m sure you’ll do great.

    Yen: I’m wearing the wrong shirt for this. I often – I think I’ve come on one of these interviews with my “test in production” shirt, but it’s a phrase that too many folks – it can be interpreted in many ways. What does testing in production mean? There is a version of it that means, well, I’m not going to run tests beforehand; I’m just going to see what breaks in production and try to fix it after it’s impacted users. Obviously, that’s not what any of us mean. And there’s a version of it that means, hey, look, when you write preproduction tests, when you write any sort of tests, what you’re trying to do is you’re trying to compare actuals versus expected. You’re trying to form a hypothesis, observe reality, and then think about that difference. Isn’t that really just a different phrasing of what we’re trying to do when we’re looking at production, where we’re using our logging or monitoring or observability tools to see what actually happened when our code is in production?

    And so, I love – I think it’s a huge intersection, a huge overlap between things like performance testing and observability because we are all using the same skill set, we’re all using the same brain sequence, but there’s – people seem to think that observability is – can be more exciting because it’s incidents and downtime and alarms are going off. I don’t think it is. I think it’s all the same. It’s just at different points in the development life cycle. It’s on different timelines. It’s with a different level of urgency. But it’s the same thing we’re all just trying to do to come up with hypotheses, validate them, and learn.

    Ashley: I really like, Christine, the way you kind of intersected them together because the environment that we’re doing all of this has changed substantially. It wasn’t that long ago where it was kind of a static flow of writing codes, we’d test it, we’d CI/CD, we’d test it, we’d deploy it into production. And DevOps helped speed that up with automation and tools, and of course tools of automation for testing. But now the software stack is fluid. The infrastructure is fluid. The code is fluid because we may have introduced three new changes to containers, for example, in a cloud-native application since we’ve really had an opportunity to go and debug or determine what the source of that was. We also have dynamic applications that start up and go away, and those things may be long gone by the time we actually look at them. So, I think this key to observability is also giving you insight into an environment that may not exist at the time that happened, and so you need that information to be able to understand “This is really something I need to fix, address” – a bug, a feature, whatever it might be.

    Bruce: Yeah, and I think there’s a lot of intersection between putting pressure on systems, whether that’s through actual users and risking revenues and all sorts of things or doing prerelease type testing or testing in production. There’s a – performance is about how does this system perform to expectations? And there are a number of different people with a number of different expectations. There’s the technical expectation, maybe articulated – hopefully articulated in something as legitimate as an SLO or an – and then measuring them with SLIs and stuff. But there’s also user expectation. There’s the CEO expectation. There’s B2B. If you’re in some kind of regulated industry and that SLA is actually part of a legal requirement that becomes really important. Like, we saw that unemployment sites – I don’t mean to pick on that kind of work, but that was a mess. Same thing with various different – trying to get the vaccines scheduled and stuff. And it’s like why are we still suffering from these systems that are out there and they’re good enough maybe for the day to day but have never been exercised, have never been – the concept of that being – that scaling has never really come about for that system.

    Okay, well, what about systems that do by default have scalability as a first class citizen? Going back to some of the stuff that Christine said – and actually, Christine, I love – your cofounder Charity has a phrase and I’m going to butcher it, but “It’s never not a good idea to have your glasses on when you go driving.” That idea that if you can’t see what’s going on that’s a big problem.

    So, a lot of people traditionally would have used something like “Let’s just throw all our APM tools at production.” And that’s great. But going back to that original thing where I was saying pressure on systems really shows us how well we’re doing engineering, the impact of that pressure matters too. So, as a performance engineer, performance and reliability engineer, I’m constantly not just trying to – as my French team would say – make load on a system, put load, put pressure on a system, but I also care deeply about being able to measure the impact of that.

    I’ll pause there because we probably some other questions. But there’s more to go into about that. It’s not an either-or. And it’s not about just production or preproduction. It’s about what’s not a good idea to have visibility. Isn’t it a good idea to have visibility on a thing, whether it’s putting pressure before release and having visibility or doing that in production? We’ve got a lot of customers who do testing – who do load testing in production on production systems. This is not an impossible thing, folks, you know?

    Shimel: So, let me try to put this in some kind of context that we can jump on. So, here’s what I say Christine saying, which is – and maybe I’m taking it up a couple notches, getting out of the SLO woods – at the end of the day what do we all want? We all want our apps, our infrastructure to run better. We want to have good predictability, visibility into how things are performing. And whether that is an aspect of performance testing, or you want to call it performance monitoring, or do you want to call it observability, at the end of the day the aim is the same, which is let’s understand how our stuff is running and make sure it’s running the way we think it should be running. So, at a very high level it’s almost equating performance or observability – I mean, it’s all with the same goal, which is let’s make sure our stuff’s running good, as the best we can.

    In my mind – and Christine, I’ll defer to you – certainly over the last year, year and a half, maybe two even, observability has taken on – as it’s become more hyped and a real word, a real thing out there, has taken on almost an even bigger mission than that, where it just doesn’t equate with – the goal may be “Hey, let’s make sure things are running the best we can make them run and how we expect them to run,” but there – observability has a whole uber mission of things that people want to do with it or use it for. Paul, you mentioned APM. Kind of observability subsumed APM, AI ops, all of these things. But yet, wait, there’s more. Observability does more than that. It’s become this whole big thing out there. And maybe it – maybe that’s right, maybe that’s wrong. I’m not here to judge.

    But that, I think, is really the crux of this. Yeah, we all have the same goals. Is observability – part of what observability does helping the performance engineer? Absolutely. Is there more to observability? I think so. Christine, you’re the observability expert. I’m going to ask you.

    Yen: You know when you say a word enough times and it starts to lose its meaning and you’re like “This is starting to sound funny”?

    Shimel: Yeah.

    Yen: We – I think you’re – there’s – we are definitely approaching that with observability a little bit. And whenever I get to that point I like to pause and remind folks “Let’s look at the – let’s take a deep breath and look at the English word we are using to describe this. It’s not a bucket full of buzzwords. It’s not a bucket full of types of data. It is an ability to see. And it’s the ability to see into our production systems. What do you want to do with that? What can you do with that if you have an improved ability to see?”

    Okay, well, now we have the laundry list of all the things that we can do once we have the ability to see into these systems that, as Mitchell mentioned, are so different than they were five or ten years ago.

    Shimel: Oh, yeah.

    Yen: And that to me is – that zooming out behavior is what re-centers me when it feels like we’re trying to do all the things. And you’re like “Well, it kind of makes sense. I want to have my glasses on when I’m driving.”

    Shimel: Yeah. Paul?

    Bruce: Christine, you’re definitely right about a buzzword merry-go-round. I have experienced that with “DevOps.” I have experienced that with – oh, man, I can’t believe I’m saying it again but I’ve got to because people will know – “shift left.” These terms that just get all the things. And we go at these things, these different terms, we latch onto them, we try to make it our own. Oh, man, vendors love to do that. We’ll take a word and we’ll make it ours and we’ll try to align it to our product strategy. And at the end of the day it’s like there’s 150,000 different versions of this thing, which as a performance guy I’m kind of like “Well, it’s not bad to have a big sample set,” but the sample set of what’s going on in the delivery chain from good idea to working software that’s making you money is not just in production.

    Let’s take an example of Nike, for instance. This is a known thing that they ship the watch, the watch that you open up on Christmas Day, and you go to connect to the internet and because the servers were swamped you don’t have a good Christmas experience. That’s not the only – I’m just picking on… But that kind of situation is a situation where there’s a lot of stuff you can do to prevent obvious faults from getting to the point where that’s the bad experience. And you also need visibility on the day where your production is having problems. And it’s not just the day; it’s every day. Every day you have a certain un-zero amount of events and issues in your production systems. How do you address those things? Well, there are plenty of situations where if you’re only thinking about observability as it only applies to production – which, by the way, production is where – what makes you money. If code is not sitting in production doing something useful for people you are not making money off of it necessarily. And production isn’t just out for consumers. It could be all the different constituents in your larger organization. So, even private systems are production. And so, if something is not out there then it’s not making money.

    So, is that the only place where we should pay attention, where we should have good visibility on logs, traces, and metrics, on understanding how the business is impacted, long and complicated queues and delays in these systems? No. I mean, yes, in our edge and final production systems, yes, of course we will want that because we don’t want to be in a bad situation not having the right visibility into problems when they’re happening in production. But how do you think problems come about in production? Well, there’s the emergent ones that we can’t possibly know about. Ooh, there’s just so much time in the day. But then there’s a whole bunch of other stuff. Every line of code is not just an asset; it’s a liability. So, as you’re constantly shipping these things out there’s plenty of things you can do to prevent obvious – and I mean dumb and stupid and obvious things from getting out there, causing toil, waste, risk, and loss of brand, revenue, and all that stuff.

    So, I think the observability for me is not just in production. It’s really useful to have classic monitoring in preproduction environments except for that it’s super expensive. Oops. And then, observability, pieces of observability, which Christine would probably be better to go into, all those things that you can do and what you get out of that, and now we can have use cases, that also applies to preproduction situations. That’s why I’m so excited about that open telemetry project and the community over there, this notion of being able to emit information from your systems, to either build it in directly from your systems or put it on sort of after the fact, sort of replacing some of the obvious early agent type stuff that monitoring used to be the only thing that could do. That open telemetry stuff now makes it a standard that we can all benefit from no matter which environment. How about all environments? That would be super nice. And to get tracing and traceability from – all the way from the Web app that somebody is touching all the way down to the database and the 30 different APIs that it touches.

    That’s really useful, not only to firefight in production but to see “Am I risking dependencies? Am I go wildly off my architecture diagrams?” You know what I mean? Those things slide. Your conceived architecture diagram is never what’s actually running in production. So, to actually be able to have visibility on that is really important.

    Ashley: Well, you know, Paul, the lines between production and, let’s say, tests are blurred, to use a phrase – are really kind of blurred, and oftentimes when we adopt whatever new technologies or approaches we’ll attempt to do what we used to do just using a new technology to do it. So, we’ll test the way we used to. And just what sort of clicked with me with cloud early on was how Netflix was doing their rolling upgrades and being able to flip back and forth. The same is really true for testing and performance testing because your availability to resources generally is not an issue, or not the issue it would be normally in an environment where you’re not in the cloud or at least partially in the cloud. So, there’s a lot of things you can do not just in your current testing environment but to set a pretty elaborate – with resources to do different load tests, performance test scenarios, in addition to what you experience in production.

    And I think one of the points that you made is salient, is if you’re paying attention to this you now understand your systems, your applications, your environment enough to identify “Well, we may be able to handle the number of unemployment claims we’re getting today but I know when we reach the next peak these are the three things it’s going to impact. I know those are the things we have to go adjust or fix or whatever it might be.” Because you’ve already instrumented it well enough and you have the information through observability to be able to do that. So, I think that’s a different mindset of how we think about performance testing.

    Shimel: Yeah. So, I tell you, I think one of the big breakthroughs of DevOps has been to create our “testing environments” to be more lifelike, to be more real lifelike. I mean, because prior to DevOps, the world I grew up in, the classic thing is the developer developed, gave it to the ops guy, the ops guy puts it up and says, “Wait a second, this stuff don’t run.” And he says, “Hey, developer, this doesn’t run.” And the developer says, “I don’t know, dude. It ran on my machine.” And so, there was that disconnect between what the developer wrote and ran it on or they tested it on versus the real life production environment. And one of the great things about cloud and virtualization and all of this stuff is that we can theoretically really duplicate our production environments and test environments and we can increase loads and we can do all those things theoretically so that there’s no or less surprises when we go live in production. And that – look, that’s – thank you, DevOps. That’s part of what this whole DevOps thing, I think, was.

    But I don’t know, is – Christine, is that really – does that dog hunt or is that just DevOps urban myth? You know?

    Yen: I think you’re on – right on the ball. I mean, when Charity and I started Honeycomb she came right – that story you told, I was that dev; she was that ops. I did the whole “It works on my machine; I don’t know what you’re talking about.” And when we started Honeycomb we thought honestly we were going to be talking to more of the Charities of the world. We were like “Oh, this is a tool for ops people. They’re the ones who understand the pain.” And as Honeycomb grew and as observability grew it became more and more obvious: “No, this is for the Christines of the world too.” This is for the people who need to understand what happens beyond that testing environment. This is – the boundaries are blurring or – I like to think of it as dev and production are coming closer together because there are more and more things we just can’t test in a test environment. As good as we make the test environment, we just can’t.

    Shimel: I totally agree.

    Yen: And there are so many things that can be learned – the same – by thinking of production as part of the development environment. With feature flags we can start to take code that is really not quite ready for production but test it out in a safe way and learn from it. And that’s so cool. It’s – there’s so much for developers to be able to learn when everyone kind of gets over this mindset of “Well, there’s a wall between development and production.” There isn’t. There shouldn’t be between people or the software. And that’s where all the exciting parts live.

    Shimel: I don’t disagree. I don’t disagree.

    Bruce: So, Alan, one thing that you said – you painted sort of the stage so that we could have part of this conversation. I’m not a big fan of setting up huge staging environments. In classic world it was like, okay, well, here’s production and it uses 36 Oracle back-ends and – if we were going to run a proper performance test of course we needed 36 on our side. And before the cloud that was incredibly expensive and you’d fax your IT person and they’d get back six months later with the right hardware. And I think there’s a myth that, well, yeah, you need to be realistic at some point, especially when you’re putting pressure on a system, to answer the final question, which is – what? – is it ready enough for me to be – what? – confident enough that this thing is going to transition like we talk about in standards, the transition process, that that’s actually going to go smoothly? And not in a waterfall way, just in a – to get away from this notion of “It worked on my machine” and now the ops person is putting out fires and doesn’t know how to operate the thing.

    But I’m not a big fan on spending a ton of money on this staging system which ultimately even if – even if at a very root level it’s exactly the same as the production system there’s always that jerk in the corner going “Yeah, but it’s not production.” And it’s like “Okay, you have a point but you don’t know what goes on.” And the point is I think almost like a – I’m going to do my idiot thing and pull in a mathematical and then probably get it wrong, a mathematical concept of the limit. You remember limits? And the idea is we need to be able to express are we talking about the very ends of things? No, it’s a continuum.

    So, there’s a lot of companies that I work with that actually do have requirements, that they go through a particular amount of testing and particular things before they can transition this to production. They don’t have the luxury of treating production like a dev system. There is a ton of people in the world, and it’s not their dysfunction – that’s not a dysfunction, just because the unicorns can fart rainbows and sneeze glitter and launch whatever they want. I see that happening in Boston all the time in some of the startups, and they do actually put themselves at risk. And even when you don’t do that, when you have an extreme amount of maturity and you’re working with relatively complicated technologies like Kafka things can go wrong pretty quickly. And the question is do you have the people and the process and the technology and the knowledge and the experience and the expertise to handle those when those things do come up in real time?

    How do you get there? How do you get to people who know how to use your monitoring and observability tools really effectively? How do you know what those systems, how close to your perceived architecture versus your actual in production, how do you know how much delta there is there? How often do you exercise the process of disaster recovery is one of those other classic questions you might ask yourself.

    And it gets back to who would be involved when there is a firefight or a last-minute thing like what we’ve done with some of the companies that I worked with very closely through the NeoLoad side of things, is when the education system, when colleges shut down and they finally said, “We are shutting our doors to the people that are paying us money,” that was a scary and real and significant flag for the rest of our US education system to go “Holy crap. We’re going – we have to provide virtual not as a stopgap for the past few months but this is going to be a serious reality for us for the next year.”

    And so, a lot of the vendors that help the education system have the right platforms in place. They go “Holy crap. We really need to test for 10X the size that we’ve ever tested before.” So, they’re testing in production on nights and weekends in various different places and doing it at a smaller pace. But who do they have on the call? Not just the Ghostbusters, but the devs, the ops. There’s people from infrastructure, the DBA teams. They have 20-plus people white-knuckled on the call, like [makes quivering sound] – white-knuckled on the call, and that’s a lot of money. And by the way, you’re asking them to do from 10:00 PM to 2:00 AM to 3:00 AM work. And so, they’re not going to be available for all those fancy meetings, architecture meetings that they should be stakeholders at.

    So, when you get to the point where you’re doing those massive things sometimes it’s necessary to go massive. But the vast majority of the exercises, you go to the gym – not so much me, but you go to the gym to exercise. You go and you get on your bike and that’s an exercise. You don’t expect to run a race without exercising first, without knowing that path a little bit before you just run a 5K race. So, why would you expect to do that with your systems and your teams?

    And that’s where some of the testing, the lower order performance testing comes into play. It’s systematizing the stuff, making a process that can be executed and re-executed no matter which environment it’s pointing to. And the telemetry that you’re getting out of it aligns in each of those environments. If all of a sudden we have really great telemetry in a pre-prod environment but we don’t have it in prod that would be a big problem. Why was it put in here and it’s not represented in production? What if we need that? Or vice-versa: We’re only doing things in production, and so now there’s only – there’s stuff that’s only in production that has no representation of how it would show up in lower environments.

    So, keeping those things in synchronous but also keeping the process and the exercised skills in place is why this isn’t just a big environment at the end kind of a situation.

    Shimel: Again, I think “Thank you, DevOps” where we have been able to say, okay, not only do we need those or can we use those testing dev environments, but as Christine said, with things like feature flags and some of the other technologies, processes that we use, it is possible to continue that – the feedback loops and iterations in real time while using the – while in production. And that again is one of the things that allows us to continually update, continually iterate, continually – and feedback loops was kind of an original concept as part of this DevOps thing. We’re going to have those feedback loops and we’re going to iterate and reiterate.

    But where we’ve gone with observability with that and with performance testing and these things with that have really just opened the floodgates in how we do these things now. I mean, just think about how – when I first started devopsy.kinsta.cloud eight years ago, I mean, there were the unicorns who were updating multiple times a day. But for many people in the world, for many organizations in the world, look, moving from once a year to twice a year, twice a year to quarterly, quarterly to monthly was huge. It was huge. It’s not so huge anymore. It’s common for people to be updating their code daily or more than once a day now.

    I saw – I did an interview this week, a recent survey of developers who they claim that they have sped up their deployments by over 100 percent in the last year and a half. That was – something like 70 percent of developers said that. Think about it. I mean, that’s – a billion here, a billion there, you’re talking real money. Seventy percent of developers say they’re going 100 percent faster. Wow. Wow. What a golden great time to be living. And it’s because of things like observability, DevOps, our better performance testing. Anyway.

    Guys, we’re coming up on our time limit here. I wanted to kind of – let’s wrap it up. Let’s – what advice – for our folks listening in, watching in here, what can we give them? What can we tell them about this nexus of observability and performance that will make their life better? Paul, I’m going to – if you don’t mind, I’m going to put you on the hot seat and ask you to go first.

    Bruce: Sure. Well, there’s a couple things I’ve been working on for a while and then there’s a couple things that I just want to give a shout out to. I’ll start with – Christine doesn’t get to do the pitch thing but I do. Honeycomb is awesome. You see that thing in play and it’s awesome because the team behind it is awesome. You know what I mean? It’s kind of the Conway’s law situation where – the version of it I hear most is the systems are a result of the teams that build them. So, definitely check that out.

    Same thing with some of the other tools around there. But don’t get too bought up into the whole observability – don’t just Google “observability” and look at the first one because most people are paying through the nose for SEO placement there. Really actually look at what’s available, and then also dig into the team, the blog, the spirit behind that thing.

    The second thing I’d suggest is we just released – I was one of the members of a working group for a standard on DevOps principles and practices that applies to highly regulated industries. It’s IEEE 2675-2021, which we’re hoping to get adopted in ISO so it’s global as well. And what it does is really align what is – how are each of the important processes that can be applied continuously through many different life cycles as they’re going along? You know, that DevOps eternity symbol means we’re all going to be stuck in hell forever, that thing is happening, all the things are happening all the time but there are precise things that need to go on in highly regulated industries, and even not. So, we’ve been working on that for four years. And you can reach out to me on LinkedIn, on Twitter – although LinkedIn is probably a better way to go – if you want to know more about that.

    But essentially it’s a body of work that a lot of people have been putting time and effort into to really say, “Hey, look, this is not about what DevOps is. This is about how to do this in these complicated situations.” And you know me; as a performance geek I tried to – I was involved in QM/QA, verification, validation, risk management process, those kind of things. And I tried to make sure that evidence-based decisions, that – which is a hook to basically say visibility, telemetry, the proper SLOs in place, those are definitely in the standard along with other things.

    So, definitely check those two things out: Honeycomb and the IEEE 2675.

    Shimel: Excellent, Paul. Christine.

    Yen: Thank you for that plug, Paul. The one piece of advice I’d like folks to internalize, we’ve talked quite a bit about – or, I’ve talked a bit about developers in production and developers in observability. An asterisk there is that that is not – the practice – the muscles and the practices should transfer well but there are a lot of things in the DevOps world in production that are, well, in some cases outright hostile to developers and in many cases just kind of scary and unfamiliar. And so, if this is something that’s interesting to you, if you’re like “Ah, yes, I would also like to bring development and production closer together,” take a step back and look for things like that. Do your production tools primarily talk about AWS instances and CPU and memory use? Those may be things that might scare your developers off. How can you make the vocabulary that your production tools talk about more familiar to developers? How can you help them bring the concepts and nouns that they’re used to in testing – customer ID or endpoint or logic, something that feels more familiar to the developer world, how can you have that exist in your production tooling as well so that developers can show up and be like “Oh – oh, this isn’t so different”?

    I think that there’s a lot of things that are encoded in certain practices because we assume that a certain persona is the one using them. And those walls are blurring – or those walls are coming down; those edges are blurring. There’s a little bit of work left to do to facilitate it.

    Shimel: Absolutely. Hey, Mitchell, you want to bring it home?

    Ashley: Yeah, I’m going to wrap it up this way because there’s some great practical advice – and by the way, both companies have fantastic products, so please check them out. I think we have this mental model, this habit of thinking of things in one state. Matter has four states – well, five states if you count Higgs boson – Bose higgson. It’s not just solid. It is also fluid. I mean, think of our systems. Don’t think of it as software tools, all the layers of it. Combined together they operate more like a fluid, in a state that’s changing, constantly moving. Things are happening. And that causes you to rethink developers and production, doing production – doing performance testing in a production environment. And the flow of software through that DevOps tool chain is really part of that – the fluidity of that state.

    So, I think we like to think things – of things like “Let’s make everything a solid so it doesn’t change, so we can look at it, observe it, do something with it.” It’s not really that way. That’s not the world we live in today. But if you think about it as something that’s under constant change like a fluid, then I think we have a much better chance of figuring out better ways, whether it’s through technology, new paradigms, new ways of working, of how to make our software the best it can absolutely be.

    Shimel: And at the end of the day that’s what we all strive for. Right?

    All right. What a great way to end this, then. This is a wrap. I think it’s episode 13 of DevOps Unbound. But it’s a wrap on whatever number this was because it’s fluid anyway. It’s a wrap on this episode of DevOps Unbound. Many thanks to Tricentis for sponsoring us on DevOps Unbound. We will be back in two weeks with another great episode. Stay tuned. We also have a big roundtable coming up this month; check that out.

    Paul, hey man, congratulations on joining the Tricentis team as part of the acquisition there.

    Bruce: Thank you.

    Shimel: Christine, they’ve all said great things about Honeycomb, but well deserved. Say hello to Charity for us as well.

    Yen: I will. Thank you.

    Shimel: Keep doing what you’re doing. I hope to see you all soon or a couple months from now in person. Maybe we will do a live DevOps Unbound at some time somewhere. But until then, this is Alan Shimel for MediaOps. You’ve just watched DevOps Unbound.

    [End of Audio]