Category: DevOps Practice

  • IT Automation Tools Are No Longer Enough

    IT Automation Tools Are No Longer Enough

    As Chris Gardner—who conducted the Forrester Wave on Infrastructure Automation Platforms—stated, “Infrastructure automation isn’t just on-premises or in the cloud. It’s at the edge and everywhere in between.” And that’s because, well, it needs to be.

    It shouldn’t come as a surprise to anyone reading this that IT automation is a must-have for organizations today. With infrastructure spread across on-premises data centers, multiple clouds and a multitude of technologies, the complexity of IT long ago outgrew manual management. Automation is the only way to keep pace with constantly changing environments that need enhanced security, greater compliance and be easily adaptable to changing business and market conditions.

    Because of these needs, the market has been relatively quick to adapt and embrace automation technology. According to a recent study by Forrester Research, 56% of global infrastructure technology decision makers have implemented/are implementing or are expanding/upgrading their implementation of automation software. Many of these IT organizations have a trusted tool in place and have been automating scripts and tasks for years.

    Along with outgrowing managing infrastructure by hand, the industry is also quickly outgrowing automation tools. The automation market has now reached a point of maturity where an automation tool alone is not enough. In order for automation to continue to have an impact on an organization and keep pace with the speed IT changes, organizations need a platform that allows them to create a culture of automation.

    Automation as a Trait, Not a State

    Typically, when an organization acquires a new technology, they try to adapt it to their needs and how they work. Automation technology, along with most technology tools, does exactly that—it lets you adapt it to how you want to use it, and performs tasks as desired. But ultimately, successful automation comes down to more than just supporting task execution. Enterprises have realized that they need to support the behavior of automation in order for it to continue to scale.

    As opposed to automation being a state, use automation as a trait—something that is done naturally without thought. The analogy we like to use is one of learning a new language. When you learn conversational Spanish, your mind still thinks in English, then translates what you want to say into Spanish based on what you have learned. Yes, you improve over time, but when you start to dream in another language, that is when it has become second nature. It is no longer something you have to do, you just simply do it.

    When automation becomes second nature, that is when it can make the biggest impact and will help address a continually evolving infrastructure. As a discipline, automation is not technology-specific. As much as we talk about the technology behind it, it actually has little to do with automation. The technology itself is the easy part, the part that is reliant on a tool. Many have reached a point of maturity with using tools and are now working toward IT staff viewing automation akin to testing or documentation—while we may not always do it, we know we should. 

    Having automation become second nature is easier said than done, and something organizations are struggling with, realizing that a tool alone will not support this effort. It doesn’t provide the coordination and collaboration that is desperately needed to unlock the next level of automation capabilities. We need more than a tool to continue to scale automation and spread it to new areas of IT.

    The Next Phase of Automation and What to Look for

    There are multiple personas within an organization that are paramount to successful business-wide automation, and, to be blunt, automation must cater to all of them. Between developers, those trying to simply consume automation or sys-admins executing automation of mundane tasks, they all need something that works for them—but can also work consistently across these personas. Organizations need a common automation language across the entire enterprise and the hybrid cloud that creates a standardized experience for solving problems (and solving them only once). 

    Not only that, but they need a repository or catalog where everyone can find an answer to what they need, along with easily accessed integrations. Part of why automation is growing so rapidly is because of how many technology vendors have embraced the concept and, in turn, are working with automation vendors to integrate their technologies and make them easier to use.

    To continue to spread automation to new areas of a business, organizations need a single repository to more securely search and find necessary modules. Additionally, having a shared repository helps create an automation community within an organization. It brings different personas together to find what they each need, fosters collaboration and ultimately encourages sharing automation specific to their organization. It also provides a place to find integrations that they may not even have been aware they needed.

    When preparing for the next step in their automation journey, organizations should look for automation platforms that provide access to analytics, specifically those that can tell you how your automation is being used and where. As the adage goes, knowledge is power, and the more you know about your automation, the more successful it can be. More importantly though, if you are trying to create it as a behavior, analytics let you track and measure progress. It allows you to better understand what is working, what groups are thriving and what areas need improvement. Using automation analytics enables organizations to create a system for enabling standardized practices. 

    On the whole, a successful automation initiative needs a supporting automation platform to provide coordination and embrace collaboration between different groups. Simplification is also essential for making automation more accessible across these groups, especially in the face of competing needs and varying levels of engagement. Scaling automation successfully applies both to technologies, as well as the users that want to benefit from automation.

    Strategy for 2020

    There is no such thing as a perfect tool, and don’t let anyone try to tell you otherwise. The good news is it’s not about having a perfect solution, it’s about the experience that can be gained from a solution. Every organization is unique and there is no one-size-fits-all option.

    What is needed when developing an automation strategy that takes organizations into the future is a foundation for growing that automation. Automation platforms offer so much more than a tool alone, allowing it to be more easily scaled and being leveraged how the organization needs. It may never solve all problems but will take away a lot of the efforts. You need to find a vendor that is not selling a tool—but selling automation as a whole.

    — Richard Henshall

  • Threading the Needle on Kubernetes Complexity with AI-Powered Observability

    Threading the Needle on Kubernetes Complexity with AI-Powered Observability

    If you run down the laundry list of major industries–transportation, retail, healthcare, automotive, financial services–you’ll see that the major players in each have essentially become software companies. What they do and how they serve their customers rests entirely on their software working perfectly. Underpinning this is a massive digital transformation that has shifted organizations away from the old, static ways of enterprise IT monoliths. Instead, today’s businesses are working in more dynamic IT environments–agile, multi-cloud and, especially for new digital services, based on containers and microservices. All of which means embracing more IT complexity than ever before.

    Kubernetes Brings More Data to the Table, but that Impacts Observability

    A key part of this more complex environment is the soaring adoption rate of Kubernetes over the last few years. In a survey of 5,000 enterprises, 40% said they were currently running Kubernetes in production and over 80% reported that they’d adopted Kubernetes over any other container management system. Another recent survey revealed that 68% of CIOs are already using containers and a total 86% expect to deploy them over the next year. Enterprise IT is increasingly gravitating to containers and especially to Kubernetes for orchestrating their containerized applications.

    What drives organizations toward containers and Kubernetes is the flexibility given to development teams to release services, features or apps faster. But faster doesn’t necessarily mean better or higher quality, unless you also apply cloud-native architectural patterns, include quality gates into your delivery pipeline, embrace new deployment models such as canary or feature toggling and provide full visibility and short feedback loops from production back to engineering.

    Kubernetes itself provides great visibility into node or pod health. Kubernetes even ensures the health of your pods through its built-in health check mechanisms. Thanks to components such as service meshes, you also get live information about dependencies between services running in your pods. This data can, in theory, provide organizations with the necessary insights into their environments to understand whether their organization is releasing better value faster to their end users and unlocking new business value.

    The reason I said these are theoretical benefits is because, in practice, all that extra data doesn’t necessarily mean more visibility. Often it can mean less visibility–and with that, less insight into how the different elements of your environment may be impacting speed of production deployments, overall system performance and the end-user experiences. I’ve seen Kubernetes environments that are packed with literally billions of interdependencies between applications, containers, microservices and clouds. Asking human beings to keep track of that level of data manually is an impossible task now. And relying on traditional, manual approaches to manage and monitor IT environments shortchanges the value that organizations should be getting out of Kubernetes.

    Kubernetes Environments Need Full Stack Observability—that Means Needing Deterministic AI

    Getting the full potential out of Kubernetes means having observability across the full technology stack and those billions of interdependencies. But as more organizations are discovering, that can’t happen without AI. Specifically, a deterministic AI engine. In fact, in a 2019 survey, 88% of CIOs said they were looking to AI as a critical solution for overcoming IT complexity. As long as Kubernetes is fueling complexity, overcoming that complexity means making deterministic AI an integral part of any Kubernetes environment.

    A deterministic AI applies specific algorithms to detect system anomalies and apply its collected dependency information toward detecting the root causes behind those anomalies. When that deterministic AI engine ingests critical Kubernetes events, such as state changes and workload changes, it provides visibility into the Kubernetes stack’s interdependencies. That AI-powered, full stack observability matters because it gives IT new capabilities for their environments, including instrumenting and mapping containers, managing and allocating container resources and drilling down into container runtimes at work. All of that is possible due to the level of observability that you can only get with deterministic AI; traditional metrics and dashboards just can’t compete.

    By providing full stack visibility and interdependency mapping across a Kubernetes environment, deterministic AI is empowering IT teams to identify, in real time, the root cause of degradations in business service availability–from reliability and resource usage to system performance and user experience–faster than they ever could manually. That level of automated speed and visibility also makes it possible to immediately resolve those issues at the source before they can impact the end user. The automation provided by deterministic AI also works its way into DevOps’ production pipelines, accelerating the release and feedback cycles of new deployments and allowing teams to more quickly incorporate user feedback into the next iteration for even better experiences.

    In short, a deterministic AI engine makes life better for both IT and the customers they’re aiming to serve. Deterministic AI takes work that used to be time-consuming, tedious and counterintuitive when done manually, and performs it fast and automatically. That then frees up the teams that used to do that to focus on work that is more satisfying and drives new business value.

    AI-Powered Observability Relies on Quality Data and Quality Monitoring

    All that said, a deterministic AI’s ability to provide reliable, precise full stack observability in a Kubernetes environment relies on it being given consistently quality data it can process. The algorithms are dependent on the data that’s ingested by the AI engine. If there’s no standard monitoring capability being used across the board, then you invite a scenario where different teams bring their own preferred monitoring tools into the fold. The bigger a Kubernetes cluster gets, the more diverse the set of monitoring tools can be. And the greater the diversity of monitoring tools, the more inconsistent the data being fed into the deterministic AI engine.

    The way out of this predicament is to leverage a single monitoring tool as a self-service capability into Kubernetes clusters, one that can support the diversity of technologies that are introduced into the stack and provide a baseline standard of system and event data that developers, IT administrators and the deterministic AI all lean on. In other words, when you feed consistent, quality data into the AI, you make it smarter. And a smarter deterministic AI engine provides the full stack Kubernetes observability that organizations need. 

    Building a Positive Feedback Loop Between Kubernetes, Deterministic AI and Observability

    Kubernetes and deterministic AI are a natural fit. Kubernetes provides a platform for managing and instrumenting containerized applications. The deterministic AI provides a tool for mapping out the Kubernetes stack, offering insights into interdependencies and degradations where they arise and resolving them before they can affect end users. The data from Kubernetes makes AI better, and a smarter deterministic AI engine means more precise observability across your technology stack.

    It’s a self-reinforcing feedback loop that eliminates the usual pitfalls of container-induced IT complexity. Instead, it creates new standards of visible, consistent and quality data that organizations can leverage for faster and better deployments–and in turn, better user experiences and greater business value.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or August 2020.

    — Andreas Grabner

  • EP 5: Outsourcing DevOps, Oxymoron or Truth

    EP 5: Outsourcing DevOps, Oxymoron or Truth

    Can you outsource DevOps? And if you can, what are the keys to doing it successfully? In the fifth episode of DevOps Unbound streaming broadcast on TechStrong TV and DevOps.com’s sister site Digital Anarchist, Mitchell Ashley of ASG and Alan Shimel are joined by Ming Gong of UST Global, Chris Rolls of TTC and Robina Laughlin of Guardian Life for a discussion on outsourcing DevOps.

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript

    Alan Shimel: Hey, everyone. I’m Alan Shimel, and welcome to another episode of DevOps Unbound. DevOps Unbound is a series, sponsored by our friends at Tricentis, that explores different topics around DevOps every other week, as well as having a monthly roundtable open to the public. Today’s episode is “Outsourcing DevOps: Oxymoron or Truth?” I’m joined, as usually, by my co-host, Mitchell Ashley, CEO, founder of Accelerated Strategies. Mitchell, welcome. Thanks for coming on.

    Mitch Ashley: It’s always awesome to be here. I’m excited about our guest today.

    Shimel: So am I. Let me introduce you to our guests. We have Robina. And, Robina, I’m gonna ask you to give your full name, title, company, background. And we’re gonna ask you to kick it off.

    Robina Laughlin: Sure. Hi. I’m happy to be here. Thank you. My name’s Robina Laughlin, I work for Guardian Life insurance. Actually Guardian Life and I have been working in the quality assurance space now for about 25 years. I’ve been working for Guardian for eight years. We have a great enterprise quality assurance journey. Most recently my department, my quality assurance department’s been working with integration into a DevOps model. So we’re starting that journey. We started about a year ago. And really happy to be part of the conversation.

    Shimel: Thank you, and thanks for joining us, Robina. Secondly, I’m gonna introduce my friend, Ming Gong. Ming, a little background?

    Ming Gong: Hey, Alan. How are you? It’s always good to be here. My name is Ming Gong and I work for UST Global, I’m vice president. I run a practice called Digital Agility Platform Solutions. So what we do is we provide all kinds of services and even product to our enterprise customers, things such as DevOps transformation, DevSecOps, digital, cloud transformation, as well as latency modelization. So, we have tons of battle scars as we’re implementing DevOps for our customers.

    Shimel: Absolutely. And then, last but certainly not least, Chris?

    Chris Rolls: Hi, nice to meet everyone. So my name’s Chris Rolls. I’m from a company called TTC. I’m CEO of Americas. And what we do is we help organizations with quality engineering, continuous testing and also Agile and DevOps transformation. So really happy to be here and excited to hear what everyone else has to say.

    Shimel: Absolutely. So, guys, when I started – so we launched DevOps.com in March 2014. Working on it all through the second half of 2013. At that time it was sorta dogma in DevOps that DevOps wasn’t about the tools; DevOps was about people; DevOps was about culture. And that therefore, to be successful in your DevOps journey, outsourcing was a no-no. You couldn’t be successful and outsource your DevOps because it would destroy – you never got to the nitty-gritty of the culture. 

    Now, that was seven, eight years ago. A lot has changed. I think we’ve seen some very successful – the world is full of DevOps consultants, large, small and in between, who are helping organizations along their DevOps journey. So I think we have kinda moved beyond the dogma. But have we moved to truth? Can you successfully outsource DevOps? And if you can, what are the keys to doing it successfully? And I’m gonna throw that out to our panel today, and we’re gonna start that as our discussion. Ming, not to pick on you, but you already mentioned you have a lotta scars to prove it.

    Gong: Absolutely [laughs].

    Shimel: Tell us a little bit about your journey or UST’s journey in this space.

    Gong: Yeah. I think you mentioned a couple of things that’s really important, which is, ultimately, I think to do DevOps correctly the entire delivery team needs to live and breathe the DevOps – not only the automation, the tool set, but also the processes as well as how you do things or the whole mindset thing. So that’s the absolutely ultimate goal. I think even over the past few years, even though all the companies all moving towards DevOps, but they are still having a lotta challenges in terms how to take some isolated success in certain areas, trying to cascade that throughout the entire organization. And I think this is where, when we talk about, “Hey, is outsourcing a myth or a reality?” I would say it’s actually somewhere in between. 

    Because it really depends on where the continuum – when I think about DevOps, I think about DevOps continuum, ___ maturity, where they are. For the company who’s really mature, they are less likely to benefit from outsourcing before the folks who’s just starting the journey or middle of the journey – they definitely can use some help. And not only outsourcing in general but also what part of DevOps to outsource as well. So typically when I work with enterprise customers, they have a centralized organization, like some kind of DevOps COE. But they also have each individual team that needs to be enabled. So where we found there tends to be more success is when we help the COE to – infuse them with the DevOps expertise and help them kinda build up the practice around DevOps.

    So I definitely think it’s somewhere in between.

    Shimel: Fair. Chris, your organization is similar to Ming’s in that you are helping organizations with their transformation. Has your experience been similar, different?

    Rolls: Yeah. I think I’m not gonna shock the world by being a consultancy who sees consultants can help with DevOps transformation, right? I would be outta business if I thought that was the case.

    But if I was to answer your direct question, can you outsource DevOps? I would say no. Because DevOps is about a cultural change within the organization as well as the technical practices. Now, similar to what Ming is saying, it’s sort of in the middle because outsourcing and consultancies and other vendors can be a part of the solution. They can make some things more challenging in certain ways because you are introducing certain barriers, as in contractual sort of challenges around it. But it definitely can be part of it and it definitely can help, particularly when organizations are trying to figure out how to do things better. 

    Shimel: Fair. So, Robina, you look at this sort of 180 degrees different than Ming and Chris do, right? You’re not providing consulting to organizations. You are the organization. What’s your view on this?

    Laughlin: Yeah, thanks for having me. This is a really interesting conversation. Some of my thoughts around this is: I think the important thing to understand is: what’s your driver to DevOps? Why are you moving to that model? Why is that model gonna be important to you as an organization or as an enterprise? And when you start to answer that question, you get to different models. For us, we adopted SAFe. We started that journey two or three years ago. So DevOps was not the first thing we tried to conquer out of that transformation. It’s on the tail end of it. And it was always on the strategic road map. However, what we found is that it grew organically based on the improvements from adoption of SAFe and agile. So for the teams to get to the velocity that they wanted to do, it became a necessity as opposed to an aspirational goal, right?

    So for us when it became that driver to success, it took on a whole different model. And then when we started figuring out how do we wanna do this with our partners, the vendors that we work with, one of the things that we always hold true to in Guardian is that frameworks, guardrails, models our Guardian culture, and we developed those. We developed them in conjunction with our partners. We’ll seek for collaboration. We’ll seek for guidance. But the model has to come back to fit our culture and how our folks work.

    And then once we have the guardrails in place and the operating model, inclusion of an outsourcing model is very successful, I find. But I speak from a QA perspective, and I’ve been on an outsourcing model now for 15 years, right? So I always kinda take the same approach to things in that if you have the operating, the guardrails, and people know what they need and why they’re doing it then outsourcing is a good solution. But you have to have a strong backbone for the company that you’re working for and with to be able to represent that model and the culture. 

    Shimel: Mitch – oh, Ming, go ahead? Whoever, Chris.

    Rolls: Oh, sorry. I think just to drill into that as well, Robina mentioned you’ve gotta understand your drivers for DevOps; what are you actually trying to achieve? I think that a lot of times organizations – they want to do DevOps because that’s what the conferences and the experts say they should do. But also you need to understand: what’s your driver for outsourcing or engaging partners and other vendors, right? So if it’s a cost-saving mechanism, which often it is in this economic climate, is something that people are looking at, right? That can be one component. That can be one reason for outsourcing. You could also be outsourcing to get additional expertise that you don’t have internally. You could be outsourcing ’cause you’re scaling up with a whole another set of products and there are sort of HR reasons for wanting to do that. But you need to understand those drivers. 

    And I think another thing that Robina said crucially which I see organizations get wrong is: the organization who’s – the business, the incline for us needs to own the cultural change and needs to own the methodology. I come back to: no, you cannot outsource DevOps. You can’t buy that on a platter. You have to invest. You have to make the changes yourself as well.

    Laughlin: Yeah. There’s some parallels to that conversation, right? ‘Cause a lot time ago this was the conversation for quality assurance. Like: outsource it; there’s an easy button; all your problems will be solved. I think that, from the experience of that, for me there was never an easy button. It was a lotta work and it was a big partnership. But what we did was, through that partnership, we started bringing in technology that my team hadn’t had a lot of exposure to so our options grew, and I found that there were some easy buttons along the way to build, to run tests against that build. So we’re moving into a model now where the assets of quality assurance can be augmenting what is your DevOps model. And that model has to work for your company and it has to be part of your culture. It is not something that just kinda develop out of a box.

    Ashley: We probably need some precision around the term “outsourcing” too. So I think in this context – 

    Laughlin: That’s a good idea [laughs].

    Ashley: – we’re not talking about externalizing it. In other words, put it in a box, ship it somewhere, have someone else do it; give it to us when it’s done. Right? Because we’re transforming the organization. I think it can happen a couple of ways. It can happen early in the journey; I think it also can happen – organizations typically get stuck at some point where we’ve gotten 2, 3 teams there but man-scaling it to a 5,000 or 10,000-person IT organization – that’s hard. It’s easy to do it on a startup basis.

    So everything to me is always driven by what the company is about, what its DNA is like. Some companies are very data-drive: We don’t make a decision unless it’s got a graph attached to it or a financial analysis attached to it. Or tied to a strategy. Or whatever it is. And different companies often taken on the personality of the founders if it’s a younger company. But it gets established with it. And if you don’t have that figured out and aligned with how you’re implementing anything – agile, DevOps, SAFe, whatever it might be – you either go against the flow and fight it all the way and fail multiple times and never succeed or you figure out what fits into the flow and the momentum, the DNA of the organization. 

    That takes knowing people. It takes knowing the organization. It takes commitment by people in the company to do it. So it’s not an outsourced give away the problem. It’s a: I need help because I need expertise, because I’m stuck, because there’s other – we don’t know what we don’t know. And we’re looking to get smart from other people. And we want you to leave someday and help us operate on our own. Maybe help us with a different problem someday. But that’s kind of our goal as an organization.

    Gong: Yeah. Mitch, I fully agree with that. And we certainly have – a couple of points you point out is pretty interesting. One is the challenge that we seeing for large enterprise customers: they usually have problem with cascading kinda DevOps excellence across the organization, right? But there’re so many hurdles, I would say. One of them is organization siloes. We have seen so many times where you have one division, usually on the digital side, front end – they have great success because technology stacks tend to be more modern so it’s easier to get on the DevOps train. But then once you start trying to cascade to a legacy side, it just doesn’t work. Either because they don’t share very well because there’s organization siloes, or because the technology stack they develop doesn’t really work on the other side.

    So where we see, again – I know you have term about outsourcing can mean many different things. But where we see a lot of support or at least success is when we help kind of the organization where they have a COE, where they establish standards – and I think Robina kind of mentioned this too. It’s very important for company to really establish their true North, the compass, right? The guardrails and things like that. But it’s also important to not have a very heavy central organization either to trying to drive DevOps. Because then you’re missing out on all the distributed innovation happening on the ground level so you can never respond fast enough. 

    So this is why when we created our model, we usually have a team, or a very small team, help with the COE side of it to help them kinda develop their whole community, the culture, things like that. But then we have very nimble – we call it deep pod. It’s like a five-member crew. They have – consists of DevOps architect and couple engineers. And what they do is they will be kinda parachuted into an area within the enterprise where they need a lotta help or the application that they using is very critical to the business. Will drop in there. Not only helping with engineering, figure out the tech stacks and figure out the DevOps – all the nuances of it. But we also do a lot of coaching to make sure we can train the team to be self-reliant. Once they are, we lift the team up then we drop the deep pod into the next area where they need help. So this is kind of the hybrid model is where we find more success. 

    Ashley: I’m really curious: Who typically brings in your firm, Ming, your firm, Chris? Maybe, Robina, who brought in someone to bring in help with DevOps. Was that at a very high level? Is it a development team level? I’m thinking larger organizations here. What’s the –

    [Crosstalk] 

    Gong: I think Chris and I probably will share the same answer, which is: all of the above. We have a large DevOps engagement to where the CIO invites in. And we operate starting from the strategy level all the way down to the individual team level. But then we also have, through relationships or whatever case might be, where we insert ourselves into a particular team to help and then grow from there. 

    Rolls: Yeah. Exactly the same. And if it’s okay I wanna go back to one thing you guys said before, which is having a bit of precision around what we mean by outsourcing. Because I think when I see this topic, I jump to full, functional outsourcing of QA or dev or opportunities. But the reality is: outsourcing also exists on a spectrum. 

    You know, when you hire a maid to clean your house or to service your car or anything like that, you are outsourcing that function of your personal life. And it’s the same with software development, right? So there’s the really far down this end where it’s just you have a consultant who’s embedded in a project team, and then there’s all the way down to the far end where you do that full-scale outsourcing of a function or a department. 

    And what I want to be clear on is: the further you get along that continuum, the harder it is. You have to be far more careful about it. And so that’s something that’s really important. And to go back to the question that was just asked: completely the same. Sometimes – and you’re very happy when the CIO or the CEO says, “Come in and help me do this transformation across the organization.” Sometimes it is a QA or dev manager has a specific problem that they want to solve. But I’d also be really interested to hear Robina’s thoughts.

    Laughlin: So how did we bring – I’m sorry. Can we –?

    Ashley: What level did you bring? Was it the CIO brought in somebody to help you with the ____ or was it on a QA development team?

    [Crosstalk] 

    Laughlin: Oh, sure.

    Ashley: How did you start down the getting external help, outsourcing help?

    Laughlin: Yeah. So this is a little bit of a parallel run to a DevOps model. So from a QA perspective, what we did was we decided to centralize our enterprise quality assurance organization. And when we did that, certainly we got economies of scale, but I needed a partner or a vendor that would come and work with me. And when I say partner, I truly mean it. Like we have team meetings. They’re all folks that work for me. I don’t distinguish. So you have to be able to have that relationship I think to be able to get success out of outsourcing, if you wanna call it that. Because it’s really partner shipping, in my mind.

    So that was the journey that we took. And that’s been underway now for almost eight years. So we have an incredible wealth of assets behind us. And I started stopping and looking at what my partners were doing on the other side of building and deploying, and how are they verifying that building? We’ve been moving this shift-left journey of discovery of quality issues for a long time now, and we’re at the point where we’re in the build stage. And I’m like, “Well, we have great assets that could help verify that a build’s gonna work. What’s the next solution to that?”

    So I reached out to my Cognizant folks under my QA umbrella and said, “Hey, if we were to think about extending our assets use, what would we do?” And they were like, “Turn to your DevOps team. Turn to your build and deployment team. Let’s understand what they’re using. They’re using Jenkins. They’re using Bitbucket, example for us. So when we started looking at being able to scale our assets and use them across the enterprise, we built them in Jenkins. We built our actual Selenium code in Bitbucket. So now we have the hooks there and we can – as these teams start to catch up or we start to catch up and we meet in the middle, we then have a problem we’re trying to solve. We are getting builds. They are broken, or when they pass, there’s quality issues. Can we discover them through automation? So that’s the easy button.

    And in this case I actually do have the easy button. I have this regression suite. I can slim it down to a smoke test or something, and say, “If we can deploy this and do all of this then, yes, we can get those results.” And then the next phase of it is accountability of teams of what they’re gonna do with those results.

    So none of that is technical. That’s all cultural. Except for the hooks in what we built. So that’s kinda been our journey and that’s where I’m looking to go strategically.

    Shimel: Excellent. So, guys, I want to bring this down or – we touched on it already but I wanna clarify it. You know, Ming and Chris, you’re in a unique position in that you guys get to see dozens of organizations and how they approach this problem. Where, Robina, your view is Guardian-based, right? This is what you guys have done at Guardian.

    Here at DevOps.com I’m also – I’m in a catbird seat in that I get to speak to many people like Ming and Chris, but I also get to speak to many people like Robina. I get to speak to people like Mitchell. I literally get hundreds of views on this. And I recognize patterns. And what do I do? What’s my superpower, superhero power? I’m good at recognizing patterns. I guess I’m a machine learner. But anyway, there usually is a pattern along DevOps adoption. And it’s similar to what Robina mentioned, which is – when I first launched DevOps.com, our tagline was “DevOps picks up where agile leaves off.” Right?

    And, yeah, because what happened was people were moving the agile development processes probably going on 20 years now. Agile’s been around 20 years. And it was great. It makes you feel old. I’m sorry, Robina. But, yeah, it’s 20 years [laughs]. Don’t feel bad. I watched this “Halt and Catch Fire” about the birth of the PC industry. Yes, that’s what it is. Einstein’s theory of relativity [laughs]. 

    Laughlin: I remember my agile training with Ken. I can’t imagine 20 years –

    Shimel: Yeah. Ken, who now of course is Scrum.org, right? Versus Scrum – the other one. Not here to get into that one. That’s another rat’s nest.

    Gong: Different podcast [laughs]. 

    Shimel: Yeah. But the thing is: the normal course, if you will, is: yeah, we do some agile. Whether it’s Scrum or SAFe or whatever we’re seeing out there, scale. We wanna start getting into some lean IT stuff. And of course when you start marrying agile and lean, it’s DevOps, right? That’s kinda what DevOps is. And now we’re gonna get into CI/CD pipelines. And of course it’s grown since then. We’ve added SRE on. Continuous testing has become huge. Right? It’s a huge piece of this. So we’ve seen the sprawl, the urban sprawl of DevOps and agile over the last 20 years. Along with it, I think it’s become all but impossible to do at a large organizational scale, at a large enterprise scale, without some outside help. 

    I think we’ve recognized the enemy. It’s us. We’ve created the need in our large organizations, the way we go about adopting these things, that we can’t do it without outside help. 

    Laughlin: That’s very true, Alan. I think that’s why you really need the outside help and you need to go to the experts. Because no one size fits all. No one tool solves everything. That’s how I got involved with Tricentis: I went out really early looking for a vendor that would grow with me and was on the same aspirational journey that I was, right? So that’s where I think it really helps in terms of consulting. You need to pick the partner that’s gonna work with you and grow with you. And sees you for your uniqueness, not an industry standard.

    Gong: That’s right.

    Shimel: Chris, Ming? 

    Rolls: I think you can’t take a single ___ of technical practices or cultural adoption and apply it to every single organization. So that point, Robina, about making sure that you tailor it for the organizations that you’re working with – or even within an organization, the different departments, the different products that you’re working with, right? There’s a different sort of cultural mode within the digital section versus the enterprise technology section, right? So that’s really important. 

    And I think that also comes to a thing that I see that often people get wrong when they do outsourcing is that they don’t align the incentives properly. Because ____, we all have different incentives. Even, Robina, if I was partnering with you and we want to work as really good partners, then I have a slightly different set of incentives to grow my company as Guardian does, right? And so we have to find ways to align those incentives. Otherwise you will create a cultural conflict which prevents you from really going as fast as you can. So that’s just something that I see often people get wrong about moving to DevOps, because they have to use outsourcing in particular, engaging firms to help them with that.

    Gong: I wanna go back to the point Alan was talking about in terms of the massive talent shortage in industry. And it’s not just DevOps. It’s cloud, everything else, right? Just in general I think we all wish there’s more qualified folks who has those technical skills. And I think that’s one of the reason why a lot of our customers want some help from us. Because they simply cannot find enough talent out there that not only knows the DevOps technology but also has the ability to keep up with a crazy changing industry, right? I mean, you see a massive change even within the last couple years where used to be all the siloed best-in-breed product like Jenkins or whatever else, and then artifactory, whatever. 

    But now you see a trend where industry’s moving towards more of a kinda end to end all in one type of tools, right? For example, you have the Azure DevOps, GitLab, and the whole – they start offering the whole suite of things. Because they recognize the massive talent short. You can’t take care of the problem until you can also solve the problem on technology side as well. So that’s kind of approach that we’re taking as well, where not only we have on people side we have our deep pods kind of coaching the team so we can get lots and lots of teams to train up to be self-reliant. But on the other side we also develop on platform to kind of try to extract all the complexity of DevOps from the user so the user doesn’t have to be an expert. So we call it democratizing DevOps for everyone, right?

    So I think for a company to be successful they need to kinda take approach of kinda both people and the tools combination. 

    [Crosstalk] 

    Ashley: ___ kinda just share a personal story in my additional adoption of DevOps when I was a CIO at a company. And it was initiated by me ’cause actually Alan had reached out to me and said, “Hey, Mitchell, there’s this thing called DevOps. You gotta go read about.” I’m like, “Okay, what is it? I’ll go figure it out.” I got hooked onto it.

    But the rest of the team hadn’t really caught fire yet of getting onto DevOps. And so as a leader of an organization, it’s tough to say, “Okay, I want us to go this direction; I think this would really be helpful for us” when the team’s not there yet. And really understanding what it’s gonna take to get the organization going and get behind it and how to be successful at it. This was earlier, so you reach out more to individuals that’ve started to build some brand and some experience in doing things like DevOps.

    Today I would go to a different kind of a team to say, “Here’s where we are. Here’s the projects that we have going on. Here’s kinda what we’re being asked to do,” right? I think every – maybe not true, but it seems like most IT organizations are always being asked to transform some way, somehow. And sometimes pretty radically. And if you can sort of set where we’re headed and tie it to the why, as you talked about, Robina, the why for the business though, not for IT, that seemed to me to be the biggest thing that everybody could rally around. Not that it was DevOps or agile or it was this. It was: “We wanna go after this market. And our ability to get there is really constrained today. And let’s figure out some ways we can go after that.” And that tended to rally around.

    So I think today – I guess my point is: Today we’re at a point where we can hire more expertise. And we wanna de-risk our journey and learn from others. We don’t have to figure it all out ourselves and be the smartest person in the room. Certainly I don’t. And that’s a great environment to be able to bring in help.

    Shimel: Robina, I’d like to throw out – there was a time when we used to look at quote/unquote “outsourcing” and consultants, and one of the terms that was thrown around was “body shop,” right? Like, “Bring in some bodies. I’m gonna rent some bodies.” I remember when Mitchell and I were at StillSecure. We’d go down to – we had a big federal practice, and we’d go to a federal facility, whether it’s a lab or a knock or what have you. Eighty percent of the people there were not federal employees. They worked for Grumman Northrop or SAIC – take your pick of the beltway bandits. But it was all body shop. And when you’d speak to the salespeople at these consultants, you’d say, “How’s business?” “I got 50 people placed.” Not what the mission was, what the tools were. They judged their success by how many people they had placed.

    I like to think – and maybe I’m naïve; Robina, I’m gonna ask you – that we don’t judge the success of our DevOps outsourcing by how many bodies I’ve got buried at a given location. ‘Cause I think that’s a terrible metric to determine the success of a transformation.

    Laughlin: Yeah. That’s a great observation, Alan. You just kinda gave me shivers when you talked about number of bodies. I remember those days. They’re painful to look back on, honestly. There was a time when you needed an army of folks to do things and I don’t necessarily think that’s the case now. For me, our journey was: I needed a partner and I needed someone that was gonna come here and work with me. And through that partnership I also needed to know that I was gonna have a 24-7 shop, and that’s gonna necessitate overlap on both sides, right? Your onshore and your offshore. So I don’t think you can ever just kinda set it, forget it, and throw it away to an offshore team. That’s just not how it works.

    And I think the other piece of it is: at the very source of it, you have to remember that these are human beings. They have families. They have lives. They wanna do good work just as much as we wanna do good work. And the way that they’re gonna feel that contribution is being part of the team. And the way they’re gonna feel that sense of that is by being present. And we actually flipped the dynamics. So we’re no longer a headcount shop. We’re an outcome shop. So I measure the success of my organization by the outcome of our testing. And for me, with my partner, if they do it with 2 people or 200, that’s their discretion. So I don’t have to get into any more of these conversations of: “We need two more bodies here. We need eight more bodies there. I’m anticipating 12 bodies.” That just turns into I think an empty use of really valuable strategic time that you have with your partner. And you should be using it a little bit more productively, right? 

    So that’s how I flipped that dynamic. And I’d be very curious to see how Mitch and Ming feel about it. And Chris.

    Ashley: Well, my first reaction was I thought you were gonna talk about “Sopranos” or something. A whole different kind of body [laughing].

    Shimel: Bada bing.

    [Laughter] 

    Ashley: You know, I really like how Robina described it. Because it isn’t about budget size and all of those kind of backwards metrics. And I’m sure they’re still used today. To me that’s a measure of how aligned you are with the business. If you’ve shed the financial size, the IT-only-centric metrics to “What are we making an impact?” – there’re a lot of ways we can do a digital transformation project. We could get the contact list services let’s say. Something very relevant today. And it doesn’t have to be with hiring 50 more people ’cause we’re probably never gonna find 50 people to hire anyway. So we’ve gotta find other ways to do it. 

    But there’s also – there are times where I need these people to fill these roles. And I’m sure the body shop approach still applies in some situations. I doubt it on strategic projects though. That would be my supposition.

    Shimel: Chris and Ming, I’d like to hear from you guys on it though. Because you’re on the other side of the equation here. Chris?

    Rolls: I hate the term “body shopping.” It’s a terrible, terrible term. And I talk to the team and I say, “We have to have stamped on our forehead, ‘We are not a body shop.’” Because if you’re going in there and your goal as a account manager is to get 50 people on an engagement, you’re not thinking about what that organization needs. You’re not thinking about what the business outcomes are. You’re just thinking, “I can make $10.00, $15.00,” whatever how many dollars it is off each person, “and what’s my commission for that?” That’s a terrible way to think about it. And what’s more, it shows an outdated way of thinking about the people who are going to do this work. People, process, technology – people, in my mind, being the most important part. 

    If you’re thinking of them and you’re calling them “resources,” which sometimes we all slip into, or you’re calling them “bodies” – that’s not just a warm body in a seat typing some code or testing some software or monitoring the systems. They’re people who have skills who are thinking about things who are trying to do the best work they can to deliver outcomes for the company. And so I hate that term “body shop.” I hope that no one associates that with me and my company. And I think it’s a terrible way to think about delivering software.

    Gong: Yeah. I definitely second that. I mean, what Robina just articulated pretty much echoes what pretty much all of our enterprise customers have said. They should absolutely transition from where it’s just TNM type of model before now to it’s all about outcome-based. And that’s what we prefer as well. To Robina’s point, why the customer should care how we deliver it as long the outcome is what they want, right? 

    We actually created a digital agility transformation manifesto [laughs] in honor of agile manifesto. And one of the top one is actually called “Value over volume,” which is exactly what, Alan, you just talked about in terms of, hey, you should not be measuring success with transformation by the volume of things got done. But it’s more about the value, right? So in this case, instead of measuring, for example, how many DevOps ____ did we create – that’s not important. What’s important is: how much faster are we able to help the team to deliver software to the production faster? What’s the frequency increase? So those things are what matters. So that’s what we absolutely focus on.

    Shimel: Excellent. Guys, we’ve got maybe seven or eight minutes and there’s one area we haven’t hit on yet that we’d be kinda negligent if we didn’t hit on it, and that is: what effect has COVID had on all of this? On one level, we could say, “Well, look, everyone’s looking remotely anyway. What difference does it make if they’re an employee or an outsourcer? We’re all remote. We’re all equal in that regard.” The other way is: “How do I graft these people into our culture? How do we make them part of the team when we’ve never shook hands in person, when the only way I ever see you is on a “Brady Bunch“ Zoom window here and you take the top left?” Right? “Can we really integrate in just remote-only settings?”

    Ming, Chris, you guys deal with it. You’re dealing with it over and over again. Robina, you are too. Right? Everyone’s remote. How has COVID helped, hurt, hindered, accelerated – how’s it affected this?

    Rolls: So, I think one thing is that if we asked people five years ago, “How do you do agile? How do you do DevOps?” you’d say, “Everyone has to be in the same room. They have to be collocated. You’ve gotta have stickies up on the wall.” 

    Shimel: Chris is kinda sticking on us here. Okay. Go ahead, Chris, I’m sorry. You kinda hung on us. We got stickies on the wall.

    Rolls: Yeah. So I was saying in the past people would say, “You have to have everyone in the same room, stickies on the wall, and that’s the only way you can do agile and DevOps.” And I think one thing that has been good with COVID is that it has dissuaded us of that illusion that that is the only way to do things because we have been forced to do it differently. That being said, it is so much harder to create culture, whether it’s in-house or whether it’s outsourced, when you don’t have that face-to-face collaboration. There’s only so many times people can do a virtual happy hour and still really get into it. So you have to be – I’m sure first time everyone’s like, “We’re having five happy hours a week. I’ve got my bottle right next to me.” And you’re two months in and you’re saying, “Geez, I cannot do this anymore. I just wanna go to bed or watch “Sopranos,” right? 

    But I think that we really need to focus on building that culture and come up with creative ways to create connections between people. 

    Shimel: Agreed. Ming? 

    Gong: Well, let me see. Hold on. Give me a sec. 

    Shimel: Okay [laughs]. You going to your happy hour background?

    Gong: Absolutely.

    [Laughter] 

    What you’re seeing here is the – my family and I, we took a hike recently to a Malibu beach, and I took a 30-second video of Malibu and use that as my background. So this is one of the things that I started asking my team to do: trying to personalize this kinda virtual experience. I’m a absolutely people person. So I miss the in-person bonding dearly. But in lieu of that, we basically, every time we have team meetings with our customers or with our own team, we pretty much strongly encourage everyone to get on the video so at least we can see each other. But to Chris’ point, there’s always a silver lining to everything. Because the COVID certainly forced or speed up a lotta things that was gonna happen anyways. So, to that, I think it’s a positive thing. Robina? I’d love to hear your thoughts [laughs]. 

    Laughlin: Sure. What a year we’re all having, huh? Surprised. So, I think in terms of working with your partners through COVID – and we’ve been talking a lot about culture – I think one of the foundation things is to pick a company or a partner that has a very similar culture to you. So then you’re not trying to find middle ground in that. And that should really be a big part of your selection process. You need to be comfortable with the folks that you work with and you need to be living kinda to the same standards I think. 

    So for me, how COVID affected me was: while we were all scrambling to get ourselves home and reaping the benefits of a long-term plan to be able to have a remote workforce – we learned that through Sandy a long time ago and we had done a lotta changes – I was also working really closely with Cognizant in helping to get all the resources that they needed to get my folks home at offshore, right? So I was as concerned about offshore as I was concerned about onshore. 

    And now to this day my job is actually pretty high-touch now. And the reason being that we’re so far along, I have people that’re really overworking themselves. And I think that – and I don’t know that it’s deadlines as it is: take pride in craftsmanship, and they’re home and they wanna – this is something for them to do. Or it’s a stress release in a really challenging time where everything’s unpredictable. But I spend more of my time saying, “When was the last time you took a day off? When was the last time you took an afternoon off? You need to recharge. You need to gather yourself.” And that’s for offshore too. So that’s been my challenge this year.

    Shimel: Fair. Mitchell, I know you guys at Accelerated Strategies Group did a study on the whole remote thing and COVID. Thoughts? 

    Ashley: Boy, I could talk for an hour on this. But I’ll just point out a couple things and I’ll point folks to the report. We looked at – software developers already know how to work remotely, right? A lot of them do. And so how has that affected them? How has COVID and the work from home affected them and how have they kind of affected the organization?

    So, interestingly enough, across the board – there’s about 350 participants. Across the board, everyone is above 60 percent of who have accelerated digital transformation projects, contact – new forms the delivery and automation. But what I really found interesting was how influential DevOps practices, whether you called it DevOps or not, had started to infiltrate into the organization. So I’m talking about things like doing daily standup meetings or more cross-functional teams, increased use of that, asynchronous communications like a Slack or a team as opposed to being on a phone call or in a meeting. So there’s all these ways of working that, as we were talking before, sort of forced everybody into it. And the teams that had really good software teams working remotely jumped on board. And you saw an increase in the use of that kind of style of work, sometimes facilitated by tools, sometimes just process. 

    So if you wanna read the whole report – and reading is minimal; it’s all infographics; it’s super easy to consume – go to ACCELST.com. And it’s free to download. And enjoy. And let me know if you have questions. Love to talk to you about it.

    Gong: Awesome. 

    Shimel: Excellent. Guys, we are outta time. We’re actually over time. Robina, Ming, Chris, I wanna thank all of you for joining with us today. This was a great discussion. In the show notes, production notes, we will have links to your websites and any other links you wanna put in there. But all three of you are dealing in the real world with real problems that we’re seeing today and helping organizations solve ’em, whether it’s your own organization or other organizations. So keep up the great work. We may revisit this topic on DevOps Unbound someday, and we’d love to have you back on here. Okay?

    Gong: Pleasure.

    Shimel: Thank you so much. 

    [Crosstalk] 

    Laughlin: Nice meeting you guys. Good luck. Thank you.

    Shimel: All right. Mitch, thanks as always, man. This is Alan Shimel for DevOps Unbound. Thanks for joining us. We’ll see you on another DevOps Unbound real soon.

  • EP 4: DevOps and the Mainframe

    EP 4: DevOps and the Mainframe

    In this fourth episode of our DevOps Unbound streaming broadcast on TechStrong TV and DevOps.com’s sister site Digital Anarchist, Mitchell Ashley of ASG and Alan Shimel are joined by Gerta Sheganaku of Tricentis, Rosalind Radcliffe of IBM and Sam Knutson of Compuware to discuss app modernization, mainframes and DevOps. Find out what’s the right way to modernize your applications and how to do app modernization in a Z environment.

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript

    Alan Shimel: Hey, everyone. This is Alan Shimel. Welcome to another episode of DevOps Unbound. This is episode four of DevOps Unbound, and then we’ve also done two DevOps Unbound roundtables. So this is actually the sixth DevOps Unbound event that we’re presenting. They’ve all been great, and I think this one is gonna be as good or better than any of ’em.

    Today’s topic is app modernization and mainframes, along with DevOps, of course. It’s DevOps Unbound. We’ve got a great panel lined up for you. Let me introduce you.

    First of all, we have Sam Knutson, and Sam is the product manager — is a product manager for mainframe and agile DevOps at Compuware, a BMC company. Sam, did I get that right?

    Sam Knutson: Pretty much, Alan.

    Shimel: Correct me. Get it right for the record.

    Knutson: So I lead product management at Compuware. We are a BMC company, and we are focused on enabling agile and DevOps on the IBM z/OS platform.

    Shimel: Fantastic. And the lesson I have learned here is I’ll let each of you introduce yourselves. My next guest is the one and only Rosalind Radcliffe. If you want to know about DevOps and mainframes, there’s no better person in the world to go to.

    Rosalind, welcome, and why don’t you give them a little bit of your title and background?

    Rosalind Radcliffe: Thank you. Glad to be here again. Rosalind Radcliffe, IBM distinguished engineer, responsible for DevOps for enterprise systems. I help clients with their transformation to DevOps for z/OS applications as well as application modernization and bringing z/OS into the rest of the world so you can do your cloud-native development for z/OS as well.

    Shimel: Excellent. Then we have – I’m gonna do my best here – Gerta Sheganaku of Tricentis. Gerta, how bad was that?

    Gerta Sheganaku: Thank you, Alan. That was perfect, really.

    Shimel: Okay. Gerta, your background and title?

    Sheganaku: I’m with Tricentis, as you said. I initially started at Tricentis in the customer experience space, where I was leading our customers through their digital transformation journeys, really helping them understand how they can enable testing as a catalyst to really get DevOps working.

    Then I have moved into the product organization, where I’m currently leading a team where we’re releasing our first AI-based test automation solution, which is really about tackling the problems that we see out there, especially when it comes to app modernization, and testing of old applications and new applications and really making testing easier.

    Shimel: Excellent. And then last but not least is my sidekick, well, partner in crime, Mitchell Ashley, CEO of Accelerated Strategies Group and also my co-host here on DevOps Unbound. Mitchell, good to see you.

    Mitchell Ashley: Good to join you.

    Shimel: All right, guys, today’s topic is app modernization, mainframe, and DevOps. It’s interesting. I’ve been kind of binging on Netflix that AMC show, oh, “Catch Fire”? I forget the name of it now. Anyone else watch this show? It’s an AMC on Netflix.

    Knutson: “Halt and Catch Fire”?

    Shimel: “Halt and Catch Fire.” You’ve watched it, Sam?

    Knutson: I’m familiar with the instruction. I’ve seen the previews for the show.

    Shimel: It’s a pretty good show once you get into it. So the company that started in Texas bets everything ’cause they’re tired of paying the timeshare people, and they buy an IBM, what is it, 3030, that’s based in Daly, San Francisco, or Daly, California, near San Francisco. They literally have to move the whole company to where the mainframe is versus moving the mainframe to where the company is. Well, it’s TV. But that was the late ’80s, right, and we don’t live in those worlds anymore.

    Mainframe applications are running on your phone, right, or at least the front ends of them are, on your tablets, on your computers. The mainframe doesn’t necessarily fill up – this was a whole basement that was one mainframe. It was pretty funny. The guy was sliding cards in and stuff like that. If you get a chance, watch the show. It’s an interesting, interesting look back.

    Certainly the modern mainframe has come into the world of DevOps. Rosalind, I think the work you have done with IBM has led the market with that. Quite frankly, Sam, the work Compuware has done has led the market in agile and DevOps on mainframes. How would you assess where we are currently in mainframes in the world of DevOps?

    Radcliffe: You mean where the mainframe is or where people are? I think those are two very different things.

    Shimel: You’re right. But what we’re not moving the whole company to the computer anymore, thank God. But let’s talk about where is the mainframe, and then we’ll talk about where the people are.

    Radcliffe: So you’ve got to remember, the mainframe itself, IBM Z, is the most modern machine out there. The IBM Z platform has evolved every few years for the last – ever since it was invented, and we keep evolving the machine.

    So the machine itself is the most modern system, and we’ve evolved the operating system. We’ve evolved things such that open source tools work there. We’ve done everything to remove the barriers from the system so that you can do exactly the same kinds of things, whether or not you’re doing z/OS development or another environment. And the Z hardware itself, we all know it runs – well, we all know on this call it runs Linux on Z.

    And so it’s a very large Linux server. It runs containers, and it runs OpenShift. So from a systems standpoint, it provides – there’s nothing keeping it out of the processes and practices. There’s nothing separating it from a systems standpoint. It’s the way people interact with the systems. It’s their existing applications that make it harder.

    Shimel: Sam?

    Knutson: I think Rosalind makes some great points. The current generation of IBM mainframes, the z15, is an engineering marvel, and the compilers and the operating system produce this amazingly efficient platform. It is the most efficient hyperconverged system in the world today.

    But I think the elephant in the room is the word that you used a minute ago when you said “modernization.” We alluded to this a little bit in the way that the practices that people and businesses have today. Well, modernization doesn’t mean moving off platform, because we already talked, and we could go on for an hour, about how great a hardware and software platform this is for running business computing. It is amazingly efficient, especially when you look at the large-scale, global business applications that the Fortune 1000 run, which is where the platform has its largest footprint.

    But the perception is that modernization is moving your application into a cloud-native application off the platform, hosted by another provider, or rewriting it in Java, and nothing could be further from the truth. The COBOL compiler alone is a perfectly paired piece of engineering with that platform, and it lets you wring every last ounce of efficiency out of it.

    But people need to get past that confusion and say, “What if I could work on the platform in the same way I’m working on every other platform?” It’s that normalization. We’ve kind of coined the phrase “mainstreaming,” “mainstreaming the mainframe” to make it different in syntax only, and Rosalind alluded to this. It’s being able to work on this platform in the same way that you work with any other platform. So when you think about DevOps, then you have a DevOps toolchain that looks like other toolchains, that you can connect all of the ecosystem partners that you’re probably already using in distributed development.

    People are definitely the problem, people and practices and culture. More than three-quarters, in fact 78%, of customers say it would be useful if they could update their mainframe applications more frequently than they do today. So there is an aspiration to change and to get better outcomes. You can see what they want. They want to be able to deliver value to customers more frequently, and that’s just one of the improvements that they want to make. But far too many people are not making progress on that.

    Shimel: Gerta, so you’re not a mainframe specialist, but certainly app modernization and testing of this. I’m wondering if you have any kind of information regarding people using Tricentis to test mainframe-based applications. How big a market is that?

    Sheganaku: That’s a very good question. One major thing that you always have to keep in mind, as we just said, especially when talking about app modernization, it is such a wide word, and it can mean so many things, too, to different people. It’s definitely something that’s challenging for our customers looking at their current system landscape. You see customers that are having everything in their landscapes from mainframe applications to also often very legacy mainframe applications that are trying to move onto more cloud-based environments. That is something that we see very often.

    Mainframes are still there, but the thing is that people really want to abstract away from the mainframe. Also thinking about where we are today, I think an important question we have to ask ourselves is why is everyone so into app modernization? Why is everyone into migration into the cloud currently?

    If you think about it, nowadays every company basically is becoming a software company, and not every company has the talent and the resources to maintain those mainframe applications because at the end of the day, honestly I struggle finding COBOL developers nowadays out there. I went to the Technical University of Vienna, and we definitely did not learn anything around COBOL there. It’s really hard to find the talent nowadays.

    When you look at it, basically every company, be it in retail, be it in banking, be it in the energy sector, you now want to have connected systems and so on. Really, everyone is going more towards being a software company. And with that, for people it’s becoming more and more increasingly important to have really good platform-as-a-service offerings, maybe going into serverless or software-as-a-service in the next step, but really having not really to deal with the underlying maybe mainframes and having a way of abstracting and using simpler technologies and only caring about the applications and the data that they’re running on those systems.

    Shimel: I’ll be honest with you, I don’t know if I agree with that. So here’s the premise of this, right? When we talk about app modernization, I’d love to live in a world where I could throw out an old app ’cause I want to have a new app. But we don’t live in that world, right, and no one lives in that world. That’s why we call it “app modernization.” We tend to modernize the apps we already have.

    And so when we look at – I think out of the Fortune 100, 98 are on mainframes. Or it’s some number like that, Sam or Rosalind, if you know. I don’t see 97-and-a-half of them throwing their mainframes out, right? As a matter of fact, they’re probably building more on mainframes. And so what’s happening is we’re seeing whether you want to call it hybrid cloud, VoLTE cloud, whatever you want to call it, we’re seeing where people can build maybe a front end for stuff in the cloud. But if that data is important, if security is important, if scalability is important, dollar for dollar, people who use mainframes know that that’s their best bang for the bank. Mitchell, I’ll let you chime in.

    Ashley: I have to jump in. So what’s interesting is we kind of do this to ourselves with the terminology that we use. “Mainframe” is a word for a cabinet used to house the computers the original IBM computers were stored in.

    If I said, “Hey, 71% of Fortune 500 companies rely on high-volume transaction-processing platforms. Ten out of the ten world’s top insurers, 23 out of the 25 world’s top airlines, 92% of the top 100 banks rely on high-volume transaction-processing platforms,” you would say, “Wow, that’s pretty cool.”

    “And it runs Linus, and it runs containers, and you can connect it to all kinds of applications from tablets and phones to wherever your application may be running, on cloud or on premises.”

    You’d say, “Wow, tell me about that. I’d like to learn about that.” That’s what mainframes are.

    So it’s interesting. It’s not going away. We do this to ourselves, and I say the analysts have done this to us, too, right? Today it’s all about this next shiny object, the squirrel moment. It is so vital to businesses today, and moving it into how we create software with DevOps and processes like that and tools to support it is essential for businesses’ both economic recovery, survival, and future. Digital strategies can’t happen without it for many, many, many companies.

    Okay, did I do a good job, Rosalind and Sam?

    Knutson: Mitch, those were great stats, and I think you did a great job quoting ’em. I’d boil that down to simply saying the worldwide economy and worldwide civilization runs on the mainframe. And I like the word “mainframe.” It has a little swagger. I don’t think it’s ever gonna go out of style.

    Ashley: Nostalgia.

    Knutson: And renaming the platform could make it cooler. It simply gets things done. We haven’t renamed cars yet. Well, maybe we’ll rename them to “highly efficient transport platforms,” but I think we’ll still be calling ’em “cars” when they have electric engines powered by portable fusion reactors. But customers see this as a long-term platform. BMC does an annual mainframe survey, and the 2020 edition is gonna launch soon. I can tell you that 90% of the customers see the mainframe as a platform for growth and long-term applications. Sixty-eight percent of ’em expect the size of that compute platform to grow, MIPS to grow. Sixty-seven percent of those organizations have the majority of their data on the mainframe. They trust it because, as you said, it’s highly available, scalable, secure.

    And to go back to Gerta’s key point about younger workers, that is absolutely essential. But what’s going on is that people are addressing that because – I think Rosalind and I both alluded to this – the syntax of COBOL isn’t Aramaic. It isn’t some incomprehensible script.

    If you put modern tooling around this, if you put modern process and a great, supportive culture that gives that great – you have a great developer experience, you have a great developer culture, then learning the syntax of COBOL is no more difficult than learning the syntax of Perl or Rust or Go. So we see that customers who see the platform long-term are making the appropriate investments. In the responses to that survey I mentioned this year, younger mainframe workforce definitely figures in, 43% less than five years on platform. Sixty-eight percent were under 49 years old, and 60% of that new generation see this as a growing platform.

    That really matches the experience that I’ve seen in Compuware’s journey is that we hired out of college and we have been hiring out of college, and those next-gen developers are more than a third of our workforce now in Detroit. Many of them did not know COBOL, and many of our applications are written in assembler. They certainly didn’t learn assembler in college. But what they did have was a strong bent for learning new technologies. Many of them came with computer science degrees. They had learned how to learn. They were what Gartner calls “polyglot programmers,” one, two, three, many in learning languages.

    So whether we needed to learn assembler or COBOL or C, it didn’t matter as long as we provided that supportive culture, the vehicles for learning, and we gave them meaningful work early on in their employment and made sure that they understood the vision, mission, and goals of the firm and how they were contributing to something that matters. I think our customers are having similar success in onboarding next-gen developers by doing some of these same things.

    Radcliffe: I think it’s interesting, the statement, “I can’t find COBOL developers,” because when I was hired in to work in this industry, nobody assumed you knew the language or the system you were going to work. That was the thing you learned. You learned how to develop, and then you came in and learned the company’s culture and the company’s tool choice.

    By using modern tools, modern practices, all of these things, the things they have learned in college on how to do, they’ve already got. And then the only thing they’re doing is learning COBOL.

    As I always joke, it’s just English. I know that works really well in the US and in parts of the world, and in other parts it’s, “Okay, it’s still a foreign language.” But it’s not a difficult language. It’s easy to understand. PL/I, it’s not difficult to understand. It’s just another language, and developers can learn other languages.

    I am kind of with the discussion about “mainframe” is not the word we should always use, and we really need to focus on how do we talk about our platform. I like the term “mainframe,” too. I still use it. But can we talk about our platform as this platform that provides this valuable, high-transaction server? It’s that big server as part of your pipeline. It’s not that old thing. It’s that big new thing that runs all this work, and now let’s focus on the applications that – Honestly, the best thing IBM ever did was a program you compiled in 19-whatever and still works today, and the worst thing IBM ever did was that program that you compiled in 19-whatever and still works today. It’s the best and the worst of the platform. You don’t have to change. It still just works.

    And if we can help companies understand if they start to change some of those, they can do the automated testing. They can do all those things. They can get more value out of their platform. They can help refactor and be able to move at the speed they need to.

    Shimel: Go ahead, Gerta.

    Sheganaku: These are all amazing points. I hope no one got me wrong at the time beginning with what I wanted to say. I really like what Rosalind just said, to talk about the underlying platform that is really enabling. But the point that I really wanted to make is that most companies are now becoming software companies. We don’t have that knowledge center of IBM, where there are so many smart people and you can learn from people and really develop that amazing platform that you offer other companies.

    But when I look at our customers, it’s usually self-built kind of mainframes that are sitting there that are not maintained that don’t apply current recent security standards. Those customers, what you see them do often is they all hear, “The cloud is the next thing. We all have to go to the cloud.”

    They all do this one big mistake that they just take whatever they have and they basically do the lift-and-shift approach, bring it to the cloud, and hope that everything’s gonna work. Of course that’s not gonna work at all. Also from a cost perspective that is insane, thinking about how much cloud resources actually cost you if you don’t do all the optimizations and so on. So what you see people do is kind of a step-by-step learning approach where they are taking things from not-so-secure systems, bringing them over to other systems, then starting to refactor the applications, and giving more and more also responsibility to the development teams.

    Once they do that, you see the development teams really getting that DevOps approach, really caring about the infrastructure, and so on. For the developers, of course it’s easier if they have container-based applications that they can simply deploy and so on. But of course underlying it’s very, very important that you have a secure and great platform that is enabling all of that.

    Shimel: I think that’s a great case for why you’d want to use a mainframe or whatever you call it, ’cause it’s probably more secure than the cloud is, I think, and it’s been proven. But here’s an interesting thing I’ll tell you here I see in DevOps.com, right? This is DevOps.com, mind you. And it was surprising to me. Our biggest article last year on DevOps.com, most views, most comments, “The Beauty Of the COBOL Programming Language” by Bob Reselman.

    I don’t know, it had half a million or something, maybe more, a crazy amount of views. Most of the comments were by COBOL programmers in India because there they recognize that there’s such a need for COBOL programmers that you have a lot of young people who are studying COBOL because they know it’s basically a guaranteed job for them, right? They come out and they start working right away.

    Here’s another interesting thing, and my friends in Tricentis will notice – will appreciate this. What do you think was another huge area of interest in DevOps.com? Running SAP in a DevOps world, right? We don’t think of SAP as DevOps-enabled or maybe not even agile.

    We did a roundtable a week or two ago on SAP in DevOps. I had to cut the questions off, and these were meaty questions: “How do I do this and that in this and that release?” It wasn’t just pie in the sky. They were very how-to, practical, “We’re running SAP, and we’re moving towards agile and DevOps and we’re doing these things.” So while everybody – we always rush to the shiny new trinket and the hot thing in our industry, I think the fact is certain things endure. As Rosalind said, it’s both their greatest and worst attribute. It is.

    But let’s talk a little bit about app modernization ’cause I think Gerta alluded to it. Mitchell, I’ll throw it out at you. No one wants to do shift and lift or lift and shift or whatever. It doesn’t work in 2005 when the cloud first started, and it doesn’t work now. But how do we do app modernization right in a Z environment? Sam, Compuware has made a – it was the reason for your being in a lot of these last couple years, right, something Chris O’Malley always harps on. How do you do modernization right in a mainframe world like this?

    Knutson: First, you do it on platform because – I can’t say that enough – when you see people try to avoid touching the mainframe apps that live on the mainframe, they do unnatural things from an architectural point of view, and they create these horrible kludges that are not robust, that are not scalable, that are not the most efficient implementation, and that aren’t easy to maintain. So you want to bring those practices on platform. You want to provide developers with helpful editors, with quality tools. You want to tap into the automation pipeline. So build a CD pipeline that goes right onto your pipeline so that you give developers a great developer experience, but you keep the business logic and the data on the platform.

    That goes to a number of things. When you talked about security, something that I learned a long time ago, the more copies of data you have, inherently the less secure it is, because typically when mainframe data is moved off the mainframe, the security controls that are on the platform are not replicated.

    So if you can take that core business logic, that digital DNA that a business has inscribed over 20, 30, 40, 50 years, continue to leverage that, and of course leverage systems off platform. Compuware really blazed this trail of using cloud and mainframe together with two-platform IT. We think these things are incredibly complementary. But what you don’t need to do is replicate that data all over the place. You don’t need to necessarily have a lot of on-prem systems. The term “hybrid cloud” was used, and that’s a nice, modern way to talk about that. We say “two-platform IT” because these things will pair very, very well.

    And the cloud, by the way, is an absolutely great place to host commodity applications. If you talk about your company’s e-mail system, I probably cannot run Microsoft Office better than Microsoft can with Office 365 in the cloud, and they can provide all the quality, the service, the latest updates. It’s a good solution. Can I do invoicing better than a cloud provider like NetSuite or Oracle? Probably not.

    But what I want to do is focus on those applications that give me unique business advantage, the things that I wouldn’t find in Amazon, Google, or Microsoft’s application catalog. Those applications give me a competitive advantage, and in updating them I increase that competitive advantage so I can provide new features, new capability. I want to take that on-platform application. I want to bring to it things like automated testing. Make it part of a CD pipeline. Give my developers a great set of tools and give them a great culture of development to help them help the firm be successful.

    That is the best approach to modernization I know of, and of course I work very hard to provide those things, including program understanding, so that developers can really get into code that they may not be familiar with, because when I talked about a younger mainframe workforce, that’s a good thing, but it also means that I don’t have people who have apprenticed with an application for ten years before being allowed to do important changes.

    You have people who are new to your firm, and they’re being tasked to do important work. So they need to be able to understand an application that may be older than they are quickly enough to inside a two-week agile sprint make a change, test it. Hopefully there’s already automated test present, but if there is not, they want to write some new automated tests and deliver that all as working, demonstrable code at the end of that sprint because that’s the way the world works today. So if you do that, you’re blazing down the road towards application modernization. It’s easy to talk about, but it boils down to just hard work.

    Ashley: Alan, I think to build on what we’ve said here, I look at it as three elements to the strategy here, and this is just also sharing my own personal experience with doing this. First is the people and the culture and getting our heads around new processes and new ways or additional ways of doing things. That’s what DevOps is about.

    Second is processes and tools, which Sam was just eloquently talking about. I think the third obstacle to app modernization is tackling the monolith because a lot of our applications were built differently, that we’re looking at how do we modernize. Well, why do you want to modernize? Sometimes it’s because we have to figure out a way to get in there and better maintain it ’cause it’s just too scary to tackle. The people that built it aren’t there anymore.

    Another reason, though, and I think the way to tackle that monolith part of it, is looking at our applications and saying what functionality and what business capabilities in the apps are most important to having the most flexibility or improving the speed at which we can improve that software to respond to and support the business?

    If we’re living in a world of changing how we’re delivering services into contactless, we’re living in a world where supply chains are going from inventory to on-demand, whatever it may be, examine those applications. Pick out where those elements are. And those are good candidates for going and saying, “Let’s carve out that piece and look at some new or alternative designs for how we can more easily maintain or more quickly develop capabilities in that part of the software.”

    It doesn’t mean you move it off the mainframe. It may stay there. It may be a combination of things. But that’s also part of app modernization is how we create software and the tools that we do it and sometimes how do we take on really big problems but divide ’em into much smaller pieces that we can do something about.

    Radcliffe: The application modernization of a Z application does unfortunately involve understanding it. And so having those tools for application understanding is the first key, but then actually understanding the business is the other key. And so it’s a combination of application modernization tools and understanding the business domains you’re trying to go after, really thinking about what are the focus areas and understanding what you need to run.

    Lots of discussion about microservices, and I always question, “Do you really want a microservice, or do you want a service? What level is the right componentization for this function?” And I always use a banking example because this one’s the easiest. “I need my credit card swipe to happen, and I only have a certain number of nanoseconds, whatever it is, to make that happen.”

    I’m not gonna make that a thousand microservices. I’m gonna make it a service. That service has to run fast. It has to be efficient. So it needs to be written as not a monolith, but a service itself. Where is that service? Is it bundled in the middle of something else that’s doing a whole bunch of other function? Can I strip it out? I need to look at the applications. And so using application understanding and the skills we still have today, because in many companies they actually still have a few people left that wrote those apps, not maybe for much longer, but they have ’em now. Let’s start to look.

    Use application understanding, the next generation, and learn from each other. I think the next generation, the newer developers, have a lot of skills in a lot of the new tools, and the people my age have the experience in the apps. So let’s bring these people together to help build this new culture, to help bring this together. Let’s take advantage of both sides and the tooling because even if they helped write the application, they still don’t know the whole thing anymore. It’s a monolith.

    Bring it together, and take advantage of these people. Use the people, both sides of the people, and bring all those skills together with the tooling to actually help you understand the business domains, the functions that you want to do, and split up that monolith. Maybe you’re splitting it up into a bunch of COBOL programs. It doesn’t matter. You’re providing the business value that you need to in a more maintainable, manageable way.

    Shimel: Fair. Guys, we’re running closer to the edge on time. I wanted to touch on the issue, though, ’cause it’s what Gerta does a lot of, and that’s around testing, right? As we’re modernizing these mainframe applications, we’re keeping them on the mainframe. Maybe we’re hybridizing them, if that’s a word, putting front end here and backend there or whatever. We’re modernizing the applications.

    Testing is an important part of it, as it is always. Anything special in particular that we’re doing around the testing of the modernization of mainframe applications that perhaps is different than – Gerta, go ahead.

    Sheganaku: When you think about testing, really just the whole topic of app modernization and also testing can be split into so many different bits and pieces. The one bit is really the architecture that we’re speaking about, where is everything hosted.

    Are we dealing with a monolith versus are we splitting everything up into service-oriented architectures or microservices or whatever we want to call those things, really making sure that we shift testing more to the left, that we make sure that the components are tested and such.

    Then the other side is really around the infrastructure, so really thinking about do you have those physical servers or do you have now more VMs, cloud-based environments? Do you have containers and so on where you’re hosting your _____ [break in audio] making sure that the intersections are tested properly?

    And then another point that’s also very important is really also the process side of things, so the delivery side. Are you having more of a waterfall environment or are you going more towards an agile DevOps environment? And all those things also go hand-in-hand.

    The typical approach would be especially you would have some more legacy type of applications that are running in the background that are really the backbone of your business, and then you would have more flexible applications built on top of that. For each of those you need different testing approaches.

    Something that we saw here, especially when we say “shift testing to the left,” was that while it was so far really possible to shift testing to the left when it comes to doing unit testing, doing API testing, and so on, it was always very hard when it came to end-to-end testing, especially across systems, when you have different systems, older systems, newer systems all working together, because there you really didn’t have that concept of just defining your tests up front and once everything is ready you just run your tests. It was really very difficult.

    Especially now I am focusing on the space of UI testing currently. In the UI space it was basically impossible to do that. That is also something where AI approaches can help in the future because if you look at the applications that we’re interacting with, no matter what technology is used in the background, no matter what type of programming language we’re using, if we’re having something in the cloud or on a server, it doesn’t really matter. How we interact with the applications as humans hasn’t really changed so much over time.

    When we looked at banking systems from 20 years ago and made an analysis, they look just the same like today. The applications that we work with on a day-to-day basis, although they move from one platform to the other, the technology keeps changing, bits and pieces keep changing, the way we interact with it really doesn’t change so much. So there it really helps to bring in AI-based approaches that help you take that human perspective when performing those tests to really be platform-independent.

    Shimel: Excellent. Sam, Mitchell, Rosalind, any thoughts on testing and app modernization with the mainframe?

    Ashley: I would just jump in and say it’s a great place to start, not to say you’re starting from scratch, but maybe you are, that whole CI/CD testing. As Gerta was saying, you can use cross-platform approaches, right? It doesn’t have to be one set of tools for this, one set of tools for that, et cetera.

    It can be, but there are some advantages to leveraging that knowledge base that Rosalind was talking about. You’ve been doing testing. Now how can you take that to the next level, whether it’s getting quality up or speeding up how quickly you can do releases, whatever the driver might be? As Gerta talked about, those are some fantastic approaches to take.

    Shimel: Rosalind?

    Radcliffe: Yeah, testing is one of the ways that we see people getting value fast, and it in some ways takes less cultural change. It’s easier maybe to get them to do automated testing, and automated testing is actually what makes you faster. You need that ability to test the system.

    Shifting left using virtual services, shifting left by virtualizing the whole system, allowing you to test your application in the build process, and doing unit test really does help in the mainframe space to get that feedback. It helps in every space, actually. But in the mainframe space, we seem to have forgotten this. I don’t know why. When I started, we did it. Okay, 20 years later people forgot. Getting to that unit testing, that automated unit testing, and making sure we’re doing that virtual testing, whatever it is to get that application transaction and really have that capability, and there’s lots of tools out there to do the testing.

    And the other part is it has to be cross-platform ’cause most Z applications are not Z-only. So open source tools like the Galasa framework that help you test across distributed and Z matter. Things like test data as a service matter. Use optimizations, looking at AI for data and understanding what data should you feed this.

    Multiple people have capabilities around that to look at the right data to test to make sure you get that fast feedback all the way at the beginning and get that developer that input so they aren’t waiting that six months of testing, that late-stage testing that is the reason why everybody says the mainframe is so slow because they spend too much time at the end testing. Let’s shift that all and bring it to the beginning, and then there are people delivering weekly into production because they have automated testing in place.

    Shimel: Absolutely. Sam, I’m gonna give you the last word, and then I’ve gotta wrap it up.

    Knutson: I think automated testing, and you have to underscore “automated,” is critically important. It’s one part of this. It’s not about the old, “Fast, cheap, good: Pick two.” People need to continuously improve quality, velocity, and efficiency on the platform, and automated testing is a key part of that. You will get measurable results, and you have to measure the results. So you have to put automated tests in place, you have to put quality aids in place, and then you have to measure that.

    If there’s one thing that we probably didn’t spend time today on that I think is a worthy thought for people to put front and center, it’s you’ve gotta measure from the very beginning what you’re doing, because any effort like this is going to entail some amount of spend in terms of resources. It is a worthy spend. It is the natural evolution of development for the platform.

    But you have to treat your developers like the high-performance athletes that they are. You have to give them continuous feedback. So use an analytics platform where you can understand what’s going on with your development process, what’s going on with your testing, so that you can see the results.

    When I looked in-house, what we were able to see is that as we evolved our development approach from waterfall to agile and we put these automated tests in place and we gave our developers a great developer experience, continually improving their tooling and our toolchain, we were able to get measurable results, like 25% more trapped defects that were caught before they ever escaped into customer hands.

    We were able to get twice as much code from individual developers and deliver at the speed of business. We didn’t get those results if we didn’t make the changes, including automated testing, including a great developer experience.

    None of it is optional. You don’t get a hall pass on any of this. Pick one and start, but you really have to think – this is a journey, and you’re gonna have to address all of these if you’re gonna get the outcomes that you want and be the most competitive that you want.

    I’ve seen customers getting amazing results that are measurable by outside firms where they’ve done studies and been able to see more than 467% ROI from implementing automated testing and building net-new applications on the mainframe. So this works. I’ve seen the proof points. I’ve seen the customer case studies. I would just urge people who have been sitting on the sidelines to get in because the competition absolutely is modernizing the way they build applications on the mainframe.

    Shimel: That was a strong finish, Sam. Thank you very much. Good stuff. Hey, Sam Knutson, VP of product at Compuware, a BMC company. Rosalind Radcliffe, Distinguished Engineer, IBM Enterprise Systems. Gerta Sheganaku from Tricentis on product for AI testing. And of course the one and only Mitch Ashley.

    This is Alan Shimel. Many thanks to Tricentis, by the way, for sponsoring DevOps Unbound today. This is Alan Shimel. We’ll be back in about another two weeks with another great DevOps Unbound. Stay tuned, but for now that’s it. Take care. Bye-bye.

  • How to Properly Approach Automation

    How to Properly Approach Automation

    Imagine if you backed out of your driveway, tapped the gas and BAM! You are at work. Nothing to worry about along the way, no traffic congestion, no other cars to worry about, etc.

    That is, unfortunately, how many of us approach automation. “Fire and forget” is real, until a problem crops up. I could talk about some complex environments, such as ARA, in a containerized environment with multiple languages, accessing mobility-limited databases and targeting several client platforms, for example.

    But we’ll keep it simple, and I’ll make an admission. Backup was one area where we at Ingrained Tech fell into the fire-and-forget trap. One part of our backup system runs desktops out to a server, the server (and other servers) out to a NAS and the NAS out to the cloud. I don’t think our security peeps will freak out if I mention we use duplicity for this backup and Backblaze for our backup cloud. Backblaze is a dream to use compared to others we’ve looked at, and the pricing is competitive, so it suits our needs well.

    A few weeks ago, we realized that while backups were completing successfully and verification was fine, it was taking nearly the entire week to do our weeklies. That’s bad news. We back up terabytes of data, but this is 2020, and our drive IO/Network can handle it. We started by looking at both of those things, and sure enough, it was drive IO causing our problems.

    Unsurprisingly, reading the disk and creating a collection of encrypted tarballs (a super-simplification of what duplicity does, but gets the point across) on those same disks was causing problems. After investigating, the one build that occurs on that server was suffering from backups stretching out, too. We had seen occasional issues with the build, but nothing jumped out as the source, and it was only one night every month or two. It was on our list. You know, the same prioritized list you have. And this build wasn’t nightly-critical. You get the idea.

    This was a relatively urgent problem, as soon as last week’s backup would not be complete when this week’s tried to start. Not good.

    We took our DevOps mentality to the problem and discovered a simple solution that solves the issues. The backup NAS is RAID 50, so it can handle a lot more abuse than the server that was slowing down backups. So, we took the simple expedient of rsyncing the source server to the NAS, then using duplicity to tar and encrypt before sending it on to the cloud buckets. Internally, we do not believe we need to encrypt backups because if someone is in our network, all of that data is laying around. By putting duplicity and both the tar function and the encryption function onto the backup NAS, we moved it to a far less busy system. Backup times dropped from days to hours instantly, and our unrelated build problems went away.

    We have to rework the automation somewhat to make sure validation is, well, actually validating the data is correct. But we’re well on our way to having a stable, faster system.

    How does this fit with DevOps? We see the same thing happening often in the automation rush of DevOps. Yesterday’s best solution is not necessarily tomorrow’s best solution. “Automate and watch” should be the catch-phrase. Improvement comes from both DevOps culture and paying attention to changes in the tools market.

    As Alan pointed out, someone will eventually win the end-to-end DevOps rush—since I feel the urge to inject my opinion on who stands the best chances, expect a blog on this topic soon. Until then, knowing what your options are and how to best achieve your goals–even on problems that were “solved” last year–is your best path for continued improvement. Yes, it means rework, but just like reworking your code, the point is to streamline things, get closer to your organization’s goals and keep improving after the low-hanging fruits of DevOps have been gathered. The rate of improvement in several areas of DevOps is stunning, so you may find you actually like a new tool better than the one you chose a couple years ago. And remember, knowledge is power.

    — Don Macvittie

  • Running Kubernetes in Production: Make Sure Your Routing Strategy Works

    Running Kubernetes in Production: Make Sure Your Routing Strategy Works

    Managing Kubernetes deployments in production raises some rather complex challenges. The dynamic nature of network communication in containerized environments produces unique operational issues–capable of bewildering even experienced DevOps teams. To mitigate these issues, software-based routing components offer key advantages across a number of areas.

    Here are a few particularly important routing strategies to consider within any production-grade Kubernetes environment.

    Leverage Tracing and Monitoring

    However thoroughly you test applications during development, a fresh round of issues will emerge in production. To help understand (and respond to) these new obstacles, tracing and monitoring tools provide developers with crucial visibility into their runtime Kubernetes environment. Choosing routing technologies that closely integrate with established monitoring and tracing backends will also help make things easier.

    All inter-service traffic passes through software routing components. This enables strategic application designs that purposefully collect microservices tracing and monitoring data. For example, tracing tools can identify the source and call flows of microservice invocations (assuming that the application is designed to support traceability). Thus, teams can determine how to fully leverage any available microservice, even those developed by different teams. By designing applications to provide metrics, DevOps teams can also use software routing and monitoring tools to understand production issues. For instance, tracking resource usage metrics for a microservice can reveal if its subcomponents are prepared to handle load at greater scale. That data can simultaneously point out performance limits and aid in troubleshooting new feature implementations.

    Secure Communications and Permissions

    Security must be a top-of-mind focus in any production Kubernetes deployment. Permissions governing internal network communications need to be tightly controlled to defend against attacks. Sensitive data transmitted within internal and external traffic must be encrypted and secured. Many businesses require data encryption not just as a best practice, but also as a matter of regulatory compliance. Unfortunately, these safeguards are absent in many Kubernetes environments.

    Routing technologies make it possible to enforce security policies using network segmentation. For example, only client services with justified business requirements should have access to microservices that handle sensitive data. Routing tools can also provide encryption. Service meshes can secure internal east-west traffic with secure TLS encryption. Edge routers can utilize provided certificates to encrypt all external north-south traffic. In order to fully automate the lifecycle management of trusted certificates, DevOps teams can also pair up routing technologies with services such as Let’s Encrypt. This automation makes it possible to continuously encrypt and secure the transport of sensitive data, with no interventions from human workers required.

    Absorb Unexpected Load Spikes

    Sudden popularity, DDoS attacks and other unexpected events can trigger surprise load spikes. As a technique, rate limiting enables operators to control the request rates to front-end services. This equips applications to absorb surprise load spikes and avoid failures. In this way, leveraging a routing tool to implement rate limiting techniques effectively limits downtime due to load events.

    Resolve Communication Issues

    Microservices can become unreachable due to errors by a container, host, network partition or short-term microservice interruption. Therefore, implementing mitigation strategies to keep these communication failures from affecting users is absolutely crucial.

    For example, the load balancer can respond to instance failures by directing requests to viable instances (and resuming normal traffic when the instance is available). During microservice interruptions, it’s critical to halt repeated retry requests to conserve resources and avoid cascading failures. Teams should use client services to employ circuit breaking to stop requests, send errors and fallback to alternative procedures.

    While it’s possible to implement these mitigation techniques using network-layer logic within each client, that approach is highly challenging and error-prone. Teams are better off utilizing routing technologies to control their applications’ mitigation responses.

    Achieve High Availability

    Compared to traditional hardware failover solutions, today’s software routing technologies offer a far more effective and cost-efficient means of introducing high availability to Kubernetes environments.

    Software routing technology enables DevOps teams to leverage architecture via a separate, horizontally-scalable data plane and a fault-tolerant control plane. The data plane makes it possible to add all the instances necessary to achieve the requisite capacity and resilience. At the same time, the control plane effectively tolerates failures to preserve uptime and safeguard seamless experiences for an application’s user base. A smart routing strategy thus ensures that production Kubernetes deployments are highly available and incredibly resilient.

    Deploy Across Multi-Cloud and Other Heterogeneous Environments

    DevOps teams may leverage Kubernetes across multiple cloud environments, on-prem environments and even alongside other container orchestration solutions. A routing strategy should meet the portability required to effectively support deployments across these different environments and solutions. By doing so, developers can deploy a common routing layer model across all deployments. This reduces key challenges by enabling a concentrated focus on a single and familiar solution.

    Selecting an appropriately advantageous software routing technology can transform the experience of deploying Kubernetes into production. Routing tools provide capabilities that can simplify or even fully eliminate many common challenges (and some uncommon ones). In this way, DevOps teams can take advantage of everything Kubernetes has to offer that much more easily and successfully, while achieving more robust and reliable applications.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or August 2020.

    — Manuel Zapf

  • Day 2 for the Operator Ecosystem

    Day 2 for the Operator Ecosystem

    The original focus for Kubernetes was very much on stateless applications, which don’t rely on coordination between instances or sharing data between client sessions. In practice, that usually means that we can stop and start our stateless applications without data loss or impacting on client connections.

    If all applications were stateless, the world of application deployment and management would be super simple, but unfortunately that’s not all use cases. The second class of applications are stateful, where the application is retaining data or state across its entire lifecycle. Whilst older monolithic applications such as SQL databases fit into this paradigm, there are also many architectures where the individual elements making up a particular service are actually clustered together in some way. They may be sharing state data between them, or they need to be synchronized together.

    One of the fundamental features of Kubernetes is that the scheduler can move pods around between agents, which works fine for stateless applications but not so well for stateful, which tend to like their storage and networking to not change during operation, unless handled in a very specific operational way. In order to try to handle the complexities of stateful applications, Kubernetes 1.5 added StatefulSets, which gave us the concepts of stable network and storage. But this doesn’t solve all of our problems, since Kubernetes still doesn’t know what’s actually going on inside the pods.

    Typically these kinds of applications have a set of lifecycle states which require logical ordering of actions in order to maintain operation. Stateful applications tend to have coordination between their elements, so, in general, if we were to just start and stop instances then we may impact on the correct behavior of the application—we may need to wait for re-balancing of data, or there may be multiple steps involved in an upgrade operation. This also applies to many operations that you are likely to perform such as scaling, and these actions also tend to be unique to the application, so a Cassandra cluster is managed differently from a Kafka cluster. That domain-specific knowledge of how to administer this class of applications isn’t captured in the paradigms that Kubernetes gives us out of the box, so how do we automate and package that knowledge?

    The solution to this problem is the Operator pattern. Operators encapsulate operational tasks in code, so we can orchestrate application lifecycle actions using Kubernetes APIs. Operators encode domain-specific knowledge about the lifecycle actions required for the application—how to scale it, how to upgrade it, how to recover from failure scenarios and so on.

    When we install an Operator into our cluster, we will get a controller, which is the code that manages the lifecycle of the particular thing and where the domain-specific knowledge is encapsulated. We’ll also get a bunch of custom resource definitions, which extend the Kubernetes API to address new resource types, so these add a new type which you can then put data into and instantiate.

    There are now many different ways to build Operators. The first one to emerge was the Operator Framework, which originated at CoreOS, but since then a whole range of different ways to build Operators have appeared, including Kubebuilder, Metacontroller and KUDO. These all take different approaches to building Operators, from SDK’s requiring Go expertise and deep Kubernetes knowledge to the KUDO approach of a polymorphic Operator, which can be configured very simply by using YAML.

    For cluster administrators managing large clusters, this proliferation of development methodologies and management interfaces can be problematic. The class of applications that Operators were developed to manage are almost always part of a larger application stack, with dependencies between them. If we have a lot of Operators written in different ways running on our clusters, how can we ensure they will interoperate with each other, and how do we validate and test them?

    When we depend on other services, we primarily care about two things: how to access and use it, and what it does. The KUDO project is working on the Kubernetes Operator Interface (KOI) to address this, enabling Operators to easily compose with each other based on a well-known set of CRDs. This interface is not just about connection strings and secrets, either; KOI aims to answer the question of “What does this Operator actually do?” By exporting an Operator’s behaviors as a CRD, tooling such as CLIs and GUIs can be written that better abstract the tasks that users need to perform rather than having to understand a library of custom resources.

    Defining behavior is one thing, validating it is another. How do we write standardized tests that confirm an Operator actually does the thing it says it is going to do? In order to try and address this problem, the KUDO project has also developed a test framework, Kuttl, which allows Operator developers to write tests to prove correct behavior. Kuttl has wider use cases outside of the KUDO project, with the ability to test conformance for all kinds of Kubernetes resources, so this is in the process of spinning out into a separate project.

    These kinds of efforts around standardization, definitions and testing will enable us to build ecosystems of Operators that naturally compose with each other, no matter the implementation—allowing users to build application stacks they can be confident will inter-operate consistently, rather than a constellation of unrelated Operators strung together by assuming the right configuration values. Collaboration and cross-project working are key to the future of the Operator ecosystem, and there are now more and more collaborative efforts emerging across the industry with these goals in mind.

    To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon EU, in Amsterdam. The CNCF has made the decision to postpone the event (originally set for March 30 to April 2, 2020) to instead be held in July or August 2020.

     

     

    This article was co-authored by Gerred Dillon, principal engineer for cloud native applications at D2iQ. Gerred builds tooling for large scale workloads on top of Kubernetes. He is the creator of the Kubernetes Universal Declarative Operator and assists with the Kubebuilder subproject, SIG API Machinery, and CNCF SIG App Delivery.

    — Matt Jarvis

  • COVID-19: Don’t Sacrifice DevOps

    COVID-19: Don’t Sacrifice DevOps

    Like everyone else, I’ve been following COVID-19/Coronavirus issues. My focus is more on the IT impact of the disease and its status as a pandemic.

    Let’s start by saying it isn’t the end of the world. Yes, it’s serious. Yes, it will be disruptive. Yes, those impacted by the worst aspects of the disease will leave a hole. But when it is over, and the tide has swept by, the vast majority of people will still be here, still trying to do what they need to do.

    That includes ITOps/DevOps maintaining whatever you put in during the rush to work remotely. Some organizations will be happy to throw something up and then rip it back down, but most will not. Once they’ve worked out how to work remotely, they’ll want to keep that as an option. Some to offer more work/life balance to employees, others to have a backup plan in case of emergency. Ask any company in Katrina’s path, remote work plans born in emergencies tend to hang around.

    It is the nature of DevOps to be nimble and reactive, planning on improving things as time goes on. That’s a good feature of both DevOps and modern IT teams. But some things are less nimble than others, and the rush to remote work is going to show that in a hard way.

    The VPN chosen or expanded to enable remote workers is not likely to be swapped out easily or quickly. While there is a sense of urgency—if you follow /r/sysadmin or to a lesser extent, /r/DevOps on Reddit, you know that urgency may even be too gentle a word. Some management teams seem to be in panic mode—you are going to have to live with some of these choices for a while, don’t make them knee-jerk.

    Normally, an MSP is your friend when moving like this, offering expertise and knowledge your organization doesn’t have internally, and shortening both deployment and the learning curve. But consensus is that MSPs, laptop manufacturers and VPN vendors are going to be booked solid for the near term.

    So, choose software when you can. Not because software is better (truly for some functions it is worse), but because it is generally easier to replace. A software VPN can be expensive, but you’re not buying hardware that is a sunk cost. Consider short-term solutions such as Amazon Workspaces or possibly  Chrome Remote Desktop, old standbys like VNC or RDP (likely with VPN in both cases), file sharing with things such as Sharepoint or OneDrive. Later on, you can look at a Cloud NAS such as Nasuni, or a more solid remote access plan.

    Yes, all the suggestions above have things you need to worry about. No, not everything will be within your control. However, these things are easier to rectify on iteration than “We bought big solution X, and it doesn’t do Y, so we need to replace it.” Start with short-term solutions and improve them as you go–this is truly the DevOps way, but it is easy to lose as you’re looking for high-volume, perfect and urgent.

    IT workloads (particularly operations workloads) are going to increase for any organization that wasn’t already set up for broad remote work. There is no way around this fact. As someone suggested online, “Kiss your family, tell them you’ll see them in a couple months and get to it.”

    You’ve got this. The internet is loaded with information to help, places such as Spiceworks and Reddit have people willing to offer advice all day. Smaller MSPs are likely happy to talk with you as they attempt to grow their business. Determine what you need, how fast you need it and implement the best solution you can in the time available, aiming to improve it on the next iteration. And keep the wheels of business running.

    It’s a flu variant, not the black plague. Businesses that deal with this well will come out well. Right now, “Deal with it well” includes a lot of IT work: from continuity planning, to fast DevOps iterations, to getting people at home productive and supporting them through this.

    — Don Macvittie

  • Coronavirus Response: Atlassian Expands Free Software

    Coronavirus Response: Atlassian Expands Free Software

    Atlassian announced today it will make cloud-based versions of Jira Software, Confluence, Jira Service Desk and Jira Core available for free as part of an effort to assist organizations whose employees are working remotely to help combat the COVID-19 coronavirus pandemic.

    Robert Chatwani, chief marketing officer for Atlassian, said given the current unprecedented situation the company has decided to increase the number of free versions of its software, including Jira project management software, which is widely employed by software development teams.

    The company already makes available free versions of its Trello, Bitbucket and Opsgenie software. Atlassian is hoping that once the current COVID-19 coronavirus crisis subsides, a much broader base of end users including ones from business units will have gained some firsthand experience employing its project management applications and collaboration tools, said Chatwani. Previously, Atlassian revealed it has a goal of having 100 million active users of its software.

    Chatwani said many organizations are finding that having employees working from home is the new normal for the foreseeable future. While regions in Asia appear to be recovering from the initial COVID-19 coronavirus outbreak, that does not appear to be the case in other regions anytime soon. Once organizations re-engineer their workflows, they may discover some knowledge workers are more productive working remotely, he noted.

    Less clear right now is whether having most or all employees working remotely is a temporary measure in the furtherance of a business continuity strategy or simply an inevitable future of work that, because of the current crisis, is arriving much earlier than expected.

    Of course, not every employee is psychologically tempered to work from home. Software developers, for example, may be more accustomed to working in isolation than project managers. Chatwani said it will be important for organizations to establish new routines to maintain organizational cohesion, including determining when development teams should launch sprints to build elements of an application. Managers also may want to designate “office hours” during which anyone can contact them online to make sure employees feel they are accessible, he added.

    Organizations should also get used to the idea that children may be both seen and heard while online meetings are occurring, given the fact that millions of children are now being schooled at home, Chatwani noted.

    Many organizations to one degree or another have been relying more on remote workers. There are even some companies that don’t have physical headquarters—all the employees already work remotely. Some of those organizations have even gone so far as to define a time zone around which the organization will standardize on regardless of where employees are physically located.

    Whatever the strategy employed, organizations that hope to succeed in this new climate need to go well beyond ensuring remote network connections are available and secure—they need to find a way to either maintain their existing processes and culture to make sure that whatever product or service they provide continues to be built or delivered in a way that allows the organization to survive and hopefully continue to thrive.

    — Mike Vizard

  • Scaling DevOps for the Enterprise

    Scaling DevOps for the Enterprise

    Information technology has been an integral piece of the modern enterprise for multiple decades and that dependency is only growing. With digital transformation, there’s the inevitable increase in demand for new applications. As such, software development services will continue to be a vital component of the business amidst the mounting pressure to innovate and deliver. This has resulted in many organizations moving from traditional development practices to more flexible and agile methodologies. Conventional software delivery methods are also being replaced in favor of enterprise DevOps practices such as continuous integration (CI) and continuous delivery (CD).

    While DevOps stimulates innovation and speeds up software delivery, adopting DevOps practices does not guarantee success as there are times when companies have been underwhelmed after implementing DevOps practices across their development teams. Knowing this, it is important to state that most DevOps failures come down to flaws in the execution of the methodology. A best-practice DevOps implementation should increase the value and impact of IT infrastructure by bridging new and old technologies. This allows for companies to better navigate change with clear time-to-value merits and lower risk. In short, it’s about scaling DevOps practices with security and quality embedded every step of the way.

    Look for Small Wins and Build on What Works

    Implementing DevOps best practices can mean a radical shift from the way an organization currently works. It is important that the entire team buys into this new approach. However, different dynamics within different organizations can lead to different results.

    Ideally, you should never have large complex projects. The best approach would be to work in small batches where you and your team can quickly learn from your decisions. By identifying the bottlenecks causing the largest constraints in your delivery process and incrementally changing, transforming and optimizing, your team could demonstrate the advantages of the new DevOps practices and build confidence in the team’s ability to implement the new methodology. Wins like these give a great start to the project before you’ve even introduced the entire IT organization to the changes needed to implement DevOps best practices. 

    Break Down Organizational Silos

    Known as well-intentioned areas of specialization, organizational silos are defined within the business to sharpen expertise and assign roles. Silos may have unintended consequences over time though, as each department potentially starts paying less and less attention to the other departments as they become consumed by their own assignments. As a result, silos inadvertently limit collaboration and have the potential to contribute to delays, bugs and conflicts. This is why C-level executives must initiate the process of removing the barriers to these silos and integrating the different disciplines within the IT department.

    For DevOps to work across the enterprise, executives must reorganize the configuration of software development projects by bringing in representatives from all important IT disciplines.

    Everyone Needs to Be on the Same Page

    Sometimes silos can be deeply rooted in an organization, and even if you do succeed in removing the barriers, there’s always the risk that they could re-emerge. Due to this, it’s important to create motivation for a silo-less enterprise.

    Anyone wanting to change a culture needs to define the actions and behaviors they desire, and design the work processes that are required to reinforce those behaviors. If your ops team is measured only by mitigating risk and your dev team is measured by delivering change, it goes without saying that you have a conflict.

    Through the incorporation of intra- and inter-team collaboration metrics into individual and team performance reviews, you can base a considerable amount of the review on lead times and quality metrics for software delivery. This is important not just for the dev and ops teams, but also QA and security. It’s easier to act your way to a new way of thinking rather than think your way to a new way of acting. 

    Real-Time Visibility Across the Toolchain and Environments

    When a team has poor visibility of their system, it can result in bad decisions and missed opportunities. Teams must replace manual project and release tracking methods that use spreadsheets and emails with an open, web-based solution that your team can leverage to plan, track and govern the entire application release management process.

    As teams and their stakeholders need real-time visibility across the entire DevOps toolchain and production environments, it is important that the chosen solution enables product team members to focus on their critical tasks while providing pertinent real-time information to their managers and stakeholders. 

    Don’t Lose Sight of the End User

    Once deployed, the application or service will interact with existing, and quite often, new systems. As it is developed for use by humans, don’t lose sight of the end user. Their satisfaction determines the success of your application or service.

    Actively involve users every step of the way, not just at the ideation and product requirements level. Teams should take the appropriate time to properly assess and fully understand the issues, challenges, priorities and constraints of their users. This knowledge can be utilized to define key performance indicators (KPIs) and continuous feedback throughout the DevOps toolchain.

    Move Forward on Orchestrating Continuous Delivery

    There’s no question that a lot of work goes into preparing and deploying a release. With the goal of delivering releases that have a high impact, it seems that companies sometimes choose to hold and then bundle together multiple application improvements into a single release. Such a strategy can lead to poor quality and failure—a result of deploying large infrequent releases. As another key driver toward an increased need for orchestration and automation, a lack of skilled employees means that many organizations have trouble releasing software in a reliable manner.

    The orchestration of continuous delivery is a fairly new way of planning and delivering business applications across the enterprise. With a focus on continuous improvement, continuous integration and ensuring you are always ready to deliver what the business needs, release control and deployment automation are vital to ensuring smooth delivery and a basis for governance.

    Conclusion

    The implementation of DevOps is a long term commitment. More a philosophy directing how you work, DevOps is not a rigid set of practices like ITIL, lean or Agile development.

    No organization will reach a state where it can comfortably say it has perfected its DevOps best practices model. Instead, it must continuously experiment with new processes, capabilities and tools identifying those that can yield the highest possible integration across the entire DevOps toolchain to deliver the best value for the business.

    — Mark Levy