devopsy.kinsta.cloud

Tag: shift left

  • EP 11: Unburdening Developers

    EP 11: Unburdening Developers

    Every app development need seems to ultimately fall to developers to fulfill. Build in security. Shift testing left. Provide users cooler, more customized experiences at cloud speed and scale. And do it all faster, of course. How do we accommodate the rising expectations without burning out developers? What can other team members do to lighten the load and allow developers to focus on the creative work they require to thrive? In this episode of DevOps Unbound, hosts Alan Shimel and Mitchell Ashley are joined by panelists Adam Kalsey of Tricentis, Tracy Ragan of DeployHub and Justin Hutchings of GitHub to talk about unburdening developers. The video is below, followed by the transcript.

    Alan Shimel: Hey, everyone. Welcome to another episode of “DevOps Unbound,” sponsored by our friends at Tricentis. We have a good, nice topic I want to talk to today, and you know, I refer to it as the straw on the camel’s back, but it’s about the ever-increasing load that developers seem to be taking on with technologies and frameworks such as DevOps and Agile and so forth that we’re seeing out here. And we’ve got a great panel to explore this. Let me introduce you to them before we go any further.

    Let me first of all introduce you to my friend Justin Hutchings. Justin is with GitHub and he’s gonna tell you a little bit about himself. Justin, welcome.

    Justin Hutchings: Thanks, Alan. Yeah, I’m Justin Hutchings. I’m a director of product here at GitHub. I work on some of our security features for developers, you know, whether that’s things like static analysis, security testing, or supply chain kind of features for developers in opensource and the enterprise. Glad to be here.

    Shimel: Nice to have you here, Justin. Welcome. Secondly, she’s a frequent guest here on “DevOps Unbound” and other MediaOps videos, my friend Tracy Ragan, CEO of DeployHub. Tracy, welcome, maybe a little bit about DeployHub and you for the audience.

    Tracy Ragan: Absolutely. Thank you, Alan, and thank you, Tricentis, for having these and always welcoming me on these panels. I like doing them. So yeah, I’m Tracy Ragan, CEO of DeployHub. DeployHub is a hub of metadata for deploying and configuring microservices. So we do automated configuration management so you know what your blast radius looks like for a microservice before you deploy. I’m also on the board of the CD Foundation, and you might see me oftentimes evangelizing around microservice usage.

    Shimel: Yes, we do. And then – welcome, Tracy. And then last but not least as a panel member today is Adam Kalsey. Is that pronounced right, Adam?

    Adam Kalsey: It is, absolutely, yes.

    Shimel: And Adam’s with Tricentis, but why don’t you tell us a little bit, Adam?

    Kalsey: Yeah, so I lead developer relations at Tricentis, and we make software to help people make better software, to test it and make sure they have higher-quality software. And my role is to help make developers more awesome by helping them make better software.

    Shimel: Thank you and welcome. And then last but certainly not least is my cohost for “DevOps Unbound,” Mitch Ashley, CEO of Accelerated Strategies Group, CTO here at MediaOps. Mitchell, welcome. Great to have you on.

    Mitch Ashley: Always good to be here, and having been part of many development teams, I’m excited to talk about this great topic.

    Shimel: Great. Let’s dive into it, guys. You know, when I started devopsy.kinsta.cloud about eight years ago, there was this constant push and pull. Was DevOps too developer-centric? It was all about the developers, the ops said. This DevOps thing is just – it’s all about the developers; we’re not getting our fair share. And you spoke to developers and it was, “You know what? DevOps is too ops-centric. It’s not about the ops. It can’t be everything ops. It’s got to be about developers.” And so there was this, you know, friction about was it too developer-centric or not.

    Over the years, though, what I’ve seen is an increasing amount of responsibility seems to have shifted left, as we say, right? And we shift left, and when we shift it left, we’re primarily – we’re shifting it to the developers. So things like testing, for instance, right, have become – developers are probably more involved in testing, or at least kicking off testing and this stuff, than they ever were. Security – certainly the whole rise of the DevSecOps movement has been about shifting security left, and in fact many of the most successful security companies in the DevSecOps space have become successful by making security tools for developers. We’re making testing tools for developers. We’re making tools to look at your blast radius for microservices for developers. Developers are deploying, right, as part of their DevOps focus.

    When is enough? When do we break the developers’ back? Not to mention the developers are the highest-paid people in the food chain, right? So by giving them everything, are we giving them nothing, right? Are we stunting their productivity? Adam, being that you’re the new guy here and you actually are in charge of developer relations at Tricentis, we’re gonna ask you to kick it off. Are we putting too much on the backs of our developers?

    Kalsey: I don’t think so. I think that the promise here is that we make things easier for the developers. I mean, sure, you have more tasks. There’s more things that you’re responsible for. But the overall effort and the amount of work that has to be done ends up shrinking. If you don’t find out about a bug in your software or a security problem or such until you’re in production and customers are using it, now all of a sudden you’ve got way more work to do. It’s harder. It’s more expensive. It’s more difficult. There’s more involved in doing that. And so by moving a lot of these things forward, we’re letting the developers be more proactive in taking care of them. We’re helping them get things fixed sooner. We’re reducing the amount of toil that developers have to do. Yes, there’s more upfront work, but that long-term toil starts to fade away and disappear.

    Shimel: Fair enough. I don’t know if you – you know, I don’t know if I’ve ever heard someone say developers are – you know, it’s making it easier or less for them. Justin, what do you think?

    Hutchings: You know, I’ve talked to a lot of companies that are the early stages of their DevOps transformation, and I think one of the most common mistakes that these companies make is they fail to recognize that adopting DevOps requires both technical changes as well as cultural changes. You can’t just get rid of the testers and say engineering owns tests and expect the same velocity. You’ve got to build out the infrastructure to support that, you have to do the training, and you also have to recognize your throughput is going to be less while you’re paying down that sort of debt on getting that infrastructure and getting the automation and getting all of those things.

    Companies that fail at the DevOps transformation, they go through sort of the motions without understanding the culture and talking about it. You know, years ago when I worked at Microsoft, the Windows team sort of famously had a one-to-one ratio between engineers and – software development engineers and test. These were folks that wrote test automation. And there was a big layoff, thousands of testers got laid off, and they said, “We don’t need them anymore. We have observability tools. This will be great.” And that was the most naïve statement I had ever heard, and it took a year for the organization to fully rebound, to figure out what the right set of responsibilities were for engineering, and how to support those folks effectively on it.

    Shimel: Fair. Tracy, in your roles, both at DeployHub and CDF, you’re dealing with both constituencies here, right, the traditional DevOps teams. But let’s face it: You’ve got dev and you’ve got ops in there, and you deal with both. What’s your take from the battlefront?

    Ragan: Oh, my goodness. I could write like a 20-page blog on this. I guess that wouldn’t be a blog. It would be a research project. [Laughter]

     Shimel: Or a book.

    [Crosstalk]

    Ragan: Yes! So I think that in my discussions with folks over the last six months, eight months, I am seeing a lot of SREs really stressed out. So I don’t know if it’s the development teams that are really taking the brunt of this, but if we think about our history and how we got here, we’ve always had somebody really important on a development team who always worked in hand with the operations side of the house to find out what happened, to help with the deployments. Even if they’re sitting there watching the operation first and hit the button, they’re the person that’s always taken this responsibility. And I think the DevOps movement was an effort to push more operations tasks to that trusted partner on the development team, and now that role has turned into an SRE.

    I would say five years ago we wouldn’t have seen this kind of stress, but because we’re going through this massive tsunami moving from monolithic to microservices, and we’re in a transition phase, we are putting a lot – what I see is we’re putting a lot more stress on the SRE role. Developers are going to the SRE, operations are going to the SRE, and the SRE is just trying to keep up with the underlying technology that’s wobbling and moving. You know, they’re trying to figure out how to stand up environments easier. They’re trying to implement Terraform or Rancher. They’re doing everything that they can right now to try to keep up, but they may be underwater for a bit longer.

    So when we think about where the stress point is right now, I think that developers have pretty much automated a lot. You know, we have tools like Tricentis to really improve the testing, but now we’re int his weird phase where we have these SREs that are trying to straddle both worlds. And they’re struggling.

    Shimel: Justin, you know, in your role there – and, guys, I don’t mean to call. If you want to say something, feel free to obviously jump on. But, Justin, I’m wondering – you know, GitHub has sort of a unique view with this.

    Hutchings: We’ve been trying to push folks towards more automation, more DevOps, you know, DevSecOps with all the security features we have. You know, again I think culture is the key. I look at anytime you have an unfunded mandate, you know, it’s gonna fail. A lot of times organizations will say we’re going to microservices or they’ll say we’re gonna adopt all these security tools, and then they don’t think about how that actually gets done and they don’t recognize the sort of pain and training and all of the things that have to happen to get folks there.

    You know, and I think there’s been a lot of advancement here. You look at the state of observability. You look at the state of CI/CD. Things are getting better, but I think, you know, we in the industry that sort of work int his space sometimes paint such a rosy picture that our customers aren’t prepared for the hardness of the journey that lies in front of them. And they’re looking at how do I get to the magical garden of DevSecOps, and the magic garden is there, but it is not a free entry, you know?

    Ashley: You know, Justin, to what you were saying, I think along with culture, part of is a role change that’s happening across the whole organization, right? A couple things are happening. We’re doing not just automation that the developers are doing, clearly SREs. Other test automation, infrastructure as code, you know, those are all development that’s happening to support those. It doesn’t say that every person is an expert developer, but there’s some development, scripting, other kinds of skills that come with it.

    So I don’t know if we’re so much loading it on the developer as much as we’re elevating our level of automation and kind of interconnected of the process, and we’re helping people evolve and growing their strengths – let’s put it that way – to also include some software development, or more closely working with people that can. I think that’s part of that cultural change, too.

    [Crosstalk]

    Kalsey: One of the things that’s actually happening there is as things become more development-centric and, oh, now it’s Toad and now it’s – the teams are making exactly the wrong lesson out of this and they’re learning the wrong thing out of it. You know, one of the things that Justin said earlier is, “Well, we got rid of all the software testers because the developers are doing the testing now,” and that’s one of the dumbest things that you can do as an organization, is say, “Well, now that my developers can do this, let’s get rid of all the experts.” And instead of saying, “What is the new role of this expert and how does their role change?” whether, okay, now they learn to code so that they can expose more of this as code, or do they become consultants to the developers and help them do this right and help them understand how to do this right? Could I remove a lot of those pieces of toil and a lot of that extra stuff from them and let them focus on the things that they’re good at – writing the code, building the code, thinking through the problems – and let me focus on the things that I’m good at – thinking about the infrastructure, designing the infrastructure, designing test plans, thinking about the non-happy path?

    And so companies – I talk to companies all the time that say, “Well, we’ve moved – all our developers are doing the testing now, so we got rid of our QA department!” And that is – that’s just frankly dumb. Don’t do that.

    Ragan: I think it was an outcome of the Agile movement, right? Because we started preaching this idea of bringing everybody into the same room together to do these tasks. So we broke up our testing departments and we spread them out across all of the teams, as well as some of the operations, which is how the SRE title came about. So, you know, first we siloed and then we tried not to silo. But ultimately it is how do you come up with a strong automation standard across the organization? And how are you gonna implement it? If there’s a testing team that has a group of people and each one is assigned to an application team, or if you have a siloed testing team and a siloed operation team, ultimately we’re still pushing the same product through the lifecycle. So how well do we do that? And maybe Agile confused things for us for a while and kind of broke things up and disrupted what we were doing. So now the task is how do we move forward with where we’re at to make it continue to be easier to get things across the pipeline?

    And I have to say that one of the biggest challenges I hear from people is that continuous integration works well; continuous delivery from dev, test, through prod doesn’t work so well.

    Hutchings: Yeah, and I think, you know, what you hit on there, Tracy, is really important because, you know, when you have an organization where the engineers are the only thing and engineers are undifferentiated and treated as some sort of fungible resource, I think things fall apart. You know, what I’ve seen in successful organizations is they have some subject matter experts, whether that’s SRE or a data team or an infra team, that are building out things that make the rest of the organization able to scale. And that’s a really good, healthy blend where you’ve got experts that you can lean on, but then engineers are empowered to do sort of the last-mile delivery there.

    Kalsey: Yeah. When a company says they’re gonna create a cross-functional team and then they eliminate most of the functions, that’s a problem. I mean, you don’t say I’m gonna have a cross-functional team, so we’re going to bring product managers and designers in and, oh, by – now we just don’t need product managers and designers. So why do we do that with operations, with testing, with all of these other things that are important for software creation? You know, eliminating those and just saying, “Well, the developers will do it,” of course that makes for a lot more work for the developer. The way that this needs to work is keep those cross-functional teams and change their role and bring them further – bring them earlier into the developer cycle. Have them involved in design. Have them involved in figuring out what the product is going to do and how it’s going to do it, not just let’s get rid of them. The options aren’t they happen at the very end of the process or we don’t have them at all. It’s let’s bring them up further along in this process.

    Shimel: You know, part of it, though – and I don’t know if anyone wants to admit it – is that there’s sort of this bias that everyone else on the IT team aspires to be a developer ’cause that is the ultimate kind of thing. So, for instance, it was this way certainly in testing, that being a tester was a waystation on the way to being a full-blown developer, right? Or maybe a developer who wasn’t that good, well, he could do testing, right, and that testing in and of itself wasn’t a “profession.” It was really just like a junior developer did testing. And, you know, security people, well, they were always the rebels, right? Half of them didn’t have formal training. But you know, operations, the SRE folks, right, how much of a developer heritage or pedigree is there in what you need to be able to do from an SRE perspective? And so when we put together these “cross-functional” teams, it’s almost with the idea that, well, you’re all gonna be developers now, right? You’re gonna reach your lifelong ambition by being in this DevOps team as part of a developer. And we lose out on the professional tester, on the professional SRE, on the professional security person, right, in this rush to all be developers. What do you think about this?

    Ashley: Well, I think we did this to ourselves in the DevOps movement. You know, I remember when I first wrote my first article on devopsy.kinsta.cloud about what is DevOps and I was sort of grappling with it myself, the mantra was, “We don’t need operations, we don’t need test, we just need developers. Code or developers writing code will eat the world.” And it wasn’t outsiders that said that; we said that, or subsets of us said that. And I think we recognized after we got into it, no, as Adam points out, those are exceptional skills. Frankly some of the worst testers I’ve worked with were developers. They were terrible at testing their own stuff. They were great developers, right? But some of it is elevating maybe too much the role of the developer to be the rockstar beyond the platform, if you will – and there are fantastic developers. I don’t mean there aren’t. But there are amazing, amazing test people.

    And so let me put it to you this way. When I was running product development at StillSecure with you, who did I listen to? The testers. I listened to them and I asked them, “What problems are we finding?” and I could tell if we were ready to ship, not just by the numbers but by their opinion. And that’s why I really relied on, because they saw it from a different perspective than a developer does. So that’s my opinion about it. Now feel free to disagree. Anybody can take me on and say I’m full of baloney.

    Hutchings: So I was gonna say, Mitch, you know, have folks heard the story about – what is it – the IBM Black Team that they developed back in the ’80s for testing where they – basically testing was a sort of poorly-regarded function in the organization? And they ended up creating sort of a hit squad that famously would go and just destroy people’s code. And they created a culture around that, and they created sort of an ethos of that being a special place to work. And I think that’s something we need to find with our developers. Like, if we only celebrate the developers that make the shiny features, we’re not gonna be in a good state. We’ve got to celebrate the developers that can destroy other people’s code with tests or can build something that scales really well or that build great observability things. Like, it can’t all be how many features did you ship; it has to be how much you contributed to the whole.

    Ragan: Yeah, and I – you know, I think, Justin, that the cultural discussion is one that any team that’s going through this experience right now that we’re describing, that the developers do feel burdened in some way, they should be looking at how they have built that team. And kind of each role should be a specialist role, and they all have to be an equal role. And if you don’t have that, you really haven’t created an Agile team and that team’s not gonna become a high-performance computing team, because it is about equality on the team and the ability for every single person to stand up – tester, the SRE, the junior developer – and be able to communicate problems as they see them in an equitable way and not feel that they are being judged.

    And I have learned that from working in an opensource community. I’m really amazed how well the opensource communities work together, ’cause you have people with all kinds of different backgrounds, and it fascinates me to see how much they can accomplish and how well they accomplish it when they’re not all working for the same company and they’re there because they want to be. So how do you create that opensource kind of nurturing culture within a particular team if it’s built especially around Agile, where we’ve taken operations and testing and developers and, you know, other specialists and put them on the team – documentation, requirements gathering – and how do you make it an equal team? That I think is the challenge to keep the burden off of the developers’ backs.

    Shimel: You know, and that – paradoxically, that’s also the big attraction to like cloud-native communities, to these opensource communities, because I think people recognize the – and I don’t mean it in a bad way – the kumbaya kind of spirit that these opensource communities foster. GitHub, quite frankly, Justin, right, if you look at the history of GitHub, that’s probably been one of the huge drivers of 52 or whatever it is million accounts on GitHub, is that sense of community, right? How do you build that sense of community interdisciplinary across your teams within an organization? Adam, you know, though the role says DevRel, it’s really TeamRel, right? And you know, it’s got to be part of your mantra there, part of your job description, isn’t it?

    Kalsey: Yeah. One of the things that is interesting about when you look at opensource is the person who wrote the documentation is a hero, because nobody wanted to write the documentation. The person who wrote the tests is a hero because, wow, they’re writing the tests, and that was a lot of work and I didn’t want to do that amount of work. But then when we go into a company we say, “Well, we’re gonna hire a couple of QA people to do the testing because they’re cheaper than the developers.” And, well, yeah, maybe if you’re hiring bad QA people they’re cheaper than the developers, but if you’re hiring good testers, they’re as expensive or more expensive than the developers.

    And so in the opensource world we look at it and we go, “These people are peers and equals,” and then we come into a company and we say, “They’re second-class citizens, and we’re gonna hire a couple of tech writers ’cause they’re a lot cheaper than having the developers write the documentation, or we’re gonna hire some testers, or we’re gonna hire some ops people ’cause the person to watch the dashboard is way cheaper than having a developer do it.” And we’ve seen it happen with the SRE role where, when we elevate that and that becomes part of the development team, the other – SREs are expensive and hard to find. That same sort of thing I think needs to happen in other roles on a development team in order for them to stop being that second-class – and also to stop having developers be the ones that have to do all of this work. If you look at it and say, “Well, the test writer or the doc writer is a cheaper person and we’ll just hire a couple of them,” it’s more likely to fall on the developer to do the hard work than to hire the good people and have them do the hard work.

    Shimel: Fair enough. Mitch, what do you think on that?

    Ashley: You know, it’s a really good point, and I think part of what you’re saying, Adam, is it’s a specialty. You think differently to be a fantastic tester, not just testing but the whole test strategy and how to do all the different kinds of testing that’s involved, and then of course how do you automate that and do you do that yourself? Do you work with part of the dev team to do that? Same thing for SRE, right? It takes a different – someone who thinks about it differently. And it’s sort of that cognitive diversity of teams that we value from a diversity standpoint. We value that also because those roles also require specialty skills. You know, someone who’s designing – and, Tracy, you could explain this better than I could, but someone who’s designing a cloud-native architecture and thinking about the microservices and what’s gonna constitute a microservice, how we’re gonna communicate between them and how we’re gonna secure them, that’s a different kind of thinker than how do I break all that stuff. That’s really nice that you did that, but watch how fast I can break that. That’s fun, and that’s where people really enjoy working together. But I think that’s part of where you’re elevating, too, Adam, is you’ve got to value all the skills, and they may be expensive, they may be not quite as much, but they’re needed. They’re part of the process.

    Shimel: Interesting. You know, Justin, we’re talking about testers. We’re talking about SREs. We’re talking about developers. Once again, no one’s talking security. Are we just sentenced to be forever the outsiders here? What’s it gonna take?

    Hutchings: Most of the security people that I talk to take that as a challenge, you know? We obviously want developers to go and embrace security as part of, you know, their everyday job descriptions, but folks in AppSec and especially in the security research community, they’ve always felt like kind of scrappy outsiders that are going to shine a light on things that other people didn’t notice. And so I think even when we have very security-minded, security-motivated engineering teams, security is a discipline that’s got to stand apart, because we want to have that sort of trusted auditor role or that, you know, lookout for – whether it’s bug bounty reports or stack analysis findings that no one looked at. Like, someone’s got to be watching those, ’cause developers, especially in the world of microservices, they might work on a thing, finish it, and then go somewhere else. But the security world changes all the time. If you walk away from that microservice and all of a sudden we learn about a new category of security problems and no one’s watching it, well, that’s a problem now. And so security has a really special role in terms of being the stewards of all the things, even when they’re not directly accountable for the functionality of it. And I think that’s a special thing that we should celebrate.

    Shimel: Okay, we’re special. But you know, as special as we are, yet we’re still seeing security tools for developers, right? That is a bona fide trend in our industry, and at the same time we’re also seeing – when we talk about the load on developers, we’re still seeing an extreme need for more security people, right? We just can’t produce enough qualified security folks, it seems. If that’s the facts, right, there’s no getting around it. What else is there to do but put this load on the developers? How else do we deal with it, automation? I don’t know.

    Hutchings: You’re absolutely right, and I think the key for tool spenders like GitHub is to figure out how to give developers enough information that they can be successful but not overwhelm them, because security tools, you know, there’s a risk of false positives. If you give them too many, they will ignore them. And so, like, as a tool vendor, I have to be extremely careful to make sure that I don’t burn out developers on the things I think are important, ’cause they might not find them as important as I do.

    Kalsey: The other thing those vendors have to do is make less work, eliminate work for the developer, not cause more, not give them more things to pay attention to, more work to do, more administration to do, but look for ways that you can remove that work through the tools.

    Ragan: Yeah, so we have a challenge facing us when it comes to automation right now, and the challenge has to do with these imperative pipelines that we’ve been running for the last ten years. Sometimes you have a Jenkinsfile that is pretty complicated, and we’ve been running them, we copy them, and we use them. It’s imperative.

    The problem with it is because it’s imperative. We are seeing a shift now to moving away from these imperative pipelines to where it’s more declarative and event-driven, which means that it should be easier then to take these older – if you had a purely declarative pipeline based on events, think about how easy it would be to say, “Okay, we’ve decided on this set of security requirements to add to sort of the workflow steps for the testing workflow, for the development workflow, and we’re gonna plug those events in.” The reason why we don’t do that now is because it’s too hard. [Laughter] It’s just hard, because we have to take these old imperative pipelines and try to modernize them. And without having that declarative framework, these kinds of tools, even adding continuous testing, even adding continuous deployments all the way to production, it’s hard and it takes a lot of work, and you have to change the tires as you’re going down the freeway in an imperative model. The faster we can get to an event-driven process in our CD pipeline where we can declare them and we can automatically generate what we need to do in a workflow, the sooner we’ll be alleviating some of the pain that the developers might be seeing, the sooner we can bring in better testing, the sooner we can bring in the DevSecOps conversation, and in our case automated configuration management.

    This has always been a problem for any of the tools in the CD pipeline to start being plugged in, because the imperative pipelines that we are dealing with on a daily basis, nobody wants to touch them. So we’re gonna have to disrupt that first, and when that gets disrupted and people start moving to something more declarative, we are gonna see an increase in adding all of this kind of automation into the CD pipeline.

    Ashley: I’m curious, Tracy. Is it the complexity of the pipelines that – the way they’re defined, is that why people don’t watch to touch them, or –

    Ragan: It’s the number.

    Ashley: The number, okay.

    Ragan: You’ve got – literally I’ve seen people with thousands and thousands of workflows ’cause they’re running one for every single application version. And to look in that and say, “Oh, we’re gonna change all of these,” because it’s not templated so you can’t just change it in one place and it’s just gonna trickle down, because they’re all these very statically scripted processes. And for whatever reason on the distributed side of the house – when I say distributed, I’m talking about everything except the mainframe – we have relied on heavily scripted processes, and we’ve scripted ourselves into a box that we cannot – we’re not Agile. We cannot easily update these pipelines to include security in them.

    We talk about it at the CD Foundation often, trying to elevate security as a first-class citizen in the CD pipeline. But that means that everybody’s got to start updating all those CD pipelines, and they’re not templated. They’re all statically scripted so you have to address one at a time. And as I think Justin might’ve said, you’ve got to plan to pay for that. So we’re in a box, we’re in a corner, and we have to somehow get ourselves out of that corner in order to alleviate these kinds of problems. And being Agile at the CD pipeline level means that we’re gonna have to dump these imperative processes.

    Hutchings: And I think you’re exactly right there, Tracy. One of the challenges we also see with security is this desire to start slow so that you don’t overwhelm people with too much, and if you don’t have a centralized control plane that everyone pulls from, you know, as you start to introduce these security tools, you can’t dial up the difficult level as you go. You have to hit those distributed CI pipelines every single time you want to go and update the difficulty, and that’s not tenable. I mean, you know, there are services out there – I think GitHub with Actions is doing a pretty okay job with allowing some centralized management, but it’s a spot that we all have to invest in because, you know, that control plane for specialists like security to go and help inject things is just not as mature as it needs to be.

    Ragan: And the CD Foundation has started an events working group to start having this discussion. Right now they’re talking about just vocabulary, but I believe that will grow into a bigger project in the future so that we can start thinking about how you can have that event-driven process so that you can dial it as you need to, right, as oppose to having them statically scripted in all of these imperative pipeline processes.

    Shimel: So, guys, we’re running low on time. Just one other topic I wanted to bring up. You know, in looking at this problem, though, I’m reminded of when I read The Goal, right, which is kind of the book upon which Gene Kim based his Phoenix Project on. The Goal is about more manufacturing; The Phoenix Project was IT. But really it talks about bottlenecks, resource constraints. And the story of The Goal – and I forgot if it was the first rule, the second rule, whatever it was – is as soon as you fix one or get rid of one problem, the next problem down the line shows itself that maybe you didn’t see because the problem in front of it was hiding it camouflaging it. In trying to make life easier for our developers or take the strain off of that point, are we really just going through this goal process of discovering the next bottleneck? And maybe that’s what – maybe that’s the secret to life. Life is just about going from one bottleneck to the next, right, and discovering what’s downstream from there. There’s something Zen about that. But is that what we are sentenced to for sure, or is there a way out of that?

    Kalsey: I like to say that when designing large-scale distributed systems you can’t remove your scalability or your failover problems; you just move them to a different part of the stack. And we do that a lot with development teams as well, is we don’t remove the problem; we just move it to a different part of the process and move it around.

    I think one of the ways out of that is one of the tenets of Agile is let the teams make the decisions. Hand them the problem and let them decide how to organize, how to do these things, how to make stuff work. You know, if management is pushing down on development and saying, “You’re now going to be the testers,” of course we’re causing more work and more problems and more things for them, and they’re going to become the bottleneck. But if we let the teams figure out – they’re the ones close to the work, doing the work. Let them make the decisions about how we’re gonna do this and what they need to make it happen.

    Shimel: Fair enough. All right –

    Hutchings: Yeah, I was gonna say I like the term incrementally correct. You know, Alan, your thinking about bottlenecks is absolutely true. I don’t think there’s ever a world where we will get to 100 percent panacea, but we can hope to get a little bit better all the time, and that’s the journey that we’re all on, right?

    Ragan: And failing fast.

    Shimel: _____ continuous improvement, and failing fast – like get to know –

    Ragan: And allow your developers to fail and not – or your testers or your security. It’s okay to fail, because then you learn what you need to do. Failing fast is part of the learning process.

    Shimel: Fantastic. All right, guys, we are out of time. I want to thank all three of you for joining Mitchell and I on this episode of ” DevOps Unbound.” Thanks to Tricentis for sponsoring ” DevOps Unbound,” as always. We will be back in – not next week, the week after with a new, fresh episode, as well as check out – we’ll have a new ” DevOps Unbound Roundtable” open to the public for questions, and you can find out about that at DevOpsUnbound.com or on our Digital Anarchist site or devopsy.kinsta.cloud frankly. Look under – it’ll be under webinars and you’ll find the Roundtable there.

    Until then, though, Tracy Ragan, as always, it’s a pleasure to have you on. You bring such enlightenment to the whole discussion. Thank you.

    Ragan: Thank you, Alan.

    Shimel: Thank you. Justin, again, always nice to have you on one of our panels and discussions here. Many thanks for coming. And, Adam, thank you and good luck in – you know, I know you’re relatively new in your position there at Tricentis but it looks like they made a hell of a hire. Good luck, and come back and visit us again.

    Kalsey: Would love to. Thanks a lot.

    Shimel: All right. Mitch, you want to take us home?

    Ashley: I just wanted to thank our panel as well, and I love the incremental progress idea, and it’s always getting outside perspectives sometimes, too, that’s helpful. So I think we’re here to help each other in this process and not one of us has the answers.

    Shimel: Right on. All right, this Alan Shimel for “DevOps Unbound.” We’ll see you on the next show. Bye-bye, everyone.

    [Outro music playing]

  • How Real-Time Debugging Improves Reliability

    How Real-Time Debugging Improves Reliability

    When designing and building software, service reliability is always at the top of the list of critical focus areas for development teams. Every team that builds software typically has, either directly or indirectly, service level agreements with their customers. These are, essentially, agreed-upon metrics or performance criteria that teams use to measure and ensure the reliability of a software system.

    Organizations may measure reliability in different ways, looking at metrics like service availability, mean time to failure or mean time to repair. Regardless of how an organization chooses to measure reliability, you can often assess the health of an organization based on how reliable their applications are. Since we know that reliability issues happen regardless of how well we prepare for them, having tools to solve those issues quickly and efficiently becomes critical to maintaining smooth operations.

    The Health of Your Organization

    Most software development organizations today have key performance indicators (KPIs) which they measure to understand how well they’re doing and to be able to quickly and easily measure the health of their application development processes. If teams are finding that they are not meeting KPI’s, it’s often a sign that something fundamental needs to change or that new and improved processes need to be put in place. Teams that are constantly measuring what they’re doing well and where they are falling short with a data-driven approach are typically the teams that come out ahead.

    In short, a team’s ability to measure and hit KPI’s and service level agreements with their customers directly influences the overall success and health of that team. If an organization isn’t producing applications that are reliable and available when their customers need them, it’s unlikely that company will have customers for very long. When reliability issues do occur, it’s critical that developers can immediately go to the source of the issue in their code to ensure that they can solve those customer issues as quickly and efficiently as possible. One such approach is to give developers the ability to do real-time debugging directly in the environment where those issues occur, without needing to create costly reproduction environments or having to figure out how to reproduce issues locally that are, many times, data dependent.

    Reliability is Shifting Left

    Historically, software reliability has fallen on the shoulders of operations or production support teams who manage the deployments and runtime of software after it’s been deployed to production (and after developers have thrown the code over the metaphorical wall). Things have changed in the recent years, in that many of today’s development teams are responsible for their applications from development all the way through to running and supporting those applications in production. Because of this, developers have more incentive than ever to ensure that their software has reliability built in, or at least the ability to quickly debug reliability issues when things go wrong.

    Real-time debuggers are one such category of tools that development teams are choosing to package with their applications in order to solve those hard-to-address customer issues faster. Real-time debuggers are solutions which allow development teams to gather code-level debugging information such as snapshots of local variables, stack traces, tracing information and profiling data. All of this data (which traditionally relies on developers having the right logs in their code and, if not, deploying new debug builds) becomes instantly available, on-demand whenever an issue arises. This faster approach to identification of issues can drastically improve the reliability of applications by allowing developers to solve issues in a fraction of the time it would normally take.

    Teams Focusing on Reliability Move Faster

    Many development teams have shifted to an agile approach to delivering software; the goal being to respond quickly to business needs and to get new, valuable features to their customers as quickly as possible. Even though this is a helpful step, unplanned events can and will happen, which need to be planned for as much as possible ahead of time. In a recent Forbes article by Rookout CEO Shahar Fogel, he discusses how businesses should consider these software bugs as mini outages. Any time teams are dealing with software bugs or other unplanned issues, it costs the business revenue and can potentially leave customers frustrated. Focusing on metrics like mean time to repair (and giving developers the tools they need to reduce the mean time to repair) can give teams the ability to better predict how long unplanned issues will take to remediate once they occur.

    In the book Accelerate by Nicole Forsgren, Jez Humble and Gene Kim, the authors discuss the notion that, by giving developers tools to fix problems when they occur, teams create an environment where developers accept responsibility for global outcomes such as quality, stability and reliability. Investments in technology are also an investment in people. So, it’s no wonder that by investing in technologies that can help improve reliability, teams will be able to move faster. This means happier customers, happier employees and a thriving business.

    By focusing on service reliability, development teams are not only able to move more quickly, but are also directly contributing to the overall health and success of their organization. We’re seeing the onus of reliability shifted from operations teams to development teams, which adds newfound importance on ensuring development teams are building reliability into their product. Real-time debugging tools are one potential solution to ensuring that developers have the ability to quickly address unplanned customer issues which can directly impact the perceived reliability of an application.

  • Prevent False Positives From Derailing Shift Left

    Prevent False Positives From Derailing Shift Left

    Static application security testing (SAST) tools are designed to balance false positives (incorrect warnings) with false negatives (missed vulnerabilities) primarily because deeper analysis requires more time and computing resources. Both of these are in short supply among developers that are tasked with meeting shorter and shorter product delivery milestones.

    So, while SAST vendors consider a true positive a correct detection of a real defect, and a false positive an alert on a bug that does not exist, this is rarely the lens used by developers. What really matters to them is whether the output of a SAST tool is useful and actionable. For example, there is a great deal of variation in how developers interpret SAST scan results, depending on the nature of the defect, the role of the user, the platform on which the application will run and the environment in which it is deployed.

    Take, for example, a true positive result of a buffer overrun, one of the most notorious classes of C/C++ defects from a security perspective. In the early stages of application development, it almost always makes sense to change the code to fix such a bug. Developers are actively changing the code, anyway, so fixing it involves little extra overhead. However, if the same defect is found after the application has been deployed, then it is much trickier to decide whether it is worth fixing.

    That’s because the cost of fixing a real bug may exceed the benefit of fixing it, and the benefit of “correcting” a false positive may exceed the cost of leaving it alone. Since SAST tools can only be relied on to give narrow technical answers, humans must interpret and determine which static analysis results should be acted upon.

    Consider a hypothetical comparison of different SAST tools in the graph below. Tool A has good recall (i.e., the ability to identify real defects) and precision (i.e., the ability to exclude false positives) which results in finding many of the real bugs with a reasonable amount of false positives. Tool B has high precision but poor recall, resulting in low false positives but a higher number of false negatives (undetected security vulnerabilities). Tool C has poor precision but high recall, resulting in detecting all the possible errors but also with a very high number of false positives.

    A comparison of recall and precision of hypothetical SAST tools.

    Developers hate false positives because they introduce unnecessary work and delays. This, in turn, has a disproportionate effect on the way tools are designed, configured and used. If given a choice between Tool A that reports 40 real defects and 10 false positives, and Tool B that reports 50 real bugs but with 50 false positives, users will almost always prefer the former, even though it is finding fewer real defects. This is perfectly understandable — users are asked to weigh an immediate concrete negative (time spent looking at false positives) against an intangible potential future positive (vulnerabilities that may not show up).

    However, if one offsets the time and risk saved in finding those 10 bugs earlier (i.e. by avoiding expensive and potentially dangerous bugs in finished products) against the time needed to assess the additional 40 warnings as false positives, then it quickly becomes apparent that configuration B is more economical. This is especially true with security vulnerabilities that slip through testing and can have expensive consequences.

    Since the number of warnings within a DevOps pipeline can be a deterrent to getting the most out of a SAST tool – especially early in the project, when it provides the greatest benefits – organizations should consider the following techniques to increase developer attention to alerts:

    • Filter and Focus: Parse the viewed data from the tool’s interface, focusing on what is most important for the project and assigning developers to fix critical issues in priority order.
    • Mark and Defer: Lower the priority, or change the state to “later,” for example, on all or a subset of the warnings based on some set of conditions that are less crucial to the project.
    • Stop the Bleeding: Use the above techniques to temporarily defer existing warnings with the emphasis on fixing new defects that are introduced as code is changed or new code is developed.

    Despite the fact that it will invariably produce some false positives, SAST helps shift security further to the left by ensuring security and programming vulnerabilities are eliminated when code is being written, rather than at the testing stage when they are much more expensive to fix. Filtering and prioritizing warnings are two effective techniques to ensure developers don’t throw out the baby with the bathwater.

  • Why Browser and API Testing is Shifting Left

    Why Browser and API Testing is Shifting Left

    For developers working with cloud-native technologies, software testing and quality assurance is a key part of product development. For most teams, testing has “shifted left,” so the quality of the code can be assessed as it moves through the CI/CD pipeline. This approach to quality assurance reflects the rise of DevOps and other modern development practices – developers today own their applications from end-to-end, and conducting testing early in the product life cycle is how they find potential bugs, save their organizations time and money and build products that delight their customers.

    More recently, developers have added end-to-end API and browser tests into their CI/CD practices, extending the scope of their testing and quality assurance practices. By testing how code performs on different browsers and how it interacts with important data sources, developers have added another layer of quality testing to the early stages of their development processes. This has brought added benefits to other “shift left” practices, such as unit testing and security testing, and has led to a new understanding of quality assurance.

    Traditional Approaches to QA Can’t Support the Pace of the Cloud

    As companies have migrated to the cloud and adopted agile development methodologies that align with DevOps practices, they’ve encountered conflicting priorities between releasing code quickly and maintaining the highest level of quality on each release. Traditional manual testing techniques were not built for agile workflows, but instead supported older waterfall methodologies, in which testing could be conducted as a discreet instance before code went to production. With agile development, having a specific team in charge of all testing right before production is not the most sustainable approach, as it hinders the rapid pace of development that the cloud enables. This is why testing is “shifting left” to earlier stages of the development process, and why manual testing is being done by people embedded within each engineering team.

    Cloud-Native Teams Have Shifted Left Into CI/CD Pipelines

    As part of these agile methodologies, companies have moved toward leveraging CI/CD practices to ensure code is always in a state where it’s ready to be deployed. As code moves through these pipelines, developers want to ensure they are able to catch bugs earlier in the development cycle. To do this, they’ve taken to testing earlier and more often.

    In particular, developers have increasingly recognized the value of end-to-end testing throughout their CI builds. API tests allow teams to verify end-to-end workflows by chaining HTTP requests and API calls, so they can validate all the layers of their systems from worldwide locations. Browser tests capture the user’s view during critical transactions, so teams can understand how UI changes are impacting web applications. Shifting both left into CI/CD pipelines saves time and money, as fixing a bug earlier in the development lifecycle is much less expensive than trying to fix it at the last minute, when it has grown in complexity and scope.

    It also means many issues are caught by the developers, without the intervention of another engineer manually validating their work. This reduces the number of cycles required to find a bug, allowing the code to be circled back to developers, fixed and re-deployed to a testing environment. When regressions are caught early, developers can focus on the details that will make the product great.

    Visibility of Metrics, Traces and Logs Drives Value

    Running browser and API tests in your CI pipelines is beneficial, especially when teams can correlate these tests with data from across their stack – front end and back end, application and infrastructure. With correlated metrics, traces and logs that can be accessed easily without toggling between different products, teams will have the full context necessary for both detecting and troubleshooting failed tests. Detecting the failed tests allows for the rollback of bad code before it goes to production, and the correlated full stack data can be used to troubleshoot and fix any issues not caught during the development stage. This creates a new culture of quality assurance, in which maintaining quality is a holistic process that involves every part of the development cycle, and draws on data from every part of the stack.

    Consistent Quality Requires Collaboration

    This change in quality assurance goes beyond the practice of shifting browser and API testing left, and represents a fundamental change in how engineering teams collaborate within an organization. Testing happens before code goes to production, continues with robust monitoring once code is live, involves data from across the stack and is the responsibility of teams from various parts of the business. With this collaborative approach, for example, a front end engineer can easily identify an issue in the integration of their latest changes with the way the back end APIs are structured. They can then engage with the back end team owning this service to collaboratively iterate and find the best solution to resolve the issue. It may start with a shift left, but, ideally, it ends with the removal of silos and the opening of communication channels among teams.

    Automated QA Delivers Customer Value Faster

    A fully operational approach to quality assurance means that issues are resolved faster, or even resolved while code is still in development, so they never become issues at all. This creates a virtuous cycle – fewer regressions means less downtime, which makes customers happy, and frees up engineers to build new products, which makes customers happier still. That baseline of reliability and customer experience enabled by effective quality assurance is what allows for new innovation that delivers value to customers.

  • Are Developers Responsible for Open Source Governance?

    Are Developers Responsible for Open Source Governance?

    There are lots of factors in the open source software world converging to make it a big year for “shift left” in software development. Heightened security concerns, an increasing need for software supply chain visibility and the growth and complexity of open source ecosystems will continue to push the responsibility for ensuring code is secure earlier in the engineering process as far left as the request and design phases.

    This means 2021 will be yet another year in which developers are challenged to take on more responsibility, and pushed further onto the front lines of open source governance. Developers will need access to the right training, resources and tools to do this job. But they also can’t be successful if they’re expected to do it alone. Shift left can’t mean shifting all responsibility to the developer as the sole protector of secure code and the only go-to open source expert. The reality is that the depth, breadth and complexity of open source governance in 2021 demands a cross-functional team and layered approach.

    Equipping Developers With Governance Knowledge

    Let’s start with how to better equip developers and other stakeholders in the governance process, including legal, security, IP compliance and product people. These teams need enough knowledge to be aware of the top compliance issues, and their corresponding remediation options, to be active contributors in this rapidly evolving space. Research from Forrester shows that of the top 40 computer science programs in the United States and the top five programs internationally, none include open source licensing and security coding practices in the curriculum.

    Instead, it has fallen to the industry itself to impart this expertise on the job. Leading companies are packaging open source training programs into something that resembles standard HR-led and managed training, including check-ins and quizzes to test and solidify knowledge, and regularly updating content and scheduling trainings to make sure people stay abreast of changes and challenges. In the spirit of open source, many industries work collaboratively to share best practices on governance in this regard, with industry experts speaking on the topic at other companies. Others are leveraging existing management training courses, such as the ones offered by the Linux Foundation.

    The growing importance of the Software Bill of Materials (SBoM) also points to the need for cross-functional collaboration. Software supply chain visibility will become even more crucial in 2021 so that companies can build and sell secure and safe products, as well as ensure compliance. More people across the organization will need to access and view the SBoM in ways relevant to their functional needs. As a result, its assembly is transforming from being assembly-line-esque (passed along and viewed in the context of siloed business functions) to becoming much more collaborative. The SBoM is becoming akin to a business intelligence tool, providing the ability for different functions and stakeholders to filter for relevant information.

    Depending on your function, you may be interested in a top-down view of the open source and third-party components across your portfolio of products in order to understand risks in the organization. For instance, as an engineering leader, you may want to understand if you are impacted by a new security vulnerability, where in your portfolio of products you may be exposed and the current status of remediation work for that vulnerability. If you are in legal, you may be interested in a significant licensing change for a component in use at your organization. This allows you to be prepared with changes to legal policies for automated reviews and obligation fulfillment guidance for engineering when the next build occurs and a more recent version of a dependency is pulled in. If you are in the export control function, and are required to provide your application’s encryption capabilities as part of the certification process, understanding which open source components are part of the application, and which provide cryptographic capabilities is essential. Finally, don’t overlook your customers, who today are requiring open source disclosures as part of commercial software contracts to protect themselves from insecure software.

    Collaboration has also been made simpler by the fact that new industry standards now enable an asset portfolio view of open source security. Open Chain obtained ISO certification in late 2020, ISO/IEC 5230:2020, providing a clear and effective process management standard for open source license compliance. It allows companies of all sizes and in all sectors to adopt the key requirements of a quality open source compliance program. Coupled with ISO/IEC 27001–the de facto standard for managing security in information security management systems (ISMS) – organizations can now take a more holistic view of all of their software assets in an effort to make the information they hold more secure.

    Open Source Program Office

    The best way to ensure developers aren’t left footing the bill for open source governance is to create a centralized team dedicated to it. Savvy organizations could form an Open Source Program Office (OSPO), which brings together key stakeholders in legal, engineering, product and security to develop, communicate and operationalize open source strategy for their entire organization. This team works to define a framework and process for consuming open source software as part of their development work (inbound use case) and/or contributing code to the community via open source projects (outbound use case). The mission and main charter of these teams is to help their organization create and manage an open source strategy in terms of the adoption, use, support, participation and development of open source software. However, they’ve also found their role has evolved to become the experts on open source internally and evangelists of the technology and its use.

    The bottom line is that the developer’s job is to develop software – and every piece put in place for stronger open source governance means less time spent focusing on the problems that could be in the open source code and more time on the promise of using it to rapidly innovate for the benefit of their customers.

  • How Shifting Testing Left Can Boost DevOps Velocity

    How Shifting Testing Left Can Boost DevOps Velocity

    DevOps is all about balance: How can speed and quality be balanced to best meet the demands of customers who want exceptional digital experiences as fast as possible? DevOps looks to meet these expectations head-on by helping to deliver a faster cadence of new builds to customers while keeping quality high.

    What’s more, quality should continuously improve, build-over-build, so that fewer fixes are needed with each iteration. While inserting more testing earlier in the development life cycle may at first seem like a hindrance to the speed DevOps promises, it will soon empower teams to release higher-quality software, faster, as shift-left testing becomes a routine part of the team’s processes.

    Many organizations wait until the very end of their software development life cycle (SDLC) before executing testing. With this structure, testing can and does find unexpected bugs late in the SDLC, where issues are the most difficult and expensive to resolve. With this approach, testing becomes a bottleneck, impacting that ever-important goal of efficient and Agile software development and deployment. An unexpected outcome: it also negatively affects team morale, as developers who spent plenty of time and effort to build and ship code quickly now have to delay their deployment process in order to correct issues.

    There is a more efficient approach that organizations can put into place, however.

    Shifting Testing Left

    Testing can be implemented during and throughout the critical stages of design and development of software, instead of after.

    This is true regardless of how a team approaches development methodologies: Agile, waterfall, continuous or a combination of these methods. When testing is shifted left, it becomes the responsibility of the entire organization, not just dedicated QA testers. That means SDETs, developers and others should all be active in testing along with QA. When these groups are involved in testing early on in the SDLC, there are plenty of benefits, including:

    • A list of user workflows and test validation conditions can be created to streamline testing during development processes.
    • Testers can catch potential issues much earlier in the process so they can be fixed during earlier stages, instead of after code is done being developed.
    • Developers and testers can work with code they are familiar with to edit existing code and fix issues themselves.

    Building a Continuous Feedback Loop

    Another major advantage of shifting testing left in the SDLC is that it creates a continuous feedback loop. Software can be updated and improved continuously as it is being built, instead of when it is “complete” and ready to ship. This model of a feedback loop saves time and ultimately ensures no show-stopping issues make their way into the hands of customers.

    Test Automation in the SDLC

    One more key element to shifting testing left is test automation. Unfortunately, automation is often not leveraged completely or to its fullest potential. A recent report found that 74% of QA testers see test automation taking place outside the regular test cycle. Moreover, only 45% of organizations use test automation, and only 50% use it for continuous integration/continuous delivery (CI/CD).

    For test automation to succeed, it must involve developers creating the automation code and testers, then editing, maintaining and executing that code. When test development, including automated testing, is moved into the SDLC properly, it leads to automated and integrated unit testing, which helps achieve that goal of increased efficiency for DevOps teams.

    Automation is obviously crucial to the success of DevOps, but manual processes must also be considered. Some types of testing – like exploratory, payments, usability and accessibility – cannot be automated (at least not fully), but they should also be accounted for as part of a shift-left strategy.

    The origin of DevOps methodology and the focus of developers involves the goal of building and deploying code as quickly as possible. But testing software to make sure it works as intended is an important step that should be included throughout the SDLC. When testing only happens at the end of the process, it can be an inhibitor to the overall goal of increased development velocity. But when testing is shifted left and involved earlier on in the SDLC, it can do the opposite – by creating a continuous feedback loop and implementing automation to drive improved DevOps velocity.

    Ultimately, shifting testing left in the SDLC leads to features getting to users and customers faster, an overall improvement of software quality and a reduction of QA testing time and bug fixes.

  • Solvo Automatically Crafts Cloud Security Policies

    Solvo Automatically Crafts Cloud Security Policies

    Solvo today announced general availability of a namesake tool that automatically creates a least-privilege policy and applies it to application workloads deployed in the cloud.

    Previously available only as a private beta, Solvo will also continually analyze a workload after it’s been deployed to enable IT teams to adjust security policies as additional updates to an application are made.

    In addition, Solvo is making available for free the Solvo Security Genie, which IT teams can use to identify misconfigurations, excessive permissions, privilege levels and third parties that may have access to accounts on the Amazon Web Services (AWS) public cloud.

    Shira Shamban, Solvo CEO, said the primary reason there are so many cloud security issues today is because developers – that lack cybersecurity expertise – are being asked to define policies. Solvo removes that burden by analyzing a workload and creating a cybersecurity policy before applications are deployed in the cloud. That cybersecurity policy can then be automatically applied by the developer.

    Shamban said that approach will reduce a lot of the DevSecOps tension that organizations are currently experiencing as they try to shift responsibility for cybersecurity further left toward developers. While no developer deliberately sets out to deploy an insecure application, shifting responsibility for cybersecurity to a developer who doesn’t really understand what’s required creates a lot of additional stress.

    The Solvo platform, available as a paid subscription or under a freemium model for individual developers and teams, addresses a lot of the cloud security issues that inevitably arise when infrastructure is provisioned as code by developers. It eliminates that stress by automating the creation and application of cybersecurity policies within the context of a DevOps workflows, said Shamban.

    However, a cybersecurity team can also manually edit those cybersecurity policies after an application is deployed, Shamban added.

    As DevSecOps best practices continues to evolve, it is clear the relationship between developers and cybersecurity teams needs to evolve, as well. For a long time, many cybersecurity teams viewed developers as the primary source of the problems they encountered. Getting used to the idea that those same developers are now going to solve those issues on their own requires a level of trust that will take time to build.

    Developers, for their part, are generally accepting of the idea that they should take more responsibility for security, so long as the tools provided to them are not selected by cybersecurity professionals that don’t appreciate application development workflows.

    In time, there is no doubt the cultural divide that exists between developers and cybersecurity teams will narrow. However, it’s arguable that more progress will be made, more quickly, if developers are allowed to choose their own security tools rather than accept an edict from on high mandating a specific approach. In fact, the best course of action for many IT teams is simply to offer words of encouragement to developers that elect to set an example for the rest of their colleagues.

  • How Shift-Left Testing Improves DevOps Efficiency and Software Quality

    How Shift-Left Testing Improves DevOps Efficiency and Software Quality

    Shift-left testing is the key to deliver quality software at speed. Developers shift testing left to shorten long test cycles, prevent defects and mitigate risk earlier in the development cycle, rather than in later stages, to avoid rework and delays.

    In this interview, IBM Z DevOps Testing Lead Nathan Cassata talks about shift-left testing, the benefits of shifting mainframe application testing to the left and how IBM Z DevOps testing solutions help accelerate enterprise cloud-native development.

    Nathan is also one of the distinguished speakers presenting at Open Mainframe Virtual Summit. During his session, he will explain how to shift left testing for your traditional z/OS applications.

    To learn more about how testing is integrated into a developer pipeline and how to create real unit tests, attend Nathan’s session at Open Mainframe Virtual Summit on Feb. 11. To check out the full agenda or to register for free, visit the Open Mainframe website.

  • Shifting Left and Static Code Analysis with Perforce

    Shifting Left and Static Code Analysis with Perforce

    Perforce Product Manager Stuart Foster, and Evangelist Steve Howard, join Mitch Ashley to discuss the importance of creating security software from the beginning of the development process. We discuss shift left, SAST, source code scanning and other pursuits towards this goal.

    The video is immediately below, followed by the transcript of the conversation. Enjoy!

    Transcript

    Mitch Ashley: I’m pleased to be joined by Stuart Foster, who’s product manager with Perforce, and also Steve Howard, who is static code analysis evangelist, both with Perforce. Welcome, gentlemen.

    Stuart Foster: Hey, Mitch, how are you doing?

    Ashley: Very good. How about if we—let’s start off with just a little bit of an overview of Perforce, and then, Steve, why don’t you do that and also tell us a little bit about yourself, and then we’ll toss it over to you, Stuart, to give you introduction.

    Steve Howard: Right, yeah. Well, here at Perforce, really, our role as we see it is to help developers write better code more efficiently and more quickly. So, these days, we tend to talk about DevOps and the whole idea of developer operations and how they put that code together, how it’s constructed, how it’s managed, how we control the processes around it as well, of course. There’s more and more regulation, it seems, these days in software development and we’re making that whole process from start to finish, of writing code to delivering it, that whole cycle. It’s hopefully being made easier, more comfortable for our developers that use the products.

    So, that’s really where Perforce sits, and we think of that is really enterprise DevOps, and doing it at scale in large scale organizations.

    So, my role, really, is specific on the Static Code side or Static Code security testing side, SaaS tools, if you will, trying to make sure that that piece of writing the code, actually developers writing the code itself is done to the highest possible standard from both the security and the quality perspective and also, at the same time, getting maximum productivity from our development teams.

    Ashley: Wonderful. Stuart, introduce yourself?

    Foster: Yeah, absolutely. I’m Stuart Foster. I’m the Product Manager or Perforce’s Static Analysis products, both Klocwork and Helix QAC. So, I work with Steve, he’s kinda my partner in crime, amongst a number of other sales engineers and developers and everything, and we’re just out to produce a really solid static application security testing tool.

    Ashley: Excellent. Well, one of the great things about talking with you both is, since you have products in market, you talk to a lot of customers, especially in an evangelist role, too, Steve. It really helps us kinda get an idea of where the state of security is in applications, what people are doing today. The whole idea of shift left, I think, has evolved a great deal. I think when I first heard it was “move left in the workflow.” And then it became, “Well, move the when the security people get involved.” And you’ve really taken it much farther than that to really thinking about how we start writing secure code.

    Maybe one of you could kinda give us an idea of where you think we are as a community on getting to that point of when the code’s written, it’s written securely.

    Foster: Yeah, so, I mean, from my perspective, you know, you kind of look historically at people using the term shift left and, you know, from our perspective, that would’ve been, you know, doing some sort of code analysis on a server and then moving that left in terms of the software development life cycle into the developer’s IDEs.

    What you see nowadays is that, you know, historically, a lot of those responsibilities would’ve been siloed. You’d have a security team, a QA team, various other teams that, in many ways, because they’re siloed and in different stages of the process, you’d run into kind of bottlenecks. What you’re seeing a lot more with the concept of DevOps, DevSecOps, a lot of development teams themselves are becoming owners completely of the code that they write. And what I mean by owners is that, you know, they are writing unit tests, they’re writing more functional tests, they are ensuring that it’s part of their responsibility that the code is secure. 

    So, that’s a shift left, but then you have to think how developers work—they’re working in IDEs, they have their own CI/CD pipelines, they may be using cloud services, various containers that are all getting spun up and broken down. So, it’s not just, to me, shift left any more, it’s analysis anywhere. It’s basically having something that can work anywhere in a workflow, right? And in this sense, you know, the Security teams still exist, the Quality Assurance teams still exist, but what it really is, is a culture of security collaboration that are coming together, right?

    Steve, I think you may have a little bit of a perspective on this, too.

    Ashley: Yeah, I’m interested in your from the field perspective, too.

    Howard: [Laughter] Yeah, no, it’s—well, it’s exactly that. I think it’s a really good point that no, we’re not just talking about moving to the desktops, we are talking shifting left throughout the whole cycle. I think that is a key point. And it is really an opportunity using tools like ours in Static Analysis security testing, it’s about being able to enforce the requirements and the needs from that expert security team where there were bottlenecks previously, trying to put that into rules and regulations that we can then push down to the development tasks and processes that are to the left of that kinda final sign-off.

    So, yeah, that—in that form, very much, I’d say it’s shift left, and it’s about that optimized process now where we can be doing that checking all of the time. So, it’s making sure we’re continuously compliant to those security requirements or quality requirements, as the case may be, yeah.

    Ashley: You know, it’s interesting, we talk about the workflow and we think about the whole tool chain and pipeline, but if you really break it down into much more granular pieces, developers are working on a linear piece of development, right? They’re in multiple places jumping back and forth, solving bugs, adding features, developing new capability—whatever it might be. So, the ability to concentrate on any one aspect of an application, sometimes, is not even possible, right?

    So, you need an environment where you’re aided in how you write and create secure code as well as think about writing secure code. You agree with that?

    Howard: Yeah, absolutely. I think that’s the point. We’re trying to make sure that all of those pieces that they’re working on at different times and, you know, our colleagues in the Version Control System team know this very well, but obviously, there are tens or hundreds of branches running in parallel, as you’re sort of saying, and developers are jumping between, you know, fixing problems on the current release, maybe, and then carrying back on with their feature work for the next release as well. 

    And all this time to try and manage what security requirements each one of those had or quality requirements as well would be a really nightmare without sort of tooling and regulation around that, around the whole process, which is a significant part of developer operations as a whole, the idea of automating a lot of that and building it into the system. So, I think it’s just enforcing that with making sure we’re adding all of the extra pieces that we can also include in those processes.

    Foster: And also, if you think about those developers, too, they’re experts in their fields or experts on some of the functionality that they’re developing, but they’re not also necessarily experts as a security team might be, right? So, part of the trick, too, is to offer kinda help, knowledge, learning via the tool set so that, you know, the developers can keep the velocity up and develop to those standards more easily. 

    And I think Steve mentioned a little bit of a buzzword that kinda we like to think about is, it’s continuous compliance, right? It’s having this throughout the process and enforcing it and continuously monitoring, remediating, fixing, and reporting that become important for these processes to run really cleanly.

    Ashley: You know, it’s interesting, there was—kinda stepping back in time, and again, I think about the myth of developers don’t wanna, don’t care about writing secure code. Well, of course, they do. They don’t wanna write insecure, vulnerable code. But I think there’s a period where we thought training was—you know, let’s teach them about how to avoid writing code with vulnerabilities in it.

    But when you talk about a continuous governance environment, you’re kinda talking about a whole layer of things, right? It’s from the person doing the work to the environment that it’s being checked in and integrated and being tested into the workflow, and then really, the data, the information about that, that’s happening on a continuous, ongoing basis. And that’s really your data set for producing the compliance information, right? We don’t have to go ask you or check your code, it’s already been done, it’s done every step of the way all the time, and here’s the evidence of it, and here’s what we learned from it and here’s where we’re doing well and here’s maybe where we can improve. 

    That’s how I think of continuous compliance. Yes, it shows up in a report down the road, but it’s continuous, because we have all that data.

    Howard: Yeah, I think that’s a really important point from the efficiency as well. I mean, the Security team or the Development Management that want to know when this product or project will shift obviously need to know where they are relative to that compliance requirement. And if we can keep it compliant from day one, we don’t ever have that risk of building up technical debt at the end of the process that we then have to solve, and I think that’s a really important evolution that we’ve seen over the last few years, that we have that knowledge to hand immediately at our fingertips, as you say. So, that’s a really important piece.

    And the other point you made there about the developer education, I think, again—yes, you can teach developers and I know myself, in the past, I’ve learned the right way to do things, wrong way to do things, but there’s nothing quite as good as actually showing the examples of when things are potentially risky within the code you are writing, because you really understand, at that point, what you’re writing and how this could then be a problem, and it is the classic on the job experience that you really want for a perfect education, almost.

    Ashley: Let’s take—I’d love to hear your kinda definition or how you describe Static Code Analysis SaaS to people about, if you’re embracing that, what does that mean to a development team, to a software project team?

    Howard: I think from my perspective, it’s the idea of having an automated code reviewer, almost, looking at the work you’re doing in a fairly pleasant and passive way. [Laughter] Not being critical in any way, shape, or form as well. So, from a cultural point of view, I think the developers perhaps accept it more when it’s from a machine rather than a colleague that perhaps is picking holes in the code they’re writing, which is another good point about having a computer do that initial check.

    But yes, it’s like having that guard at the end of the process, and then we often joked historically about the value of the tools and saying, “Well, you know, if you had two developers that were both going for the job, you know, you’re advertising for a new development role, and one of those developers is, you know, the cost of the license is more, but they never make a mistake in the code, they check in clean, compliant code with every commit, whereas the other developer, they both have the same experience, they’re both writing good code, but the other one could make mistakes and could check in, potentially, security vulnerabilities. Which one would you pick as a manager?” That’s quite an interesting way of looking at it, but of course, you would always want to have that assurance that, essentially, it’s clean code from day one.

    Foster: Yeah, and also, you know, in terms of the perspective, Steve talked a little bit about the developer perspective or the hiring perspective. From the business perspective, you have to think about the costs of reputation, lives, revenue, anything of a product being in the field and discovering an issue. You know, the costs to fix a bug exponentially gets more expensive as you go further down the software life cycle, right? So, it’s cheapest to fix it then and there, right at the start. And, you know, in terms of kind of pain points that the software is trying to fix is, you know, providing that knowledge, providing that quick remediation as developers are working, ensuring the velocity stays up, ensuring that the continuous compliance enforcement throughout the process means that there’s not a heavy lifting of work later in the process, because you know, you weren’t reaching compliance requirements. 

    You know, the ease of using the software however a developer wants to work, it’s all about velocity, you know? And in terms of the business having that oversight and understand, you know, what are our risks, and how soon are we gonna be able to ship this product? Because, you know, we know with many products, the faster to market you get, you know, you have that first mover advantage, and that’s what’s needed in this fast-paced software market.

    Ashley: Well, that’s very tough to recovery from reputation damage and quality issues or perceived quality issues around your products, too.

    You know, one of the ways that I explain it to folks outside of being a developer or within the security world is, it’s kinda like taking your spell check of a document and adding in the grammar checking—so, this is the higher order, more complex, specific things around grammar. In this case, it’s around security and helping you as you’re developing or creating code, right, make sure that you do it the best way, you know? Avoid introducing security issues.

    I know that’s a gross oversimplification, so don’t worry, I’m not saying Perforce is like a grammar checker. [Laughter] But that is sort of a non-technical way of describing what we’re doing. I mean, we want it to be that natural, as part of the process.

    Howard: Yeah, absolutely. And I think that’s very true. I think, you know, we’ve seen the evolution of Static Analysis tools very much in the same way we’ve seen the evolution of things like the Microsoft Word, you know, spell checker to grammar checker and so on, and it’s kind of evolved to be more and more all of the time, you know, of the systems similarly.

    But on top of that, there’s also the opportunity, then, to, what we find quite interesting is to write your own checkers and rules that are specific to your application. So, we can find in some places where there have been security problems in a certain application for some special reason—not necessarily related to the constructs of the code, as most of the coding standards check, but something that’s specific to the project because of the way the project has designed all the code, and that project interacts with outside external interfaces. 

    And it has been interesting to work with customers in some cases. I worked with very large telecom customer on one occasion and he said, “Well, you know, we have this translation module between sort of the application and the platform, and nothing should ever, you know, go direct between the two unless they’re going through this one piece in the middle.” And it’s kind of the golden rule we’ve taught to all the developers from day one and nobody will ever break this rule.

    So, you know, obviously, I was slightly challenged by this and said, “Well, it’d be nice to check the rule. Let’s put that enforcement in just to see if we can, you know, use a custom rule to see if there are any cases and, of course, as you say, it should be clean.” And obviously, you know, a couple of hours later, once the analysis is run, it’s a very big system, we found a couple of cases—only a couple, but there were a couple of cases where exactly what they said should never, ever, ever happen for the security of the entire system, it happened. 

    And so, we were able to then remediate those problems and keep that rule in for future development, and of course, it will always be enforced. So, that was a really interesting case where there was also that, where you have something specific, it’s not just generic grammar issues, if you will, or spelling issues, but something very specific to that document.

    Ashley: Mm-hmm.

    Foster: Yeah, you can imagine some of these things coming out of requirements or an internal coding standard or things that a Development team might be following and be able to enforce these. Especially when you start looking at legacy code, giant monoliths, you know, that have been around for 5, 10, 15, 20 years, decades, you know, and you don’t have those original people who are developing, right?

    So, you know, it’s—we sometimes think of our tool also as, you know, kind of like a quality assurance on a product or a code base that, you know, it’s legacy and all the new people have no idea how to use it, but you need to expand on it, you need to develop it, because it’s valuable IP, and how do you manage developing new functionality and moving forward when, you know, it could be as brittle as a glass house, right?

    So, you know, that’s the other benefit you get out of being able to enforce and write your own standards, I’ll call them, and rules, yeah.

    Howard: Another classic case there, Mitch, is—and you’re well aware of this as well from the field is, this system was never intended to be connected to the Internet, you know? This was something that—

    Ashley: [Laughter] You’re guaranteed it will be, then.

    Howard: Yeah. [Laughter] It was an embedded system that was entirely supposed to run in an embedded environment with no connection to an outside world at all, and all of a sudden, you know, it or something around it that’s connecting to it is now connecting to the outside world and there’s open attack interfaces, and now we have to make sure that they’re not gonna compromise the original legacy code base.

    Ashley: Mm-hmm. Yeah, there’s just sort of the rule of unintended uses always occur, right? The thing we don’t want to happen is usually what happens. [Laughter] So, yeah, the demo becomes the production system is another great example.

    I’m really interested in your perspective on where you think we are. If I made an analogy to the network security world, the best time to be a network equipment or network security salesperson is the day of or the day after a compromise, and automatically, you’ve got budget and we need to solve this problem. I mean, I think we’re well past that in most cases, but where are we in the software world? Do you see this as a, it’s just a natural thing, “Okay, what’s our SaaS strategy, how are we gonna be doing security within the application, what tools should we use for that?” Or is it something where you have to mature to a certain point or have experienced some issues before you’re really gonna take it seriously? 

    Where are development—and I’m asking for a generalization I know, but what’s your sense of where people are today?

    Foster: I’d say, you know, in general, it seems as though most people don’t take these kinds of things seriously until it bites them. I think, you know, there’s some kind of safety feelings of some types of products that you can use where, you know, you find out about a security vulnerability or intrusion and you’re using a piece of software that basically just tells you that, you know, the barn doors have been opened and all your horses are gone, right? But that’s too late in the process. I think we all need to understand that, right?

    So, the point of dealing with it sooner and using tools like security application and security testing tools like Klocwork or QAC is that, you know, you’re trying to build the risk reduction, the known unknowns that you can at least deal with sooner, right? So that, you find out later, in the worst case, that the doors were open, but, you know, nothing happened, you still have your horses, because the stables didn’t open at the same time or whatever, right? Just to make a kind of analogy to it, there.

    Howard: Yeah, I think in the—obviously, we do a lot of work in the embedded space, and in that area, having seen it over the last 10, 15 years, I think, you know, there was a time where we were educating people a lot more about what it was, you know, what it was that we could do with Static Analysis to actually find some of these problems. Whereas I think—and there’s a lot more awareness now of the risks in the security field and, obviously, what Static Analysis can do to help. So, most people are starting to think about it if they’ve not already got it built in.

    So, I’d say it is becoming more of an expected insurance policy against those kind of things within the tool environments. And obviously, the other kind of driving force there is the regulation. So, we’re starting to see standards themselves creeping into the market like the new ISO 21434 standard for automotive, for example, where they’re actually going to require some level of looking at the security of the system and making sure that we’ve done some of those things correctly and, you know, probably that will include what we expect it to include, some level of Static Analysis and coding standards and so on and so forth.

    So, that stuff is really becoming almost regulated into the process just like functional safety regulation has been in that process from the beginning.

    Ashley: Definitely.

    Foster: Yeah, you’d imagine—

    Ashley: Sorry, go ahead.

    Foster: I was gonna say, just on that topic there about cyber security for automotive, you think about it, the cars are effectively rolling entertainment systems, right? So, there’s a lot of new kind of vectors for attack, right? An insecure Bluetooth module inside a car and its infotainment system that is also connected to the bus controller that’s driving all the data from every sensor on the car that’s controlling your fly by wire driving and your fly by wire steering and all those things, right? It’s very important—very important.

    Ashley: And usually, you have, I believe, you know, both cars and trucks that have starters can start by Bluetooth as one of the mechanisms. So, yeah, you have potentially the same attack vector, you need to be able to isolate that, and now you’re into the function of the engine versus the entertainment system, and there’s a lot of work going in, I know, to the automotive systems. They’re very complex. I mean, just as complex as many, you know, many business applications—maybe even more so because they’re integrated from so many different suppliers, and they are, of course, largely based on, built on embedded systems.

    It seems like embedded systems, though, are much more accepting of an update process, of keeping the software current and requiring some way of being able to update software in an embedded system or maybe making it an automatic process so it’s kind of touchless. Is that your experience as well?

    Howard: Yeah, I think so. I think that is coming into many systems now. I mean, there are certain manufacturers driving that kind of process—automotive, for example, is a prime case. And it is interesting, because if you go back 10 years, that just was never even considered. You know, once you kind of flashed that system, it was staying as it was until you climbed up the mobile phone master or whatever it was that the embedded system was doing and flashed it again with the latest version. So, yeah, it is, the over the air updates is creeping in, and that will lead us to a more continuous delivery type architecture with the delivery of the software as well, that we’ll be able to go to that.

    So, yeah, it’s gonna be an interesting time. And as part of that, I guess now the focus has moved slightly from the cost of fixing an issue out in the wild to just, we don’t want to have the issue because of the potential impacts on having that issue even for a day, you know? [Laughter] The idea that a car might be possible to take control of from an external source, for example.

    Foster: Yeah, and I think you’re seeing a lot more emphasis on the security around a lot of these embedded systems because a lot of them are so, you know, requirements driven, compliance driven. They have ISO requirements that are needed to be validated by external bodies to make sure that these things are safe to use, because you know, a lot of our infrastructure is built on the backs of these things, that they’re, in some ways, leading the charge of seriousness for these reasons, right?

    Ashley: Mm-hmm. And it seems, also, that industries like, I mean, we were talking about the automotive industry, are putting together their own standards, I think in part of fear out of, “We don’t want government standards to dictate to us, we’d rather us solve the problem, you know, industry solve it” in a way that they’re more accepting and flexible to them. But do you see more regulation coming at a kinda geographical country, government level? Is more of that happening? You talk about ISO standards, of course, being more industry.

    Foster: Yeah, I’d say so [Cross talk]—

    Ashley: You see it happening? Like, our privacy standards, now we’re talking about software security standards?

    Howard: [Laughter] Well, yes, there is that as well. Perhaps in the European level, for example, of something we see on this side, but yeah, mostly, it’s industry standards, from what I can tell that we’re experiencing and seeing come downstream at the moment.

    Ashley: Mm-hmm. [Cross talk]

    Foster: Yeah, I think it depends on the level of kind of sensitivity of the data. So, you do see requirements as they relate to finance and, like, PCI, DSS, and things like that. You see security in terms of—I’m running a blank, here, but they typically come from when the data is so sensitive, you know, the U.S. government has their STIGs, DISA STIG rules.

    Ashley: Mm-hmm.

    Foster: So, it tends to come down from the level of sensitivity. I wouldn’t be surprised if we do start seeing more requirements, perhaps, for public data, but right now, we have things like the HIPAA compliance and things like that, as they relate to very, very sensitive data.

    Ashley: Yeah, the defense industry is also a good one to start to drive standards, too, and I know they’re doing a lot in security.

    So, just to change, shift focus here, we have a few minutes left. We were talking earlier about where people are in their maturity level of adopting SaaS and writing more secure software using those kinds of tools. If you were someone leading a project, maybe an architect or a lead developer or somewhere in the management chain that said, “You know what, we’re almost there, I just need to do a little bit more convincing, or I need something else to help my organization get there to see why we need to do this, and I’d like to do it before we have a major incident”—what’s your best suggestion for kinda doing that? I don’t know if it’s internally selling or if it’s demonstrating the value of it or how do you see people successfully getting the organization to begin to support adopting it?

    Howard: Hmm. I think that’s a tough question, just because there’s so many industries and so many different kind of topics of particular interest, it always helps to have experience of a well-publicized kind of issue [Laughter] that’s gone out into the—

    Ashley: “We don’t wanna be like those people.”

    Howard: Exactly, yeah. That always helps drive things a little bit when you’ve got something tangible to say, “We don’t want to end up there.” I think, again, the regulation piece is where I see a lot of that being driven. You know, the industry bodies are actually saying, “As a whole, the industry doesn’t want to end up there. That’s not somewhere we can afford to be, and so let’s try and make sure we get this right from day one” and that trickles down into individual projects and so on. [Dogs barking]

    Ashley: Well, speaking of dogs [Laughter]—okay, hold on.

    Howard: Okay. Yeah, so, that’s really where we’re—

    Ashley: Okay, just start your answer again.

    Howard: Yeah, so, that’s really where we’re seeing it, predominantly, from the—you know, first of all, we don’t want a case where there is an incident, because, you know, if we can  use that as a tangible case and say, “We don’t wanna be like those guys, because that didn’t end well for them.” So, that’s, obviously, always a good driving force. But then it’s regulation from the perspective of the industry bodies themselves getting together and saying, “We really can’t afford for this kind of thing to happen, and we want to make sure that, you know, across the industry, we start to follow a best practice that’s gonna do our very best to prevent that,” I guess, is a good way of describing it.

    Ashley: It seems this is also an area where security team CISOs can be a real colleague asset for making the case to the rest of the organization, because if you’re _____ on the software team, you have one set of justifications or definitions for why you need that. Another is, there are corporate—you know, the CISO is trying to accomplish these goals, these things with the organization. And it’s always easier to fit your objectives within a larger objective and get that supported and get funding—that kinda thing. So, that’s another suggestion I would throw out there, too, that also helps folks—and plus, it brings security and software teams closer together.

    Howard: Yeah. [Cross talk] Sorry.

    Foster: Yeah. I think we do see—we do see, we do have, you know, security evangelists that are, as you just said, who are championing it, right? And when we talk to those people and we go over Static Analysis with them and this is the security focus, right? It comes down to, you know, being able to help understand their size of team, what kind of current development practices they have, and what that means and how you can deploy, roll out the tools and, you know, not slow down that velocity, because that’s always the goal. Don’t slow down the velocity, but add more value. 

    So, that’s what we try to help them with, help them understand. And also, you know, help—sometimes, you know, Steve is sitting with these people and we almost help them mature in their software development process to, you know, move from a couple guys who are just doing nightlies and they have no version control, they’re not using quality gates, they have no type of automated pipelines to setting them up so that they can scale their development groups, they can speed up their testing, their development processes, and that all comes out of engaging with vendors like us.

    Ashley: Great. Well, gentlemen, this was a lot of fun. I really enjoy talking security and creating secure applications and, you know, it’s something that I think everybody wants to improve at to different levels and degrees, and hopefully, before those bad things happen. It’s always better to do it before, but if you have to do it after, you do. Maybe sometimes that’s what it takes.

    So, it’s been great talking with you. Again, this is Stuart Foster, who is Project Manager with Perforce—thank you, Stuart—and Steve Howard, Static Code Analysis Evangelist with Perforce as well. Take care, gentlemen.

    Foster: Great, thank you.

    Howard: Thanks, Mitch.

  • Code Security: SAST, Shift-Left, DevSecOps and Beyond

    Code Security: SAST, Shift-Left, DevSecOps and Beyond

    Automation is the key to pushing code security beyond DevSecOps

    In virtually every industry, developers are dealing with ensuring the safety of their code. Regardless of whether it’s enterprise or applications for automotive, aviation or industrial controls, small systems or large systems, every organization struggles with security. In response, the industry continues to evolve to improve automation, speed of delivery and code reliability. Let’s consider the key building blocks of today’s application development ecosystem.

    Static analysis or static application security testing (SAST) helps enforce coding guidelines and detect undefined behavior, and is commonly used across the development ecosystem. One advantage of SAST is automation. Since it doesn’t require input, SAST is easy to introduce into an existing software development process where it will inspect source code builds and generate warnings. Coding guidelines such as MISRA, CERT C and others help avoid common programming mistakes. However, a second category, undefined behavior, is a bigger issue. That’s because undefined behavior introduces risk. For example, a null pointer dereference, buffer overrun or uninitialized variable are exploitable vulnerabilities. Depending on their severity, they typically will require issuing a patch for customers.

    Next, let’s consider DevSecOps, which spans development, security and operations. Although it is becoming widely adopted in enterprise markets such as financial services, markets such as automotive and industrial are lagging. Nevertheless, the emphasis on security is driving the DevSecOps push to make life easier for software developers and to bring them more in line with security and operations functions.

    For example, quick feedback cycles are extremely important in every industry. Consider a use case in which it can take up to an hour to deploy a newly developed feature or a fix once it is written. That’s unthinkable in today’s day and age when new source code must run almost instantaneously.

    One of the most important elements in DevSecOps revolves around a project’s branching strategy. In addition to the main branch, every developer uses their own separate branch. They develop their code and then merge it back into that main branch. A key requirement for the main branch is to maintain zero unresolved warnings so that it passes all functional testing.

    Therefore, before a developer on an individual branch can submit their work, it also needs to pass all functional tests. And all static analysis tests need to pass. When a pull request or merge request has unresolved warnings, it is rejected, must be fixed and resubmitted. These include functional test case failures and static analysis warnings.

    Functional test failures must be fixed. However, the root cause of the failure may be hard to find. A functional test error might say, “Input A should generate output B,” but C comes out instead, but there is no indication as to which piece of code to change. Static analysis, on the other hand, will reveal exactly where there is a memory leak and will provide detailed explanations for each warning. This is one way in which static analysis can help DevSecOps deliver the best and most secure code.

    Finally, let’s review Lean and shift-left, and see how they are connected.

    Lean development is focused on delivering value to the business. If we fix this defect, is the business going to get value? If we fix this warning, is the business going to get value? In other words, eliminate waste, make the right decisions, at the right point in time.

    Zero defects is actually one of the big underlying philosophies behind Lean. And of course, zero defects never means zero defects. It really means near-zero defects. For example, when there’s no value in fixing a particular defect, it can be classified as don’t fix. Meanwhile, if a defect exists such that an application will behave badly when a user inputs a negative value in the add function, there are two options: Fix it or don’t fix it, if the expectation is users will never input negative values.

    The objective is to achieve a zero-defect baseline and continuously improve. In this scenario, zero defects refers to source code problems as well as warnings from static analysis.

    Last is the concept of shift-left, which is a movement to bring certain processes closer to the developer—testing, in this case. By providing faster feedback cycles on new code, developers can avoid slowdowns. When working on source code projects, if regression tests run several days after a block is submitted and reveal defects, it can be difficult for developers to recall what they worked on at the time. Faster feedback that occurs in minutes, not days, improves quality and security. In this way, shift-left is a central component of DevSecOps and Lean. And static analysis is part of that shift-left.

    So what’s next? In a word, more automation. Making sure developers have as little cognitive overhead as possible will allow them to build code faster and more securely. This can be achieved by combining the concepts described above as well as analyzing code as soon as new functionality is developed using behavior-based testing or test-driven development. The ultimate goal is faster feedback cycles that accelerate development cycles.