devopsy.kinsta.cloud

Category: DevSecOps

  • Game On: Secure Coding Virtual Summit 2021

    Game On: Secure Coding Virtual Summit 2021

    Agile organizations are looking for ways to win at security without slowing down their release cycles. That’s where DevSecOps comes in. The days of a hands-off security approach for developers are over. Organizations implementing DevSecOps to integrate security into their DevOps framework are able to release secure software faster. Developers test code for potential security vulnerabilities earlier in the pipeline to fix security issues in real time to avoid costly delays.

    At Secure Coding Virtual Summit 2021, industry-leading AppSec and DevSecOps practitioners, analysts and visionaries will come together to share their best pro tips and power ups to level up your code security. The agenda for the virtual event, taking place March 24 from 10 a.m. to 6 p.m. ET, features a mix of topics including AppSec, threat modeling, API security, container security, case studies and more.

    The most brilliant security luminaries from OWASP, Okta, Microsoft, Secure Code Warrior, WhiteSource and many other well-known IT companies, will discuss:

    • How to level up your secure coding strategies
    • How to run an AppSec program with open source projects
    • The best practices for securing your APIs, the attack surface of an API and common authorization problems
    • The concept of threat modeling, its objectives and its role in the SDLC
    • How to transform developers into security people
    • How easy (or not) it is to get a container or Kubernetes cluster hacked on purpose
    • How to create an AppSec pipeline with containers in a short period of time (demo)

    This year’s Secure Coding features outstanding industry leaders, such as:

    • Vandana Verma Sehgal, global board of directors at OWASP
    • Simone Curzi, cybersecurity principal consultant at Microsoft
    • Keith Casey, API problem solver at Okta
    • Matias Madou, CTO and co-founder at Secure Code Warrior
    • Matthew Coles, senior principal product security engineer at Dell
    • Sarah Young, security architect at Microsoft
    • Chris Romeo, CEO and co-founder at Security Journey
    • Dr. Phillipe de Ryck, web security expert and founder at Pragmatic Web Security
    • Richard Greenberg, founder and CEO at Security Advisors LLC
    • Izar Tarandach, principal security engineer at Squarespace
    • Jeroen Willemsen, principal security architect at Xebia
    • Shiri Ivtsan, director of product at WhiteSource
    • Justin Hutchings, director of product management at GitHub
    • Matt Tesauro, senior partner at 10 Security
    • Matthew Butler, principle engineer at Laurel Lye

    Attendees will have access to useful resources and will be able to interact with sponsors to power up their secure coding strategies. There will be live Q&A chats with keynote speakers to get your burning questions answered. Attendees will also have the opportunity to hunt for Easter eggs to boost their player score for a chance to win a PlayStation 5.

    To see the full agenda and to register, please visit the Secure Coding website.

  • GitLab Releases Updates to CI/CD Platform

    GitLab Releases Updates to CI/CD Platform

    GitLab this week made available a 13.9 update that adds more than 60 new features and updated capabilities to its namesake open source continuous integration/continuous delivery (CI/CD) platform.

    Brian Glanz, a product marketing manager for GitLab, said chief among those capabilities is a Security Alert Dashboard that enables DevOps teams to triage high-priority alerts more easily. As more organizations embrace DevSecOps best processes, application development teams need to be able to prioritize which security issues are more pressing than others, Glanz said.

    Other new capabilities include the ability to reuse a pipeline of a CI/CD configuration, and a maintenance mode option that keeps instance available while administrative tasks are being performed.

    Additionally, it’s now possible to view group-level analysis of release metrics, in addition to the ability to view a version of a pipeline configuration with all of the include and extends configurations merged together.

    Glanz said GitLab continues to gain traction, because all the capabilities it provides are based on the same integrated code base. The effort a DevOps team needs to exert to take advantage of new capabilities is substantially less than rival platforms, Glanz said.

    At the same time, Glanz noted that the number of contributors to GitLab coming from outside the company also continues to increase. The latest version has 299 merged contributions from the wider GitLab community.

    It’s not clear how much of an appetite organizations have, these days, to replace an existing CI/CD platform. However, it’s not uncommon for organizations to have multiple CI/CD platforms employed by different DevOps teams. In the meantime, the number of teams within enterprise IT organizations that are embracing DevSecOps processes to advance digital business transformation initiatives continues to rise. As more organizations realize how dependent they are on custom software to differentiate themselves from rivals, more enterprise IT organizations will opt to standardize on a single CI/CD platform to both reduce costs and increase overall efficiency.

    A report published last year by GitLab noted organizations can improve application development and delivery by 87%, resulting in a savings of more than $23 million. Overall, the report claims organizations can see a 407% return on investment (ROI) on the GitLab platform.

    Naturally, competition among CI/CD platform providers remains fierce. With most developers working from home to help combat the spread of the COVID-19 pandemic, these platform providers have been touting their ability to support remote work. It’s not clear if developers will be returning to the office en masse now that COVID-19 vaccines are being distributed. However, it’s clear that even once the pandemic subsides, the way application code is created and maintained will be a lot more fluid. Many organizations are now more comfortable hiring developers wherever they can find them.

    Of course, many of those developers are going to be wary of joining organizations that have disjointed workflows, especially when it comes to shifting responsibility for application security further left toward developers. In fact, the ability of any organization to effectively recruit developers is now directly tied to the overall sophistication of their DevOps workflows.

  • Application Security and Continuous Observability with DeepFactor

    Application Security and Continuous Observability with DeepFactor

    Today’s modern applications are becoming increasingly complex and released more frequently than ever before, therefore increasing security, privacy and compliance risks. How do you identify the security and compliance risks hidden in your application before you deploy it to production?

    Continuous observability enables DevSecOps teams to observe millions of application telemetry events and find runtime security, compliance and privacy risks within the DevOps pipeline, while providing AppSec teams the ability to set guardrails and have continuous visibility with every build.

    In this TechStrong TV interview with DeepFactor Founder and CEO Kiran Kamity, we discuss the concept of observability and how to apply it for security and compliance. We also talk about the increasingly important role of AppSec and DeepFactor’s new Observability-as-Code API, which enables DevSecOps engineers to leverage observability functionality in their CI pipeline and gate builds.

    The video of the conversation is below, followed by the transcript. Enjoy!

    Transcript

    Mitch Ashley: I have the pleasure of being joined by Kiran Kamity, who is founder and CEO with DeepFactor. Welcome, Kiran.

    Kiran Kamity: Thank you, Mitch.

    Ashley: Great to have you. We’re gonna be talking about observability, super-hot topic, but before we get there, tell us a little bit about yourself, a little bit about your background, and then a little bit about DeepFactor.

    Kamity: Awesome, yeah. I came to the Bay Area about 20 years ago. I did my undergrad at the Indian Institute of Technology in India in Chennai, and then I came here to Stanford for my grad program. Once I came to the Bay Area, you get addicted to the entrepreneurial spirit, you never leave, this becomes your home. Happened to most people—happened to me as well.

    So, the Bay Area is my home, and I’ve had a mix of enterprise and infrastructure cloud computing, virtualization, and DevOps/AppSec kind of background over the last 20 years. I’ve done two companies as a Founder, which were acquired by other companies like Citrix and Cisco. My first company, RingCube Technologies, was acquired by Citrix. We built a container run time for the Windows Operating System before the Docker days. This was acquired by Citrix in 2011, so I was responsible for some of the product teams at Citrix after the acquisition.

    Then the bug bit me again, and I started ContainerX, which was a turnkey containers in a box type company, which was acquired by Cisco, and it’s now called Cisco Container Platform. While at Cisco, I was Head of Product for the Cisco Cloud Business Unit, which included container—container networking, OpenStack, and some of the other initiatives that they had.

    During that time, as I was looking at what to do next and analyzing the market, a lot of the learnings came to the following. There’s a lot of momentum that has happened so far in cloud native architectures, DevOps, et cetera. And the learnings were that the next wave of innovation is going to be as DevOps becomes DevSecOps. And that’s why I started DeepFactor, and we believe that the next wave of innovation in the world of DevSecOps is observability.

    Ashley: That’s something I both relate to and also am just fascinated by other entrepreneurs is that story of, you’re working on what you’re doing now and you’re still kinda looking at, “Well, what’s the next thing coming up, and what might I go do next?” and then you go off and solve the next problem.

    Kamity: Yeah, absolutely.

    Ashley: Start another company to do that—like you said, you get the bug, it’s kind of a fever. [Laughter] It definitely is.

    Well, cool. We’re talking about observability, and it does seem to be the hot topic of kinda late 2020, early 2021. So, let’s first start with kinda how did we get there—get here? How did we get to this place where suddenly observability is a thing and not only a thing, but important to a lot of people?

    Kamity: Yeah. Just to qualify it a little bit, our focus at DeepFactor is applying the concepts of observability for security and compliance, which is truly novel, and there’s very few companies that are trying to do that.

    Ashley: True. Observability can mean just sort of the next elevated monitoring, right?

    Kamity: Exactly.

    Ashley: So, you’re [Crosstalk] around security.

    Kamity: So far, a lot of the observability plays have been in the logs metrics tracing. Applying the concepts of observability, truly as the next generation of evolution of APM. But at DeepFactor, we’re leveraging the concepts of observability for identifying security and compliance risks in your applications before they go to production.

    So, to answer your question about how did we get here, let’s take a step back and analyze what has happened over the last few years, with respect to application development, right? Apps have become a lot more complex than they have ever been today. I mean, think about it—a common application is made of a bunch of different languages. You have a Java service, you have a Node.js service, and you have Angular in the front end, you have lots of different pieces, you have a database, et cetera.

    You have a lot of third parties—third party libraries that are baked into your services, third party components that are not built by you such as Nginx and MySQL and MongoDB, et cetera, that become part of your application.

    Ashley: A lot of open source, right, in that stack.

    Kamity: A lot of open source. Applications are now also being deployed in a bunch of different ways. You have applications that are not containerized, still mainstream. You have containerized applications, you have full on Kubernetes applications, and you have land applications as well, so the deployment of applications has become complex as well.

    So, in this world of applications becoming a lot more complex than they’ve been, what essentially happened from a security perspective is that the attack surface area of your application has significantly increased. It is the highest it’s ever been so are, and it’s only gonna go that way.

    Let’s take a look at a few other trends that have bene happening in industry. The velocity of releases—that has gone up as well due to DevOps. You’re no longer building or releasing an application once at three months, once at six months. You’re releasing that multiple times a week with DevOps, and different companies are different aspects of that spectrum, there.

    Now, when things are more complex and they’re being released at such a rapid pace, there is no way humanly possible for your AppSec teams to find all of the security risks and compliance risks that lie in your application before they deploy to production. At the same time, the risk of these issues being in production is higher than it’s ever been. If you think about it, the sophistication of the bad actors has increased significantly as well over the last few years, and therefore, the nature of the breaches, the size of the breaches, and the damage that they can cause has increased as well. Coupled with that, privacy concerns have increased, resulting in standards like GDPR and CCPA and all of those things emerging as well.

    Now, in such a landscape, you’ve got to automate identifying the security and compliance risks that are hidden in your application before you deploy your application to production. And that is where the current tools of just code scanning or identifying issues based on signatures or just DAST scanning your application, et cetera, don’t cut it. You need something that watches your application while the application is actually running in both pre-production environments as well as in a canary deployment of minimal production deployment before you release it to your broader production deployment.

    And in that environment, you’ve got to be able to watch the millions of events that are happening inside your application, take that data, analyze and find the needles in the haystack, and therefore determine where the security and compliance risks are and pass on all of this information with rich evidence such as stack traces, metrics, and whatever it takes to fix these issues back to the developers. And a lot of this has to happen in a seamless manner that is nicely tied into your DevOps kinda pipeline, and that’s where observability applied to security and compliance comes in, and that’s exactly what we call continuous observability at DeepFactor.

    Ashley: Interesting. I know, talking with you previously and kinda hearing about DeepFactor, you all talk about start left, not shift left—this idea that, really, it starts from the beginning, not, “Let’s kinda push it as far left as we can.”

    Kamity: Yeah, absolutely. If you think about it, we’ve had operator specific tools emerged in the last five years with respect to security. So, you’ve had container security tools that identify issues in your Kubernetes configuration or your Docker configuration or vulnerability scanners that identify issues in your host operating system if you’re deploying your app in a VM.

    Now, those are all the types of insights. Are there any misconfigurations that could lead to security issues that are very relevant for the Ops people? And those tools are coming up with the idea of shifting left, because you need to find these configuration issues before you go to production so that the Ops people can deploy these applications properly.

    But when it comes to developers, their agenda is not securely deploy your application. They don’t care, honestly, about how you’re Docker or Kubernetes is configured, for the most part. They care about what is in their application, securing at the source. And when the agenda for Dev teams is to secure at the source, then all you care about is—what is my application doing, whether it’s my application or some third party that I brought in, or just my interpreter itself when it is running? And can I identify the risks involved and understand my attack surface within the application ahead of time? And that’s why we call it start left.

    So, we think that operators care about shifting left. Developers should truly care about starting left.

    Ashley: Yeah, it’s interesting, because your point about code scanning and DAST approaches to security is, from a code standpoint, you do wanna be writing a secure code, so it’s a valid thing, of course, to be doing. But you mentioned Lambda, you mentioned, of course, infrastructure as code, essentially, where all this is dynamic, right, in our environments today, and more so, not less, every day, right?

    Kamity: Yes.

    Ashley: Every day, it gets to be more dynamic. Now, you’re really talking about the whole context of the environment with the application. Kind of, if you thought about it in the network, you’d say all seven layers, right? If you think about it in the security stack of software, it’s application plus all of the other things, right?

    Kamity: Yes.

    Ashley: It’s interesting what—how do you find people are accepting this idea of run time information being provided with context, you know, this rich information you were talking about, is that something new to developers or like, “Oh, my gosh, I’ve been waiting for this forever” or, “Well, this is really cool. Look at—let me see what I can do with it”? What’s more the experience that developers have?

    Kamity: Yeah, so, I think different organizations react differently to something like this. The old way of thinking is—oh, developers don’t care about security or compliance.

    Ashley: I’ve never met a develop that wants to write insecure code or unsecure code. [Laughter]

    Kamity: Exactly, right? That is exactly what’s happening now where, given the consequences of shipping insecure applications and given the need to be more and more secure at the source, I think developers—and just simply because it’s just not humanly possible for your AppSec teams to identify all these things. Given the talent shortage of the AppSec skill set as well, it is, the responsibility if identifying these risks has to be borne by the developers. Because at the end of the day, the developers are the ones that are writing the code and you’ve got to make sure that you’ve shipped secure code.

    Engineering leadership and AppSec teams in various organizations that we’ve been working with have been fully accepted. In fact, they’re fully aware. In fact, a lot of them want this kind of concept, it’s just the lack of tooling today. Because if you think about the current tools that are available, they’re doing the best they can with the existing set of tools.

    I mean, they’re scanning the code, they’re trying to scan the application from time to time—but those tools don’t lend themselves well, nicely into seamless integration into your CI pipeline, and therefore, giving developers the insight about, “Hey, here’s where we saw your application did this.” That can only happen with observability tools like what we’re building, because we watch every thread of every process of every service in your application when it is running in test and even canary deployment in your part of production before you deploy that thing widely to a larger section of production. And that is the kind of insight developers do care about.

    What we have gotten repeated feedback as we have been building this product with a few development partner type enterprises so far is that, the insights that we have needs to be enriched with richer evidence information so that developers can actually take that evidence information and directly go fix the problem. So, we’ve augmented our insights with stack traces wherever appropriate. You know, obviously, the good old things like CVE numbers and things like that, we have that.

    But what we focus more on is not just giving you a list of CVEs, but complementing the insights that your software composition tools give you by giving you run time usage information of that CVE. For example, we can tell you if a tool like Snyk will tell you there are 250 vulnerabilities in your application, DeepFactor will tell you, “Hey, there are 250 vulnerabilities according to what Snyk is telling you. Fifteen of those things have been used by your application at this time, when your application was executing here, ad here is the function, here is the call stack.”

    So, that will make it easy for a developer to go, “Okay, now I don’t have to get overwhelmed with fixing all of the 250 things or adding that to my backlog, I can actually prioritize which ones to fix first, because that’s what my application is actually doing.” It’s not about the whole thing that has been checked into your source code, it is what your application’s actually doing that will help you catch the more important things. It’ll help you also reduce the number of alerts that can inundate you.

    Now, with all of these changes, the rich evidence, reducing the number of alerts, and to the point, alerts based on what your app is actually doing, developers are finding it more and more meaningful, our insights. And as we grow the company, that’s gonna be our primary focus.

    Ashley: That’s always been the challenge with vulnerability scanners and code scanners is, yes, I get all of those results. What of those are real? Yes, they’re all theoretically possible. I may not be using it in a way that actually exposes me to that CVE, that vulnerability.

    But what you’re saying is, you’ve actually detected the use in the application or the software stack of something around that CVE that’s worth looking at. Hopefully, you’re cutting down on the number of things that are actually, they go look at. You can ignore the rest, if you will, or at least know that they’re much less likely to be an issue.

    Kamity: Yeah. We do not give you too many alerts. We give you the alerts that are relevant for your application because we watch the millions of events that are happening in your application in every part of your application, across various layers of the stack—like, system caller interface, library calls that your application has made, network behavior, web and API, all of that.

    And based on that data, we treat it as a data problem. So, we take that data and then we try to identify anomalies and detect that these are the risks that lie in your application from a security perspective, and this is how it maps into compliance. So, we will tell you that these are the security risks in your application and in our next module of the product, we’re gonna tell you that this can hurt your PCIDSS, and here’s the number, basically.

    Ashley: Mm-hmm. It definitely is gonna improve your relationship with the security and compliance team, right? If you’re solving these problems before it gets to them, they have to ask, “Well, did you do anything about this? Are we susceptible?”

    Kamity: Yeah.

    Ashley: Talk a little bit about—so, where in the tool chain and the workflow pipeline, where do you fit? Where does DeepFactor play?

    Kamity: Yeah. The best way to think of us is, if you take a step back, if you are thinking about your DevSecOps initiative in total, you want to plug in into a bunch of different points from a security perspective into your CI pipeline. Your developers write code. The next step should be you try to find risks in your code. That’s where the SaaS tools come in, they find the risks that are there in your code.

    Then you create your build artifacts—let’s say, your executables, your container images, and whatnot. Then you try to find risks that lie in your build artifacts. That includes your code, any third parties that you’ve compiled together and created your services. It’s still static, though. You’re not running them yet. Tools that allow you to identify risks in your build artifacts could be container scanning tools, like image scanning tools that scan your container images and find issues or software composition analysis tools like Snyk and Black Dock and tools like that. These tools find vulnerabilities that exist in your components after they’ve been built.

    Then you run your application, you’re testing your application. Either you’re testing it in a test environment or you’ve deployed and you’re testing it in a minimal configuration of your—like, a canary deployment in your production. During that time, you’ve flicked DeepFactor on. DeepFactor doesn’t require any changes to the code of your application; you’re simply running your application.

    And that’s where we’ve built a significant amount of IP. You know, my Co-founder, Mike Larkin, he’s the CTO, he’s the father of the OpenBSD hypervisor. And our senior engineering leadership members came from Qualys and FireEye, having been in the area for 20 plus years working on security products, each.

    So, we’ve created something that involves zero touch, pretty much. You don’t have to touch your application at all, you simply run a command called DFCTL, just like KubeCTL, we came up with DFCTL. So, you do DFCTL, run your app; or DCFTL, run your container; or you drop it into Kubernetes. We have a webhook admission controller that you simply drop into Kubernetes, and then any pod that you want to be monitored, you simply can specify a config file and that pod gets monitored automatically with DeepFactor.

    Now, the way DeepFactor inserts itself is completely behind the scenes. It’s language agnostic. You don’t have to worry about, is it Java, is it Node.js—DeepFactor figures out that it’s Java or Node.js and does some specific things and finds out more information.

    So, you simply run this, your application is running in your test environment or your minimal production environment, DeepFactor observes everything that it’s doing. When the tests are done, it tells you—you throw your automated tests at it while your application is running, just like normal. And then when it’s done, DeepFactor gives you all of that information about what insights have been found and what’s the evidence information that’s pointing to the insight. Is the issue or the risk presenting the attack factor lying in the code that you wrote? Is it lying in a third party that you brought in, or is it at the interpreter level itself? All of them are risks as far as the attacker is concerned that could be exploited, so you need to be aware of where your risks are.

    Ashley: Now, are developers putting in some library calls or DeepFactor code into their application stack as well, then? Is that how you’re getting into the software, knowing what’s happening inside as well as outside?

    Kamity: Actually, that’s the beauty of our IP, and we’ve filed a few patents in that area. Developers do not need to write any code to change, or change anything in their application. You simply enter DFCTL and run your app and DeepFactor inserts itself into the application at your run time, completely seamless to the developer, and when it’s done, it unplugs itself and your application and gives you all the insights.

    Ashley: Wow. That sounds pretty cool. I can imagine that’s a big plus for developers. [Laughter] Yet something ls they don’t have to debug, right?

    Kamity: Exactly, yeah. That’s the significant part of our IP.

    Ashley: You know, it’s interesting you mentioned your team comes from a number of other security companies that have been around for a while—you know, you mentioned Qualys and others starting in the scanning, the vulnerability management space some time ago; that’s about when I got into security, too.

    What do you think we’ve learned from those types of companies and those experiences that we’ve sort of changed our perspective about where we are today, just given—yes, we have much more complexity in software, much more porous security issues, but what have we learned from all of that security scanning that’s helping us?

    Kamity: Yeah, no, I think there have been numerous, numerous learnings, and the learnings have evolved over time as the landscape of application development has changed as well. You know, you’ve had tools in the past that were designed based on the applications of the past. But tools of today and tools of tomorrow are different, because the applications of today or the applications of tomorrow are definitely trending in a different direction.

    One example of the thought process itself is that, you know, when things are in single digits or a lesser number, it’s okay to have manual scanning, manual testing, once in a while test it and then find information, which is kinda the old kind of scan once in a while, identify it, or bring in a pen tester once in three months, you know, check some boxes and get it done kind of thing.

    Those are all great and they’re still relevant even now. What has become even more important is why you do those once in a while scans and checks. You’ve got to make sure that you get in the habit of securing at the source. You can no longer imagine that your infrastructure or the perimeter is gonna secure your application properly. You’ve got to make sure that you build your application securely to start. And that fundamentally has changed the kind of tools that we need to build in order to equip the developers who build applications to secure at the source.

    At the same time, I think the role of AppSec has become even more important than it has ever been. Because, you know, sometimes I keep getting asked, “Oh, are you saying that developers should use these security tools? That means the AppSec teams are no longer relevant?” That is totally wrong, because AppSec teams, their job is increasingly more relevant. Because if you take a step back and you look at the number of attack vectors that are there in your application, we just talked about how your pipeline should be architected—secure at the source, making sure that you scan your code, scan your builds, test your running applications, and identify risks in all of these areas—who ties them together? Who comes up with this overall strategy? Who comes up with that triaging process and passing the evidence back to the developers, identifying which ones are important and which ones are not?

    All of that should come from the AppSec person in the team—AppSec or if AppSec is assimilated into your DevOps and we call it the DevSecOps person, that’s the role of this person is to make sure that he or she is the shepherd of your CI pipeline from a continuous security perspective and then feeds all of that information back to the relevant parties. Some of that could be your developers, some of that could be to your engineering leadership, and some of that could be to your security leadership.

    Ashley: Mm-hmm.

    Kamity: That is how I see the organizations of tomorrow shaping up, learning from what has happened over the last 15 years to 20 years of innovation in this area.

    Ashley: I think that makes a lot of sense. Otherwise, (a) we’re just throwing tools at it, everybody’s doing whatever and who knows whether that’s helping us improve and improve security. And you mentioned the data coming out of it, right? That can also be used for continuous improvement, continuous learning, what are trends that are happening in our application?

    Suddenly we’re seeing much more of this technique or this architecture or this environment being deployed that, you know, not always do the security people have exposure to that early on in the process. So, it’s a good way just to know what the heck is goin’ on and then also, you have an idea about how all of your security tools, developer or non-developer, fit into an architecture that makes sense.

    Kamity: Yeah, exactly. I mean, imagine flipping a switch and identifying, “okay, these are all the touch points that my application has and these are the millions of things that it’s doing” and condensed very well to, “these are the red dots in that whole story, and this is where your risks lie.” That’s powerful, and I think it’s the AppSec person’s job to set all of that up and make sure that a framework is created and then say, any time something that comes up, you know, make sur that you pass it on to the developer with the appropriate evidence and work with the engineering team to decide.

    Because when you have so many different services and so many different languages, the AppSec people, especially because they’re already short staffed, cannot possibly know everything about the business logic of the application, resulting in those decisions that could have been made to bring in component X and component Y and whatnot.

    Ashley: Mm-hmm. You know, Kiran, one of the things that I’ve said over time is that security people need to get a greater understanding of software architecture. They don’t need to have to be developers, right, that’s not what their skill set is, nor necessary their desire. But you can’t treat security as this hard outer shell, you know, we just keep making it smaller and smaller. And now all you’ve got is thousands or hundreds of thousands of small outer shells and now you’re not securing anything, any more.

    It seems like the AppSec team can play a really important part of helping bridge that gap of what are we doing specifically, what are the architectures that we’re using, what might be some of the security risks that maybe a security engineer is just not fully up to speed on or trying to keep up to speed on as it’s changing?

    Kamity: Yeah, absolutely. Imagine the power of flipping the observability switch and the AppSec person going and saying, “Hey, Dev team, we noticed that in this particular piece of your code, you’re using an API that has been deprecated by the operating system” or, “In this particular piece of code, you’re invoking memory improperly, because this tool is showing me exactly what was done when this particular area of memory was allocated with the wrong permissions. Or, “This is an area of code where your interpreter is doing something wrong. Is it possible for us to go change the version of the interpreter that we’re using, maybe upgrade it to the latest version or put a patch?” Or, “Do we need to, in order to mitigate this thing, assuming that there’s no fix in the interpreter, do we need to go put something in our WA in our infrastructure to block the particular attack from happening?” Or, “Should we report this problem upstream to Java itself, the Java community itself, so they can go fix the problem in there?”

    So, lots of different things that could be done once the AppSec team is empowered with this much rich information about what’s happening in the application.

    Ashley: You know, something I don’t know that we talked enough about, we talked about how rapidly and how quickly we’re delivering software applications now. The environment of the software, the API calls we’re using, the libraries, the services, interpreters—all of that is changing in parallel very quickly. Sometimes you don’t feel like you have time to keep up with it, or that’s all you do is spend your time keeping up with that, not kinda delivering business value.

    I think folks that are maybe not involved too deeply in the software creation process really realize not only the complexity but how much change there is to deal with. There’s a lot that goes into that. We talk about technical debt or backlog or whatever—it’s a lot more than that, there’s much more to it. [Laughter]

    Kamity: Absolutely, there’s a lot more than that, and I think having an informed security team as well as, you know, coupled with a structure and the right tools baked into your pipeline, which results in informed Dev team, can result in goodness overall. You’ll end up identifying security and compliance risks earlier in your Dev cycle. You’ll end up fixing those things earlier, and therefore, you will end up shipping a much more secure application, and it’s a cycle. Every time more developers check in code, you know, this kind of identification happens automatically, and you end up identifying more and more risks and fixing more and more things, resulting in more and more secure applications into prod.

    Ashley: Mm-hmm. Awesome. Well, what are some things folks might have access to you can help them get started? Obviously, they can download HIV testing product or get involved that way. Do you have some other tools or things that they might be able to use to make it approachable to get into this kind of an approach?

    Kamity: Yeah, they can go to DeepFactor.io to check out the product, understand more about the product. And if they want to get started, they can either download the community edition that is now free for 10 user teams or less. If you need any help with setting up the product, you can reach out to our Customer Success person.

    We’re also giving away a free DevSecOps Risk Assessment to companies that are working with us as we expand our organization more. This will help any engineering team or any security team to get a free assessment of what DeepFactor can find in their applications so that they will be able to identify, “Okay, these are the kinds of things DeepFactor finds, and it’s useful for me and here’s where I need to go next” kind of thing.

    So, that DevSecOps Risk Assessment is completely free. Contact us, hello@deepfactor.io or just go to the website to learn more.

    Ashley: Awesome, great. I know you’re also speaking at the Cybersecurity 2021, I think you have a couple of sessions there, right? The dates are March 4th and 5th for that event.

    Kamity: Yeah. We have a couple of speaking sessions at Cybersecurity ’21. We have a virtual booth there as well, so do check us out. There is a—you know, our team can answer any questions or even try to explain to you or help you understand what DeepFactor can do and identify in your application.

    Also, today, we’re launching Observability-as-Code. It’s a new module that we’ve created, and a product that allows people to bake in observability into their CI pipeline, similar to how Infrastructure-as-Code enables DevOps and DevSecOps folks to spin up, spin down infrastructure in their CI pipeline itself and other related scripts. You can do, now, the same thing with observability for security and compliance with DeepFactor. So, you can use our API, we have a Swagger doc ready, you can use our API to invoke DeepFactor as part of your CI pipeline, get the results, triage the results, use that results to block, if there’s any P1s you want to block the _____, you can do that. You can do things like that with the Observability-as-Code API that we just announced.

    Ashley: Excellent. Well, I hope folks will check you out. I think it’s a great path to be going down, and I’m sure developers appreciate writing more secure code, but completing an application that’s more secure that they’re involved in developing.

    So, Kiran Kamity, great to have you here, CEO and founder, co-founder—I guess founder [Laughter] of DeepFactor.io. And be sure and check out the speakers that you all have at Cybersecurity 2021. It’s a great conference and we look forward to hearing more from you, there. Thanks for joining me today. Great to talk with you.

    Kamity: Thank you much. Appreciate the opportunity.

    Ashley: You bet.

    Kamity: Bye.

  • Engineering Practices Can Overcome DevOps Challenges

    Engineering Practices Can Overcome DevOps Challenges

    When I think about the challenges of DevOps transformations, I love the viewpoint expressed by Ryan Harmon, strategic accounts manager at Trace3: “DevOps is a complex maze that has many leaders frustrated. In many large enterprises, ‘DevOps’ is simply a buzzword they are striving to achieve but struggle with what the end state looks like or even where to start. There are many layers across the organization that are instrumental to truly executing on DevOps. DevOps is not something you go get a quote for and simply buy. It’s an evolving journey.”

    In my previous article, The Time Has Come for an Engineering Approach to DevOps, I agree with Ryan.

    Too many organizations launch into a DevOps transformation initiative without considering the full scope of what it takes to engineer a successful DevOps solution. DevOps is more than “just” a CI/CD pipeline or toolchain, as complex and challenging as this alone may be to accomplish, for even one application. DevOps is more than simply a culture change, as difficult as that always is.

    There are 24 DevOps practice categories that need to be mastered to realize and evolve well-engineered, high-performance enterprise DevOps solutions. Not all 24 need to be mastered to begin the DevOps transformation, but, ultimately, all of them do need to be mastered before the solution will achieve the high performance levels to which many enterprises aspire. The good news is, successful enterprises have demonstrated that the investment in people and technologies required for the journey is worthwhile, and a strategic approach will get you there systematically if you follow a disciplined engineering implementation roadmap.

    As explained in my book, Engineering DevOps, the core of every successful DevOps practice includes nine core pillars. These are the pillars to focus on as you start on your DevOps journey. The following paragraphs illustrate requirements for each of the nine pillars.

    Leadership – Leaders set an inspiring DevOps directional vision for the organization, and proactively stimulate and sponsor team activities that help progress toward goals.
    Collaborative culture – The organization encourages cross-functional collaboration, shared responsibilities and works to reduce restrictive boundaries between organizational functions.
    Design for DevOps – Products are architected to support service-oriented, modular, independent packaging, testing and releases, in accordance with the principles of twelve-factor apps.
    Continuous integration – Changes to software code and artifacts needed for releases are frequently merged, built, tested and packaged into executable artifacts for application releases.
    Continuous testing – Software tests that are most important to verify the results of pipeline stages including pre-flight, integration, regression, performance and release acceptance tests are automated as much as possible.
    Elastic infrastructure – Infrastructure-as-a-service (IaaS) resources (i.e., software, computing machines, storage and networks) are integrated into a toolchain for automating builds and testing environments to efficiently support variable workload demands.
    Continuous monitoring – Applications, pipeline tools and infrastructure components are instrumented with metrics, and the collected data is analyzed to manage health and performance.
    Continuous security (DevSecOps) – Security practices, processes and technologies for applications, databases, pipelines and infrastructures are integrated into the culture and pipelines to assure security.
    Continuous delivery – Release artifacts are validated, prepared for deployment to production and may be automatically deployed to production once acceptable deployment measures are achieved.

    A Comprehensive Engineering Blueprint

    While the above nine practice categories are the core of every successful DevOps solution, 14 additional practice categories are essential parts of a comprehensive engineering blueprint for accomplishing high-performance DevOps.

    DevOps infrastructure practices are used continuously by the nine core practice categories as follows:

    Version management systems – Every stage in the end-to-end DevOps value stream depends on keeping track of versions of software source code, configuration data and executable images using source code management systems, artifact repositories and configuration management systems.

    Infrastructure-as-a-Service – Virtualized cloud and dedicated resources must be served up ephemerally to meet variable workload and storage demands, which requires mastering engineering practices for orchestrating dedicated and cloud, hybrid cloud and multicloud services, and deployment of applications to the infrastructure.

    DevOps pipeline practices, which constitute a layer just above the nine core practices, include two additional categories that make use of the nine core DevOps practices.

    Application release automation – Directing and monitoring software changes through the CI/CD pipeline requires process orchestration and data analysis working at a level above the CI/CD pipeline itself.

    Value stream management – A deployed service, composed of multiple application microservices and data services, requires orchestration and analysis of a federated set of services, end-to-end, across the value stream from planning through to operations.

    Management Practices

    DevOps management practices include four categories that make use of the DevOps Pipeline practices, as follows:

    Service catalogs – Organized collections of software stacks, tools and pre-configured pipelines, made available through portals, enable DevOps-as-a-Service operations and simplify access, configuration, control and evolution of DevOps resources needed by enterprise application teams.

    Disaster prevention and recovery – Proactive disaster monitoring coupled with high-availability configurations and disaster mitigation practices, such as automated processes, work together to minimize the blast radius of disastrous events, and orchestrate rapid restoration of essential DevOps services, when needed.

    Site reliability engineering (SRE) – A combination of practices which include service level objectives (SLOs), error budgets, monitoring, work-sharing policies, toil reduction, proactive anti-fragility practices and blameless post-mortems help to “shift left” production wisdom from operations into development, thus improving the efficiency, readiness, security and resiliency of applications and infrastructures in production.

    Governance – The scale and scope of enterprise DevOps solutions are controlled using governance policies-as-code implemented through service catalogs, management dashboards and error budget policies.

    Project Practices

    DevOps project practices include four categories that guide enterprise adoption and evolution of DevOps, as follows:

    Adoption – Enterprise-wide DevOps adoption strategies that engineer and build DevOps capabilities in logical phases. This begins with a small set of representative applications that serve as a model for other applications; this method has been shown to be more successful than “boil the ocean” strategies that attempt to transform all applications at the same time.

    Model applications – Applications – that can serve as a model for greater enterprise adoption – are selected and monitored according to criterion that spotlight the value of DevOps for other applications.

    Return on investment (ROI) – DevOps investments must compete with other IT projects for resources. An ROI model that demonstrates measurable cost savings is an approach that has proved to achieve greater numbers of DevOps project approvals compared to other methods.
    Continuous improvement – Competitive forces and industry trends require that DevOps solutions continue to evolve to keep up with enterprise requirements.

    The Human Element

    Human development practices include two categories, as follows:

    DevOps engineering – The development of DevOps engineer roles, skills and the structure of DevOps engineering within an organization is critical to the success of designing, building, maintaining and evolving DevOps solutions.

    Continuous learning – A DevOps continuous learning strategy is critical for the development and transfer of DevOps knowledge needed to develop and continuously improve DevOps for the enterprise.

    In summary, the importance of practice categories – and specific practices within each of the categories – will evolve as each application does, alongside the changing needs of the enterprise. DevOps is complex, but a comprehensive engineering strategy that considers the 24 DevOps practice categories described in this article can help enterprises achieve the high-performance DevOps solutions they need to justify the investment and to meet business goals.

  • Security Policy Management and Hybrid Cloud with Tufin

    Security Policy Management and Hybrid Cloud with Tufin

    In this TechStrong TV / DevOps Chats interview, Colby Dyess, director of cloud product management at Tufin, joins Mitch Ashley to discuss security policy management across hybrid cloud, multi-cloud and cloud-native security. Recorded in late 2020, Colby shares some experience from working with customers during the rapid migration to the cloud in 2020.

    The video and podcast audio of the conversation is below, followed by the transcript. Enjoy!

    TechStrong TV Video

    DevOps Chats Podcast Audio

    Transcript

    Mitch Ashley: I’m joined today by Colby Dyess who is director of cloud product management at Tufin. Welcome, Colby. Welcome, it’s good to be talking with you. Would you introduce yourself, tell us a little bit about you and of course, tell us about Tufin.

    Dyess: Yeah, well, you’ve got my name and my title, so I’m dialing in here from beautiful Southern New Hampshire where it’s fantastic weather. So, I’ve worked for Tufin Technologies, Tufin is really well known in the industry for providing security policy management, mostly in the organizations like the Global 2000 complex environments where you need to make changes to the network environment and you need to be able to do it quickly, safely, ideally you do it automatically and it’s secure. The Tufin product suite helps organizations do that and we’ve done that in the traditional space. Of course, our customers have adopted cloud, cloud native platforms, Kubernetes, and we’ve been taking these same capabilities for securing the environments and bringing them into the cloud, so no matter where you put your workloads, we can help you, and that’s mostly what we’ve been focused on.

    Ashley: Good stuff. Boy, the center for the universe for a lot of folks. Kinda speaking of that and just thinking about the timing of where we all are, you know, there’s a lot of statistics. I run an analyst firm as part of MediaOps organization called Accelerated Strategies Group. Anyway, we have a lot of data showing some of the acceleration that’s happened, both in digital transformation and also move to the cloud, just in the last six to nine months. I don’t think that’s news to anybody, right? I don’t know if you need data to say that, because it’s happening probably in most people’s organizations.

    Dyess: Right.

    Ashley: If I put on my security hat for a moment, we always get plenty of notice when we’re making changes and doing new projects and accelerating stuff—sarcasm—so, I’m sure there’s a lot of security folks who are scrambling still just to not only keep pace, but make sure we’re all doing a good job of securing environments that our developers are taking this into, making sure we’re understanding the services we might be using now that we weren’t before, cloud services that we’ve been in. I’m sure you’ve seen a ton of that happening with your customers.

    Dyess: Yeah. It’s no surprise. Organizations have been moving to public cloud environments for a long, long time, at least, technically, for the past 15 years, they’ve been doing it, but it hasn’t always been sanctioned, right? Let’s say over the past five or eight years, organizations have really begun to intentionally adopt the cloud. And now this year, of course, everybody’s moved to the cloud about as quickly as they could.

    What we’re seeing, at least what I’ve seen a lot is, security is now scrambling to try and get ahold of some visibility—what’s really going on inside these cloud environments—and trying to put some sort of guardrails around the services that the Dev teams are using. That’s a real challenge because they’ve been focused so much on how to secure most of the on prem environment or use firewalls to protect the cloud space, but now they realize they’ve got to go a bit deeper, because the threats are pretty real out there and they’re really dynamic. And if you’re not sure how to secure the cloud environment, you’re going to leave yourself open to threats.

    So, security has really been spending a lot of time trying to learn more about cloud and cloud native security controls. We spend a lot of time helping customers figure out the difference between cloud native security controls and on prem stuff and then how to combine them so they can be assured it’s all secured. We see that a lot, yeah.

    Ashley: Well, you know, I mean, we’ve all been doing firewalls and intrusion prevention and access controls and the list goes on.

    Dyess: Yeah.

    Ashley: So, the topics aren’t new, but the environments are different and the services. It’s not like you can go into the cloud—no, let’s use all the things I’m normally using in my on prem environment. Maybe you can get some software versions of that. Sometimes it’s just better to use because it’s better integrated, you spend less time making the environment work by using what’s provided by the cloud provider.

    Dyess: Yeah. That pattern is the same everywhere. I’ve seen that you’re gonna go to a new environment, you wanna take some of the things that have worked for you in the past and carry them forward. And a lot of times, that makes sense as you try to move quickly into the cloud. If you’re not familiar with cloud native security controls, you probably think, “I’ll just use firewalls and the process for managing firewalls in the cloud.” And there are a lot of options to do that, a lot of the traditional vendors in that space have tried to make a pivot to offer their same services, same capabilities in the cloud environment. So, it’s no surprise that we see a ton of that.

    At the same time, those processes around managing all that stuff still takes too long. If you moved to the cloud, you went there for agility, right? You want to rapidly build applications and get them out into the market as quickly as possible. But if you’ve got to fill out a ticket to get a firewall changed and it takes three days for that stuff to be processed, you’re no longer as agile as you used to be when you could twiddle a little configuration in a Terraform template, redeploy, and—poof, you’re up and running all over again. That’s something that security teams—

    Ashley: If you take another two days to answer that ticket, you’re probably gonna have a fourth after your third _____ while you’re add it to the mix, right? Some developers are gonna be like, “I’m outta here, let me go set up the next environment.”

    Dyess: Yeah, it’s too difficult. You can’t wait. You don’t want to see shadow IT any longer, because IT was taking too long to respond, but you’ve got to respect that IT’s there, the security site of IT is just really trying to help protect the business.

    But you know, you also saw Amazon and AWS, they both announced their own firewalls, right? Azure did a little bit back and AWS just did within the past month. And I think that’s really just paying homage to the fact that those are really good technologies. Firewall technologies really are good stuff and it’s more than security groups or NSGs, it goes beyond that, and the businesses really need these. These do address customer problems. So, we’re seeing a lot of folks ask about how do they adopt those new to the cloud provider, new technology, and how do they integrate them into the very agile processes? And I expect we’ll see a lot more of that.

    In fact, what we are finding that we do often is try to help organizations see how they can get the most value out of the cloud native security controls, because these cloud environments are really, they’re pretty solid. And like you pointed out, the security controls are integrated into the services. So, it makes a lot of sense to use the services, they’re there to protect your assets and they’re so integrated in the services, you really should be trying to use them, but it’s a learning curve for security, so it means a lot to try and help them get the visibility and then start to affect control over that, so that they don’t have to slow down the rest of the business, swapping out, let’s say, a cloud native control with a legacy control [Cross talk].

    Ashley: Mm-hmm. It is that tradeoff, and I think that’s one of those where it’s sort of, I think it puts security teams in a position where, are we gonna stick to our guns of this is the environment we’re gonna have consistent across all of the environments we operate in, or for sake of speed but not sacrificing security, maybe we do look at some other options and can those integrate into the monitoring and management systems as opposed to using the same firewall in every environment that we take with us there, right?

    Dyess: Yeah, yeah.

    Ashley: Easier said than done. I appreciate it’s not an easy job.

    Dyess: Right. You know, whether you decide to use the cloud native security controls or use the traditional kinda controls, the virtualized versions of those or a combination of them, one thing I think we’re sharing is, we see a lot of conversations around how—which technologies to adopt and then how to actually manage those technologies.

    And from my point of view, it seems like we’re almost too focused on getting down into the weeds about the specific security control and, as a security person, I think it makes a lot of sense to step back a little bit and try and focus on what’s the overall security policy. Instead of thinking what firewall should I use, should I use one from the cloud or should I use from my traditional vendor, maybe a better question to help organizations remain agile is for security to be able to focus on what the security policies are, what the guardrails should be, and then begin to get the practice of checking cloud environments against those guardrails, against these policies as part of your CI/CD pipeline, the whole buzzword of shift left.

    That’s a real thing, right? That’s really important. That security can focus on defining security policies where they have the strength, but the actual implementation of enforcing the security policy is probably handled by the cloud operations or whoever actually twiddles the bits on the security controls. Like, if you can marry those two things together and you can do it in the CI/CD pipeline, then that’s a huge win.

    We’ve seen organizations move a lot faster because security focuses on what they do and they get it codified in a way that can play in a pipeline, and now the app teams, they can make changes to an application or a cloud Ops can change the configuration of an environment and get an alert saying it doesn’t comply with the security policy. That’s great. To step away from choosing which sort of security device you want to use and focus on do things comply with the security policy? That’s a major shift, and one that seems to be really hard for organizations to do. I think to kinda get away from security being hands on with the control and now focused on the policy—it’s trust but verify, I think.

    Ashley: [Laughter] Back to that old saying, right?

    Dyess: Yeah.

    Ashley: You know, in a way, it also comes down—you know, we’re used to, let’s first take an inventory and make sure we know what’s all out there that we’re trying to secure.

    Dyess: Sure, of course.

    Ashley: But it’s equally just as important to have, whether it’s change management practices or software that can help you with notifying when changes occur or understanding, so when a change occurs, how does that map back to policies that we have in place and are we still in alignment, are there gaps that we have to take care of? Because the environment isn’t—it isn’t an environment we control everything in any more. It changes by itself, because other people are changing it along with us. So, it seems like it’s a more dynamic environment than maybe what we had 10 years ago, if you reflect back a bit.

    Dyess: Intentionally more dynamic, right? Businesses have to move faster, we also have to react more quickly to customer demand or competitive threats or whatever. So, the fact that we’ve got these dynamic environments is just a reflection of what we actually needed to have built, anyway. And now, we need to find a way to incorporate that back into our regular practices.

    So, you made the comment about being able to track what’s changed and when and all that, but if you adopt more of an infrastructure as code model, you already have that. You don’t need a CMDB—maybe you do, but you don’t have to have a CMDB in place when you could go back to your Git logs and see, oh, here’s the pull request for the change that we made that allowed for this security group to be open or for this Azure firewall to have this particular rule. You can actually see that information, but these are, on the security side, these are new tools. As a security person, when was the last time you looked into a Git repo to see what the configurations were? [Laughter]

    Ashley: The repo—right, GitOps and security, right?

    Dyess: Right. Right, but we’ve got to get somewhere around there, as a group collectively, whether we’re writing the code or we’re managing the operations or we’re managing the security collectively, we’ve got to be able to collaborate and use these tools that I can’t say are merging, they’ve been around a long time. Terraform and Helm, Ansible, Chef, Puppet, the whole bit—these things have been around for a while, but to incorporate them now into our security practices is a pretty big leap for folks.

    Ashley: Mm-hmm.

    Dyess: So, we really try to encourage that the security team gets the visibility that they need, that’s the first thing, as you pointed out, right? You absolutely need to understand what’s deployed, what’s talking to what and whether or not things are properly segmented.

    You’ve got to first get that visibility, but I think before you start jumping into let’s throw a bunch of other control points in there, consider what resources are there and how can you, on the security side, participate in automation, participate in these flows that your CloudOps, DevOps, SRE, whomever had set up. Because there’s a really good opportunity now on security to be part of this agile process and to allow the business to be secure, but to do so in a way that won’t break the developer productivity.

    Ashley: Mm-hmm.

    Dyess: And that’s, it’s uncomfortable, because it’s a big change, but it’s a change that I know it can be done successfully. We’ve worked with companies that have done that, you know, make the conscious decision.

    Ashley: I almost wish we had stepped back and redefined shift left, because if, you know, I were king of the world for a nanosecond, I would think about it as, it’s not just about getting security people to work with the software teams earlier in the life cycle of software—yes, it’s that. But think about it this way of, the old world was, set the policies and then how do we implement the enforcement mechanisms, right, to make sure that those are being followed.

    Dyess: Mm-hmm.

    Ashley: I think in the world we’re in now, it’s, security people are, if you were gonna set up a security practice in an organization today, what I would do is, let’s start with how software is created, let’s work with the team about how the entire tool chain is set up, how that flow happens, both for application software, but also for the infrastructure and how that gets created where we’re talking about test environments and production environments and maybe you’re stepping into infrastructure as code if it really, truly is kinda dynamic Terraform or whatever kind of environment.

    But start there, because if you understand how that’s done, now you know how to secure it. You know how to work with teams to help them secure it, then the next layer would be getting into the software architecture itself around microservices and things that are more porous than we’re used to, kinda the hard outer shell of monolith applications of the past.

    That, to me, is the mental shift, the big mental shift for a security team to not necessarily be an expert at all that stuff, but understanding that’s what you’re really securing now.

    Dyess: So, two things that stand out, like responses I can imagine people having right now and that’s, I don’t have time to go and learn all that stuff. [Laughter] I’ve already got 20 other tickets for all high priority things. I’ve gotta get on that right now, so I can’t go and learn all that stuff. Valid, that’s definitely the case in a lot of organizations, yeah.

    And another problem related to this is security teams who maybe do want to know, maybe have some time to do that, but struggle that the application teams themselves don’t even know what makes up the application and what connections do they need. So, even the folks who you think would know because they built it, you’d think they would know the answers to some of these questions, but it turns out that they don’t—in a shockingly large number of scenarios, that’s the case.

    Ashley: Mm-hmm.

    Dyess: And so, if you’re gonna shift left, you’ve got to address not just the problem of the security team getting visibility in this—or maybe, actually, it’s the same thing. The application teams have to be able to come to the table to say, “This is what the application looks like. Here’s how it’s put together. Here are the resources it needs and here are the services that will rely upon it.”

    Ashley: Mm-hmm.

    Dyess: And so, if application groups can come to the table with that, then a security team can look at it usually very quickly, look at a map and say, oh, I can see where it needs to be segmented this way. I can see where he’s a resource that has too close of an access to PII, PCI, right? And so, that conversation could start a lot sooner if it’s part of your normal bill practices, you were generating these views, you had some way to produce an item that a network team or security team could look at, right? They don’t need to look at the source code, they need to look at your network topology, they need to understand what you’re connecting to.

    So, if you can bridge that gap, that goes a long way. So, to your point, it’s not just like, “I just need to check if it doesn’t comply with the policy.” Sometimes, it’s even just trying to define the policy at the outset.

    Ashley: Yeah, on what you’re trying to secure.

    Dyess: Yeah, but also, while security never seems to get the heads up about the things that are coming down, the joke that you made at the beginning, it’s also the case that when the application team is building something that they don’t really have all the rules, they’re not sure what all the restrictions are, and there are certainly plenty of guardrails, guidelines that they could have if you give them access that they could pull from an API to do their own checks.

    So, you know, there’s a model. We’ve done this before in the industry, back when I wrote code, we had a separate QA team, so we’d write code, we’d do a little bit of testing, but just enough, frankly, to get us through the build, right? And then we throw it over the wall and it went over to QA. And then thankfully, our development practices matured to a point where we realize we actually need to integrate testing as part of what we do. So, as developers, we had to own some of that testing, some more of the testing, but also importantly, the test team got involved earlier in the application life cycle, the model you were talking about. So, the QA teams could come in and learn a bit more and begin to set up some of the additional tests and participate in the build process, so what they did showed up in NMake.

    So, I think security is on a path like that as well. I think security, instead of app teams just throwing it over the wall and asking security to figure out how to lock it all down, I think we’re going to, and in some places, some customers for sure are moving to a place where security is part of the process earlier, and if they can say, “Here’s some of the tests,” so now the guardrails, they can contribute some of that and they can go into this build process, then now you’ve really begun to get that shift left.

    But I know this can happen. We’ve seen it happen before. Now QA is just integral to what we do. You can’t even imagine putting a product out without some part of QA being part of the build process and tests happening really early on, test planning happening early on. I imagine security just going in that same sort of loop.

    Ashley: Yeah, of course Tufin’s been in this space for a while, has built up, I’m assuming, a body of knowledge about this. If you were gonna advise maybe a new customer or an existing customer on—okay, we’re always playing catch up, we’re really playing catch up now with how much more cloud services that organizations are using in such a short period of time, what are some of the sort of best practices or recommendations of, “Here’s to sort of get the first thing established, here’s the next thing to work on, here’s the next thing to work on.”

    Dyess: Yeah.

    Ashley: Any suggestions about that?

    Dyess: Sure. We have this whole crawl, walk, run methodology, and it’s designed to help organizations take these small steps towards getting to really secured environments. And the first step right away is just to get visibility, which is kind of a “no, duh” statement, but it’s actually one of the big problems that orgs have. You’ve got to be able to see what’s deployed inside your cloud, what’s deployed inside your cluster, and which resources are communicating with each other, where are the dependencies.

    So, understanding that is a really major first step. It’s true for the application teams, but for the security, it’s to know are things properly segmented, where are the biggest risks? So, getting that awareness, that’s really the crawl phase. And in the walk phase, that’s where you take what you’ve learned about what’s deployed, what talks to what, and whether or not things are segmented properly and then you begin to isolate. Here are specific accounts that we need to tackle or here are clusters that we need to tackle—like, hone in on the higher priority items and begin to define a security policy for those cloud native environments.

    And as you develop the cloud native policies, which you then share with the application teams or the cloud operators, whoever’s responsible for configuring it. As you develop that sort of, that skill, you’ve gotta get to the run phase, and the run phase is when you begin to automate this. You automate generating security policies, you automate validating security policies, but at the very beginning, where most organizations are at right now is, first get the visibility and then the second step of defining a security policy. Not defining what is my security group definition, not defining what is my AWS firewall definition, but what is my security policy? What are the rules that should govern access to applications, assets, resources of certain categories? Those are two big steps to take right away.

    Ashley: I think that’s some great advice. I’m curious, any thoughts as we—I think we all want to turn the page on 2020 and kinda get to 2021, and hopefully, we’re looking at a different kind of a year. But one of the things about 2020 is, we’re not gonna go back, I don’t think. We’re not gonna go back to, let’s rewind to January and start working in that environment again, right? Everything is changed with remote work or how much we’re using the cloud.

    What are some things that you’re anticipating for 2021 as you look forward in the next 6 to 12 months. Do you kinda think it will be topics maybe not as talked about yet but will be more topical in 2021? I’m just kinda asking you—

    Dyess: Are you asking for predictions of 2021?

    Ashley: You know, I don’t see it, but I’m pretty sure, there might be a crystal ball right behind your chair, just look back a little bit there and kinda check back. [Laughter]

    Dyess: [Laughter] Well, I think, given the maturity of the market adoption of public cloud environments, I think we’re going to see more security teams getting their arms around how to help in securing these cloud environments without necessarily always relying on the traditional devices. We’re already starting to see that even now. So, I think we’ll see more of the cloud native security control adoption. I know organizations have seen the benefits of doing that to lower cost of operations, just total lower cost of ownership, plus the benefit of being able to say when there’s a problem, I can call my cloud vendor, I don’t have to worry about vendors poking at each other. So, I think we’ll see more of that.

    I think it’s the Kubernetes space that’s probably the most interesting, because it’s the least well known for most folks on the security side.

    Ashley: Mm-hmm.

    Dyess: So, I’d say maybe over the past year, there’s been a lot of emphasis, just like with cloud, slap a firewall on it and we’ll solve the problems of micro-segmentation later, but later is gonna happen real quick, and I’d expect in 2021, you’ll see securities start to get more involved in getting visibility into the clusters and then looking to define the security policies.

    And not—again, not to try and define the policy of an individual cluster, but just the general rules, because there’s no way we can keep up with what will really be thousands of services running inside of a cluster.

    Ashley: Yeah.

    Dyess: So, we’ll see that. I think we’ll see more organizations at the security side have to deal with the scale and the rate of change and begin to participate in the automation processes that the Dev teams are using to manage it.

    Ashley: Maybe that’s something we’ll hear more frequently. I wouldn’t be surprised if the—let’s think of it as cloud native security, not just as security, sort of think about it as one of you architectural approaches to how you do security.

    Dyess: Cloud native. The native controls of the cloud provider—yeah, that’s a big shift for folks, I think.

    Ashley: It is. We’ll see if something like that happens. And I think the things you said made a lot of sense. Well, it’s been great talking with you, it’s the first chance you and I have had to talk before, and—

    Dyess: Yeah, it was good to talk to you, Mitch.

    Ashley: – a lot of fun. Tell us a little bit about where folks can learn more about Tufin.

    Dyess: Yeah, of course, you can go to Tufin.com, that’s T-U-F-I-N dot com. On the cloud side, like what’s really relevant to what we’re doing here, we’ve got a lot of resources at Tufin.com/try-securecloud. It’s cloud native security controls. It’s really designed to help you get visibility and control the cloud native security components. Great stuff there, free to use product. There’s a great trial for people to give a go. Tufin.com would be the place.

    Ashley: Great. Perfect. Thanks. Well, we wish you the best and a even better 2021, we’ll look forward to that.

    Dyess: Yeah. From your lips, man.

    Ashley: [Laughter]

    Dyess: Good luck in 2021, everybody.

    Ashley: There you go. Hang onto that, hang onto that. Alright, well, take care. It was very nice talking with you, Colby—Colby Dyess from Tufin.

  • Protecting Data from Insider Threats

    Protecting Data from Insider Threats

    “Insider threat” can be defined as the threat an employee or a contractor will use their authorized access to, wittingly or unwittingly, do harm to an organization’s security. Certainly, as organizations become data-driven, embrace data democratization and release new applications at breakneck speed, the probability of an insider threat is rising sharply.

    A common example of an insider threat is granting permissions to on-call SRE engineers who may need to access system tables across multiple databases, inspect contents of S3 buckets and make changes to the production environment to fix availability issues. Every two weeks, the composition of the team may change, and the composition of services and databases may vary dramatically.

    While it is impossible to completely safeguard against insider threats, organizations can better protect their data by investing in best practices that (a) reduce the likelihood of such a threat playing out, (b) limit the blast radius from an insider attack and (c) help detect the attack as quickly as possible. The third point is especially important – the Ponemon Institute’s recent Cost of Insider Threats 2020 Global Report claims that it takes, on average, 77 days to detect and contain an insider attack.

    A DevOps Framework for Protecting Data From Insider Threats

    For security measures to be effective, they must be non-invasive, especially in agile organizations where DevOps teams are expected to push releases several times a day.

    Implementation of Access Policies-as-Code

    Implementing access policies-as-code automates workflows for user and service access authorization. Using this approach, manual policy practices are replaced with “code” (policy files readable by your team and by the systems that enforce the policies) so that the right people on your team can request, approve and publish policy changes instantly.

    As an example, here’s how the above scenario may be implemented as code:

    Engineers can add themselves to on-call status with a code push in GitHub to a Terraform script, with only the team lead authorized to approve this action. (See example 1, below.)

    insider threats
    Example 1

    Using frameworks like Sentinel, changes to production environments can be limited to only on-call engineers. (See example 2, below.)

    insider threats

    Example 2

    Similar policies can limit only those engineers who are able to read unencrypted content of S3 buckets by fetching keys from Vault. (See example 3, below.)

    insider threats

    Example 3

    The above examples demonstrate guardrails that allow engineers to perform their duties, while drastically reducing the administrative burden on leads, resulting in fewer errors. An additional benefit of this approach is that it forces a public review of all changes to policies, which makes the security model more robust.

    Zero Standing Permissions

    Zero Standing Permissions (ZSP) is a DevOps-friendly security tenet that requires any administrative privilege to be available to users only when needed, and only for as long as needed. It enables teams to configure access, via administrative privileges, to highly critical components of their infrastructure for a limited duration, with restrictions on the scope of those privileges.

    Here are a few examples of how teams can implement ZSP for their databases:

    • Create an SRE role on the database that can update and delete database tables and a private Slack channel of db-admins that can allow on-call engineers to assume this role.
    • Implement a service that can be used to request and grant access via this SRE role for a default of 60 minutes. After that time, the access is automatically revoked and needs to be granted again.
    • Further implement policies in BrewOPA that restrict an SRE from dropping tables and databases, which avoids any disasters.

    Audit Trail for Privileged Activity

    SIEMs are increasingly the nerve center of every security organization. While effective threat-hunting and incident response requires several years of expertise, DevOps teams can significantly bolster the security of their cloud platforms by investing in log collection and basic analytics for all privileged activity in their environment.

    These tools include services such as the following:

    • Session recording for all Linux boxes using Cmd
    • Logging activity across databases with Data Cloud services
    • Centralizing and collecting all environment logs using services like CloudTrail.

    These logs can be visualized in tools like Kibana, where teams can look at activity pattern anomalies. These could include:

    • Someone opening up an FTP connection to outside the organization
    • Creating a manual copy of tables outside of the database
    • An increased number of authentication failures across resources

    While the concept of “least privilege” as a safeguard against insider threats to data has existed for some time, the advent of cloud-native and DevOps methodology has exacerbated the challenge of implementation. In recent years, security-as-code and infrastructure-as-code have emerged as good models for implementing security solutions, and are rooted in the above framework. The future encourages collaboration across various stakeholders, documents security policies for easy review and builds on existing tools and workflows. It will help limit insider threats, limit privilege sprawl and prevent “user fatigue.”

  • Cloud Misconfigurations Threaten Cloud Migration

    Cloud Misconfigurations Threaten Cloud Migration

    If we’ve learned one thing during the COVID-19 pandemic, it’s that digital-native business models are essential to survival. That means cloud adoption is more important than ever before. Research shows that the public cloud market is expected to grow to $308.5 billion in 2021, an increase of 16% from 2020.

    But while the cloud holds incredible potential, we have an outstanding security issue to address. That issue is cloud misconfigurations. Through 2025, 99% of cloud security failures will be the responsibility of the customer, quantifying the need for security professionals to turn their attention to security hygiene issues like cloud misconfigurations.

    Digging Into the Cloud Misconfiguration Problem

    To get to the root of the problem, we must fully understand the risk of cloud misconfigurations and how they differ from more traditional vulnerabilities. While traditional vulnerabilities can be patched, cloud misconfigurations can create vulnerabilities in otherwise secure applications and infrastructure. Imagine cloud infrastructure like a highway system and cloud misconfigurations as road hazards. They can be incredibly dangerous and lead to accidents.

    As a specific example of cloud misconfigurations, let’s look at identity and access management (IAM). Poorly configured IAM, such as misconfigured roles or a lack of multi-factor authentication (MFA), can lead to compromised administrative accounts. If a threat actor hijacks a legitimate administrator account, they potentially can take full control of an entire cloud environment. Because IAM sits above the cloud infrastructure layer and all workloads and data within, once it is exploited a threat actor will often be able to circumvent your other security barriers, such as network segmentation, leaving you defenseless.

    Cloud Misconfigurations Can Easily Scale

    Where misconfiguration risk dramatically increases is through the adoption of cloud-native applications and practices like infrastructure as code (IaC) templates used by DevOps teams. These templates offer teams greater speed and scale for building and managing applications, but the downside is that misconfigurations can be unknowingly replicated from development environments to production environments (where sensitive data is stored) at greater velocity.

    In short, the biggest risk to organizations right now is scaling these misconfigurations through the cloud. As the cloud grows in adoption and scale, we’re witnessing these misconfigurations scale right alongside it. Where before, a misconfiguration might have been limited to a siloed application or environment, today, that same misconfiguration can impact the entire organization without checks in place to catch it. Worse, many cloud storage buckets have logging disabled, meaning once threat actors are able to identify a misconfiguration and access an internal cloud bucket, an organization won’t even be able to see what data was accessed.

    Understanding the Shared Responsibility Model

    This all points to a higher-level, hygiene-related issue to keep in mind when securing cloud environments: responsibility. Particularly, organizations struggle to understand the shared responsibility model and how it applies within their own organizations. While some may fail to delineate between cloud provider responsibilities and their own, the major issue at stake is the shared responsibility between various internal teams that often goes undiscussed and undocumented until there is a security event.

    When thinking about software-as-a-service (SaaS), to a much lesser degree the responsibility falls on consumers. However, when considering infrastructure- and platform-as-a-service (IaaS/PaaS) and all the moving parts such as network, user credentials, resource configurations, workloads, identity configurations and more, cloud consumers become responsible for much more. A key consideration to note is that one can never outsource accountability, no matter which cloud model is used. To put this more simply, if one puts data in any cloud provider, they are still accountable for that data.

    Enabling a Holistic Cloud Security Strategy

    When advising security and business leaders on how they can better secure their cloud environments and applications, I highlight what my team calls “The Big Cloud 5,” a set of recommendations developed to help organizations adopt a holistic cloud security model that accounts for proper security hygiene and shared responsibility.

    • Gain awareness and deep cloud visibility
      The very first step to ensuring cloud security is understanding how teams are using cloud technologies, leveraging shadow IT and cloud provider APIs. This allows you to get situational awareness and make informed decisions today as well as in the future. This is not a one-time event, but something you’ll need to do continuously.
    • Set guardrails to automatically prevent the most serious cloud misconfigurations
      Drawing lines in the sand around the most offensive (and potentially destructive) misconfigurations that should never exist in an environment is key to automating protection in the cloud. This will help keep templates and practices controlled, so that poor hygiene doesn’t inadvertently take root and spread. Think of this as your “dirty dozen.” What configurations should never exist in your cloud environments?
    • Standards are the precursor to automation
      One can’t automate what hasn’t been standardized, and while there aren’t widely accepted security standards yet, key stakeholders in an organization must be in agreement about how to secure cloud infrastructures.
    • Train and hire security engineers who code
      To fully leverage APIs, security teams must have engineers who know how to code and automate security processes. An assessment of skills that exist across your security team (e.g., knowledge of coding in the likes of Python or Ruby) can point to areas in training and hiring that need further investment.
    • Embed security in the development pipeline
      Map out who, what, when, where and how your organization pushes code into the cloud. Once mapped, identify the least disruptive insertion points for security processes and tools, so that they can exist in as much of the development pipeline as possible.

    If you feel your organization is a step behind on proper cloud security hygiene, know that you’re not alone. Earlier in 2021, the Cybersecurity and Infrastructure Security Agency (CISA) released a report highlighting the importance of strengthening security configurations based on the uptick in successful cloud attacks, often attributed to poor hygiene and the mixed use of computing devices in the remote work environment. While this is a growing problem, it is also a reversible trend. Best practices, cyber hygiene and a shared responsibility model can help companies safely and securely migrate to the cloud.

  • Building Trust Into Your Product by Design

    Building Trust Into Your Product by Design

    In today’s world, every tool, every solution and every technology has the potential to do both good and bad. Good, if used according to the way it was designed, but bad if it falls into the wrong hands, or is manipulated or abused. As the ‘Peter Parker’ principle says, “With great power comes great responsibility.”

    As the COVID-19 pandemic continues and consumers look to digital communications to maintain some sense of normalcy and connectedness during these desolate times, the onus is on us, industry leaders, to ensure that the products and services being developed to connect and communicate with disparate employees, customers, and partners aren’t weaponized by bad actors.

    The best way to do this? By building trust into your design.

    You Have to Define Trust to Know How to Build Trust

    According to Merriam-Webster, the dictionary definition of trust is “one in which confidence is placed.” But the reality is that in today’s world, there are many ways trust can be perceived.

    Think of the automotive industry, where trust can be tangibly seen in the growth and success of certain car brands over decades. There, trust can be seen as ‘brand loyalty,’ or customers who repeatedly purchase from the same brand because they trust the company and trust the product.

    Trust in technology is the same. It has to be engineered and cultivated, but at the end of the day trust in the product must be mirrored by consumer trust in the organization. The two go hand in hand.

    The best way to approach building and sustaining trust when developing new products boils down to three key things: integrity, third-party accountability and consistency. Integrity stems from delivering high grade products and solutions, in conjunction with demonstrating accountable, transparent communications at the organizational level. Third-party validation further underscores that integrity, and restates the quality of products and services being delivered from a third-party perspective. And finally, consistency; being able to deliver exemplary products and solutions on a continual, consistent basis – the most essential ingredient in maintaining and bolstering trust among customers.

    Transparent Communication is Key

    As we enter 2021, the threat landscape is fraught with bad actors looking to take advantage of the latest threat vectors and emerging technology. As our lives continue to transition from physical to digital, and as more personal information is stored online, companies have to prioritize proper and timely communication about cybersecurity incidents.

    This brings us back to organizational trust being foundational for trust in a product. If you’ve seen the Netflix documentary ‘The Social Dilemma’ you know that social media is an oft-cited example of technology being misused or manipulated by bad actors. But after Twitter suffered a social engineering attack in July, the platform issued a swift update to its users. They were open, prompt and transparent in their analysis of the event, and highlighted not only the updated safety precautions put in place in response to the breach, but also underscored their ongoing commitment to trust and transparency within the Twitter community.

    Incidents like these, though harmful in the short term, can be less detrimental if addressed and communicated in a timely, transparent manner. There’s a saying, ‘It’s not what happens to you, but how you react to it that matters.’ The same goes for proper breach disclosure within the tech industry as a whole.

    Trust has Never Been More Foundational to the Business of Building

    As we enter 2021 and the COVID-19 pandemic rages on, more companies than ever are finding themselves in the business of building. Healthcare providers are drumming up new ways to digitally engage with patients and providers outside the physical confines of a hospital. Financial services organizations are adding more immediate, personalized ways to engage with customers when, where and how they need it. Educators are creating new digital solutions to keep millions of children, learning from home for the foreseeable future, engaged. And retailers are rebuilding the in-store experience for customers entirely online.

    With so much building going on around us, it’s important to keep in mind that the building process today requires not only outlining what you’re trying to build and defining how you intend to use it, but also analyzing how it can potentially be abused, manipulated or rendered ineffective.

    Consistently running your products and solutions against the latest threat models is one way to ensure your products’ security posture is up to date with the latest attack vectors. Another idea is to incorporate ‘security champions’ into your product development process and teams, to act as mini consultants who evaluate potential cybersecurity vulnerabilities. Once you’ve incorporated methods like these, turning to a third party to assess and sign off on your product’s cybersecurity resiliency is a good way to validate your products’ security adherence and effectiveness in the face of ever-evolving threats.

    Trust and Development Should go Hand in Hand

    At its core, trust and development should go hand in hand. Developers don’t need to recreate the wheel every time – there are plenty of libraries available online to leverage and pull from. What’s most important is for developers to understand the libraries they’re using to build, and ensuring that proper security protocols are built in from the start.

    Trust can seem like a huge lift, or an unnecessary distraction on the way to an MVP, product launch or update. I assure you, this is not the case. No matter how big the effort may appear, investing in trust – and the various methods, ways and changes that harden your products and services against abuse – will pay dividends down the line. Customers will trust your product and your company, and will, therefore, be more likely to remain loyal. If nothing else, keep in mind that building trust takes time, effort and consistency, while losing trust can happen in the blink of an eye.

  • DevSecOps May Be Key to Prevent Data Leaks

    DevSecOps May Be Key to Prevent Data Leaks

    Data leakage has become something of a norm in the digital realm, despite the fact that cybersecurity budgets are growing and more engineers are assigned to these very issues. Nowhere does this seem more true than in the realm of health care, where breaches that expose sensitive patient data are, sadly, a fairly common occurrence. 

    A recent report by Approov illustrates how severe the problem has become. According to Approov’s  All That We Let In research report, many of the mobile healthcare applications in use today are leaking sensitive patient data through APIs, potentially compromising millions of patients. 

    The study tested 30 popular mobile health (mHealth) apps, and the research estimates that some 23 million mHealth users are exposed, at a minimum. The average number of downloads for each app tested was 772,619. Analysts expect that the total number of users exposed by the 318,000 mHealth apps now available on major app stores is likely far greater. Among vulnerabilities detailed in the report:

    • 50% of the records accessed contained names, Social Security numbers, addresses, birthdates, allergies, medications and other sensitive data for patients.
    • 50% of the APIs tested allowed users (medical professionals) to access the pathology, x-rays and clinical results of other patients.
    • 50% of the APIs tested did not authenticate requests with tokens.
    • Of the 30 popular apps tested, 77% contained hardcoded API keys, some of which don’t expire, and 7% contained hardcoded usernames and passwords. Seven percent of the API keys belonged to third-party payment processors that warn against hard-coding their secret keys in plain text.

    “Look, let’s point out the pink elephant out in the room,” said Alissa Knight, researcher and author of the report. “There will always be vulnerabilities in code, so long as humans are writing it. Humans are fallible. But, I didn’t expect to find every app I tested would have hard-coded keys and tokens, and all of the APIs to be vulnerable to broken object-level authorization (BOLA) vulnerabilities allowing me to access patient reports, X-rays, pathology reports and full PHI records in their database. The problem is clearly systemic.”

    The report helps to illustrate a critical point: Sloppy code can lead to data leakage. “These findings are disappointing, but not at all surprising. The fact is, leading developers and their corporate and organizational customers consistently fail to recognize that APIs servicing remote clients, such as mobile apps, need a new and dedicated security paradigm,” said Approov founder and CEO David Stewart. “Because so few organizations deploy protections for APIs – that ensure only genuine mobile app instances can connect to backend servers – these APIs are an open door for threat actors, and present a real nightmare for vulnerable organizations and their patients.” 

    For organizations deploying apps using a DevOps methodology, it becomes very clear that much more must be done on the cybersecurity side to prevent data leakage. However, the general consensus is that data protection falls under the purview of CISOs and cybersecurity teams in the enterprise. Perhaps there is no better argument for adopting DevSecOps than the data leakage problems caused by insecure code and sloppy APIs. The report echoes that sentiment and offers several recommendations for developers creating and organizations using mobile applications. In other words, organizations should take several key steps to protect their customer data and sensitive resources, including:

    •  Address both app security and API security: Recognize that synthetic traffic to the API is an issue, and arises from bots and automated tools, not from genuine apps and legitimate data requests.
    • Shift left and shield right: Secure the development process and harden apps, but ensure that runtime protection is also in place.
    • Protect against X-in the middle attacks: Certificate pinning is critical, but often left undone because expired certificates can block apps and impact customer experience. However, when done correctly, certificate pinning does not impact either performance or availability.
    • Improve visibility into controls: Organizations and developers need to monitor the effectiveness of the controls they implement and adjust them easily – both for compliance with legislative mandates and to sustain data security and privacy.
    • Continuous testing: Penetration testing and static and dynamic code analysis should be performed regularly.

    Interestingly, those recommendations fit squarely into the role that DevSecOps should take in the DevOps process. 

  • DevSecOps Implementation: Patch Management

    DevSecOps Implementation: Patch Management

    Patch management has been a thorn in IT’s side for decades. While it is undeniably necessary for security and stability purposes, the staggering array of patches that are available make managing them effectively nearly a full-time job – or a job better left to users. Few IT professionals want to be the “patch person,” and yet, nearly all IT professionals want their systems protected by the latest patches. No one wants to find out an intrusion was easily preventable via patching. It goes without saying that the SolarWinds attack has left quite a few IT departments scrambling for an alternate solution.

    The world of patch management has gone through several evolutions that we won’t cover in much detail here. Instead, we’ll focus on what’s available, what you should look for and where the market is headed. Suffice it to say that patch management as a practice has grown from an individual applying patches to servers and server apps while employees were encouraged to patch their equipment, to complex systems that manage the patching themselves, with IT oversight. Currently, patch management is often bundled with endpoint management, simply because many of the patches are going to end points (user machines and devices).

    Out of this growth have come several market differentiators that you should be aware of when looking into a way to manage the avalanche of updates the average organization is seeing.

    • Centralized or de-centralized? Increasingly, patch management is run on servers/cloud and pushes changes out to those machines/devices that need them, or agents on each machine manage updating themselves. Which is best for a given organization really varies. If a lot of machines are running at the edge of their capability, an agent running updates in the background might be too much to ask. In that scenario, a server-managed install environment is probably better. Conversely, if a centralized location to run updates from is considered an architectural weakness (because it requires access to the update pushing engine), agents might be the better plan.
    • OS support. We’re way past the days when single-OS solutions were considered useful. List the operating systems used in your environment, and choose a product accordingly. Most products on the market support Windows/Mac/Linux, but there are variances. First off, make certain your chosen product supports all three. Then ask, “Which flavors of Linux?” Make certain the flavors running in-house are supported by the software.
    • App support. There are a lot of applications out there. Which are supported by the tool? How much leeway does the customer have for adding applications by pointing the tool at update locations? Does it support FOSS updates? There are a lot of those too, and which ones matter. Make certain the tools the organization uses are supported or can be added.
    • Many tools will scan machines for update issues. Understand what your chosen tools scan for, and how the scans impact performance on the target machines.
    • Updates are not 100% perfect. There will inevitably be problems. Make certain the tool can roll back changes. Updating 50 core infrastructure machines, only to find, hours or days later, that the update breaks something demands roll-back capability.
    • Inter-relationships. Updating a piece of software can require that another piece of software or a driver be updated, which can demand that UEFI/BIOS be updated. Or the CMOS on a RAID controller. Understand how cascading updates are handled by the tool, and if the cascading updates can be packaged, so that a failure automatically rolls back the updates applied thus far.
    • Approvals. Speaking of driver and BIOS updates, what tools are available to allow a human to approve changes? No one wants willy-nilly updates to the drivers on core systems – or even to drivers on user systems.
    • Security of the tool itself. We’re in perilous times where IT security is concerned, and frankly, an update command-and-control server is an attacker’s dream machine – take it over, put your back-door update into the mix, and tell it to update every machine it manages. It cannot be understated: an update management tool must be as protected as possible – both by the customer locking down the installation and by the developer making sure the code is solid.

    Once the system is chosen and worked into your DevSecOps environment, updates will be more responsive, which leaves one more thing to consider – does it feed into your agile tools? If the update management tool notifies a project management tool like JIRA when it updates a server, then if code on the server breaks, the developers aren’t in the dark about changes, and will have breadcrumbs to help them track down the source of the problem.

    Imagine if manual updates became a rarity instead of the norm. There is a cost associated with this process, but most of the products on the market can be had for a reasonable price – if you waste a couple hours a month (or for some products, a year) on updating, they’re paying for themselves. Of course, your time is a sunk cost, and the cost of these is an operational cost, so there is an evaluation your organization will have to undertake. Given how busy everyone in DevOps and DevSecOps is, it is likely worth the investment for all but over-staffed organizations. What? There are a few!

    Find what suits your organization, and put it to use. Faster updates, less time investment. The idea is sound, you just need to figure out what suits your needs. And keep rocking it. You are managing updates already, this is just a way to make your life easier. And we could all use a bit of easier in a busy, complex IT environment.